Image retrieval method, image retrieval device, image retrieval equipment and medium

Abstract

Disclosed are an image retrieval method, an image retrieval device, an image retrieval equipment and a medium. The image retrieval method comprises: extracting global features of an input image, and obtaining a global feature code based on the global features; extracting content features of the input image, and obtaining a content feature code based on the content features; splicing the global feature code with the content feature code to obtain a composite feature code of the input image; comparing the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result. By comprehensively considering the global features and content features of the image during the image retrieval process, the accuracy and efficiency of the retrieval result are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more specifically to an image retrieval method, an image retrieval device, an image retrieval equipment and a medium. Background Art

[0002] Image retrieval is a key technology in the field of image processing. With the widespread application of image processing in civil and commercial fields, image retrieval is also facing higher requirements. Currently, the commonly used method in image retrieval is to obtain the overall features or global features based on the input image, and further search through the global features, and finally output an image similar to the input image.

[0003] However, when using the global feature-based image retrieval method, when searching for some special images, such as art paintings, photos of artworks, or other images with more prominent or core content, the retrieved results tend to be more similar in color and texture, while ignoring the similarity of content in the paintings.

[0004] Therefore, there is a need for an image retrieval method that has a high image retrieval accuracy rate under the premise of achieving image retrieval. Summary of the invention

[0005] In view of the above problems, the present disclosure provides an image retrieval method, device, equipment and medium. The image retrieval method provided by the present disclosure can effectively improve the image retrieval accuracy and retrieval efficiency on the basis of realizing image retrieval, realize high-precision image retrieval, and the method has good robustness.

[0006] According to one aspect of the present disclosure, an image retrieval method is proposed, comprising: extracting global features of an input image, and obtaining a global feature code based on the global features; extracting content features of the input image, and obtaining a content feature code based on the content features; concatenating the global feature code with the content feature code to obtain a composite feature code of the input image; and comparing the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result.

[0007] In some embodiments, extracting global features of an input image and obtaining a global feature code based on the global features includes: extracting features of the input image through a first convolutional neural network to obtain global features; converting the global features into a binary feature code with a first preset number of bits to obtain a global feature code.

[0008] In some embodiments, extracting content features of an input image and obtaining a content feature code based on the content features includes: locating a target in the input image through a second convolutional neural network to obtain a positioning result; cropping the image based on the positioning result to obtain a local area image, wherein the local area image at least includes the target; extracting content features from the local area image through a third convolutional neural network; and converting the content features into a binary feature code having a second preset number of bits to obtain a content feature code.

[0009] In some embodiments, comparing the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result includes: calculating the Hamming distance between the composite feature code and each preset feature code in the preset feature code library; and determining the image retrieval result based on the Hamming distance.

[0010] In some embodiments, the preset feature code library includes at least one preset feature code, and wherein: for a preset image in the image library, the global features of the image are extracted, and the global feature code of the image is obtained based on the global features; the content features of the image are extracted, and the content feature code of the image is obtained based on the content features; the global feature code of the image is concatenated with the content feature code of the image to obtain a composite feature code of the image, and the composite feature code is used as the preset feature code.

[0011] In some embodiments, before extracting the features of the input image through the first convolutional neural network and obtaining the global features, it also includes: generating a sample set based on preset images in the image library; training the first convolutional neural network through the sample set; and saving the trained first convolutional neural network.

[0012] In some embodiments, locating the target in the input image through the second convolutional neural network, before obtaining the positioning result, also includes: generating a sample set based on preset images in the image library; training the second convolutional neural network through the sample set; and saving the trained second convolutional neural network.

[0013] In some embodiments, before extracting features of local area images through a third convolutional neural network to obtain content features, it also includes: generating a sample set based on preset images in a picture library; training the third convolutional neural network through the sample set; and saving the trained third convolutional neural network.

[0014] According to another aspect of the present disclosure, an image retrieval device is provided, including: a global feature processing module, configured to extract global features of an input image, and obtain a global feature code based on the global features; a content feature processing module, configured to extract content features of the input image, and obtain a content feature code based on the content features; a composite feature code generation module, configured to concatenate the global feature code with the content feature code to obtain a composite feature code of the input image; and a feature comparison module, configured to compare the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result.

[0015] In some embodiments, the global feature processing module includes: the global feature processing module includes: a global feature extraction module, configured to extract features of the input image through a first convolutional neural network to obtain global features; a global feature coding generation module, configured to convert the global features into a binary feature code with a first preset number of bits to obtain a global feature code.

[0016] In some embodiments, the content feature processing module includes: a target positioning module, configured to locate the target in the input image through a second convolutional neural network to obtain a positioning result; a local area image generation module, configured to crop the image based on the positioning result to obtain a local area image, wherein the local area image at least includes the target; a content feature extraction module, configured to extract content features from the local area image through a third convolutional neural network; and a content feature coding generation module, configured to convert the content feature into a binary feature code with a second preset number of bits to obtain a content feature code.

[0017] In some embodiments, the feature comparison module includes: a Hamming distance calculation module configured to calculate the Hamming distance between the composite feature code and each preset feature code in the preset feature code library; and an image output module configured to determine the image retrieval result based on the Hamming distance.

[0018] In some embodiments, the preset feature code library includes at least one preset feature code, and wherein: for a preset image in the image library, the global features of the image are extracted, and the global feature code of the image is obtained based on the global features; the content features of the image are extracted, and the content feature code of the image is obtained based on the content features; the global feature code of the image is concatenated with the content feature code of the image to obtain a composite feature code of the image, and the composite feature code is used as the preset feature code.

[0019] According to another aspect of the present disclosure, an image retrieval device is provided, wherein the device comprises a processor and a memory, the memory contains computer-readable instructions, and when the computer-readable instructions are executed by the processor, the image retrieval device performs the method as described above.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, characterized in that computer-readable instructions are stored thereon, and when the instructions are executed by a computer, the method described above is executed.

[0021] By using the image retrieval method provided by the present disclosure, image retrieval can be well completed based on the global features and content features of the input image. In particular, it can have a high image retrieval accuracy rate and the algorithm has good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. The following drawings are not deliberately scaled according to the actual size, and the focus is on illustrating the main purpose of the present disclosure.

[0023] Figure 1 An exemplary flow chart of an image retrieval method 100 according to an embodiment of the present disclosure is shown;

[0024] Figure 2A An exemplary flow chart of extracting global features of an input image and obtaining global feature encoding based on the global features according to an embodiment of the present disclosure is shown;

[0025] Figure 2B Shows Figure 2A The network structure used by the method in

[0026] Figure 2C An exemplary flow chart of training a first convolutional neural network according to an embodiment of the present disclosure is shown;

[0027] Figure 3A A flowchart of an exemplary method 300 of extracting content features of an input image and obtaining content feature codes based on the content features according to an embodiment of the present disclosure is shown;

[0028] Figure 3B Shows Figure 3A The network structure used by the method in

[0029] Figure 3CAn exemplary flow chart of training a second convolutional neural network according to an embodiment of the present disclosure is shown;

[0030] Figure 3D An exemplary flow chart of training a third convolutional neural network according to an embodiment of the present disclosure is shown;

[0031] Figure 4 A flowchart of an exemplary method 400 for comparing the composite feature code with a preset feature code library to obtain an image retrieval result according to an embodiment of the present disclosure is shown;

[0032] Figure 5 An exemplary block diagram of an image retrieval device 900 according to an embodiment of the present disclosure is shown;

[0033] Figure 6 An exemplary block diagram of an image retrieval device 950 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, rather than all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present disclosure.

[0035] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0036] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0037] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0038] Figure 1An exemplary flow chart of an image retrieval method 100 according to an embodiment of the present disclosure is shown.

[0039] First, in step S101, global features of an input image are extracted, and global feature encoding is obtained based on the global features.

[0040] The input image may be an image captured in real time by a camera or a video recording device, or may be an image obtained in advance in other ways. The disclosed embodiments are not limited by the source and acquisition method of the input image. For example, it may be an image directly captured by a road camera, a surveillance camera of an unmanned supermarket, etc., or it may be an image obtained after being pre-processed by a computer.

[0041] Next, in step S102, content features of the input image are extracted, and content feature codes are obtained based on the content features.

[0042] The process of extracting the global features and content features of the input image can be implemented through a deep learning algorithm, or can also be implemented through other methods. The embodiments of the present disclosure are not limited to the method of extracting the global features and content features of the image.

[0043] It should be understood that the operations of step S101 and S102 can be performed in parallel or in sequence, and no limitation is imposed on them. Further, as required, steps S101 and S102 can be performed based on different images obtained after preprocessing or local feature extraction, as long as these images are derived from the same original input image.

[0044] After obtaining the global feature code and the content feature code, further in step S103, the global feature code and the content feature code are concatenated to obtain a composite feature code of the input image. The concatenation can be obtained by directly connecting the global feature code and the content feature code.

[0045] For example, the obtained global feature code is (w1, w2, w3), and the obtained content feature code is (c1, c2, c3, c4, c5). After concatenating them in sequence, a composite feature code (w1, w2, w3, c1, c2, c3, c4, c5) corresponding to the current input image can be obtained.

[0046] Further, as required, the splicing may first group and splice the sub-elements of the global feature code and the content feature code according to the different types to which they belong, or based on the subsequent determination of the demand classification, the global feature code and the content feature code belonging to the same category. The embodiments of the present disclosure are not limited by the selected splicing method.

[0047] For example, the obtained global feature encoding is (wa ,w b ,w d ), the obtained content feature code is (c a1 ,c a2 ,c b1 ,c b2 ,c d ), where w a 、c a1 、c a2 For example, both represent the color features of the image, w b 、c b1 、c b2 For example, both represent the texture features of the image, w d 、c d For example, if they all represent the shape features of the image, they can be spliced ​​based on different categories to obtain the composite feature code corresponding to the current input image (w a ,c a1 ,c a2 ,w b ,c b1 ,c b2 ,w d ,c d ).

[0048] After the composite feature code is obtained through splicing, in step S104, the composite feature code is compared with the preset feature code in the preset feature code library to obtain the image retrieval result.

[0049] The above comparison process can be achieved by calculating the spatial distance between the composite feature code and the preset feature code, or by comparing and distinguishing the corresponding data between the two and generating the final comparison result based on the respective discrimination results. In the embodiments of the present disclosure, there is no limitation on the comparison method.

[0050] By extracting global features and content features from the input image, corresponding global feature codes and content feature codes are generated respectively, and the composite feature codes obtained by splicing the global feature codes and content feature codes are used to realize image retrieval. In the retrieval process, the overall similarity of the image and the similarity of the core content of the image are comprehensively considered, which solves the problems of poor content similarity and low accuracy of retrieval results in image retrieval, especially retrieval of images of art paintings, realizes high-precision retrieval, and further improves the retrieval efficiency.

[0051] Figure 2A An exemplary flowchart of extracting global features of an input image and obtaining global feature encoding based on the global features according to an embodiment of the present disclosure is shown. Figure 2B Shows Figure 2A The network structure used in the method.

[0052] Reference Figure 2A ,First, in step S201, the features of the input image are ,extracted through the first convolutional neural network to obtain ,global features.

[0053] The first convolutional neural network can be selected based on actual needs, for example, using a convolutional neural network algorithm of a deep residual network (Resnet) model, a convolutional neural network algorithm (R-CNN) based on a visual geometry group (VGG) model, or a convolutional neural network algorithm based on an Inception model, or using a composite neural network formed by integrating it with a fully connected network or other comprehensive processing network architecture. No restrictions are imposed on the type of the first convolutional neural network selected.

[0054] This step can be done, for example, by Figure 2B The first convolutional neural network shown in is implemented, specifically, first, an image is input into the input layer of the first convolutional neural network, and then, after being processed by multiple convolutional layers of the first convolutional neural network, the global features after image feature extraction are finally obtained in the fully connected layer. The obtained global features can be in the form of a multi-dimensional global feature vector, such as 4096 dimensions or 2048 dimensions. For example, it can be a 2048-dimensional feature vector, which is specifically composed of (6.24, 0.98, 3.12, -1.46, ... 0.32). The disclosed embodiments are not limited by the dimension of the feature vector of the obtained global features and their specific values.

[0055] After the global feature is obtained, in step S202, the global feature is converted into a binary feature code of a first preset number of bits to obtain a global feature code.

[0056] The first preset number of bits may be, for example, less than the dimension of the feature vector of the global feature, equal to the dimension of the feature vector of the global feature, or greater than the dimension of the feature vector of the global feature. For example, the first preset number of bits may be, for example, 512 bits, or may also be 1024 bits. The embodiments of the present disclosure are not limited by the specific value of the first preset number of bits set and its relationship with the dimension of the feature vector of the global feature.

[0057] The binary feature code represents that each bit of the code is 0 or 1. The binary feature code may be in the form of a vector, an array or a string, and the embodiments of the present disclosure are not limited to the form of the binary feature code.

[0058] By converting the extracted continuous global features into binary feature codes with a first preset number of bits, it helps to simplify calculations and improve retrieval speed in the subsequent retrieval process.

[0059] The conversion process of the binary feature code may be implemented, for example, by a hash coding algorithm, or may be implemented by other methods. The embodiment of the present disclosure is not limited by the method of obtaining the binary feature code.

[0060] When using the hash coding algorithm, refer to Figure 2B In the non-limiting embodiment shown, a hash conversion layer is provided after the fully connected layer of the convolutional neural network, and the hash conversion layer may include a fully connected layer, an activation layer, a loss layer, and a thresholding layer.

[0061] Furthermore, the hash coding process can be described more specifically. For example, the global feature vector obtained is an m-dimensional vector W T , the first preset number of bits of the global feature encoding is set to n, then the m-dimensional global feature vector W is first converted into T Mapped to n-dimensional global mapping vector W S Then, in the activation layer, the global mapping vector W is transformed by using the hyperbolic tangent function, i.e., the tanh function. S Converted into a global activation vector W Q , the global activation vector W Q The value of each sub-element in is constrained to be in the range of -1 to 1. Then, the global activation vector W Q The output is sent to the loss layer, where the target structure of the global activation vector is optimized to compensate for the errors caused by discretizing the continuous value encoding.

[0062] In some embodiments, the error calculation in the loss layer can be implemented, for example, using an S-shaped growth curve function (Sigmoid), or can be implemented using a method that takes into account both cross entropy loss and quantization loss. The embodiments of the present disclosure are not limited to the specific method of performing error calculation.

[0063] When implemented using a method that takes into account both cross entropy loss and quantization loss, the loss function can be:

[0064] F=min L+λQ (1)

[0065] Wherein F represents the loss function, L represents the cross entropy loss, Q represents the quantization loss, and λ is a parameter used to weigh the cross entropy loss L and the quantization loss Q, which can be selected based on the characteristics of the image. The cross entropy loss L and the quantization loss Q can also select different functions according to the characteristics of the image. The embodiments of the present disclosure are not limited by the specific selected functions and algorithms.

[0066] The global result vector W obtained by calculating the loss function R It is further output to the thresholding layer, in which the global result vector WR Perform threshold binarization processing based on a preset threshold value for the global result vector W R Each sub-element W in R _i (i = 1, ..., n) performs threshold binarization, the specific formula is as follows:

[0067]

[0068] Where W O _i is the global feature code W O The i-th sub-element in , and i = 1, ..., n, n is the first preset number of bits. For example, when the global result vector W R is (1,-1,1,1,1,-1), then based on the above formula, the final global feature encoding W O is (1,0,1,1,1,0).

[0069] By converting the extracted continuous global features into binary feature codes with a first preset number of bits, it is helpful to simplify the calculation and improve the retrieval speed in the subsequent retrieval process. Furthermore, by calculating the error, the feature loss caused by discretizing the continuous value code into a binary code is compensated, taking into account both speed and feature accuracy.

[0070] In addition, the existing image library can be used to train the first convolutional neural network.

[0071] Figure 2C An exemplary flow chart of training the first convolutional neural network 210 according to an embodiment of the present disclosure is shown. With reference to the above figures, the training process of the above first convolutional neural network can be described in more detail.

[0072] Reference Figure 2C For the first convolutional neural network, first, in step S211, a sample set is generated based on the preset images in the picture library. The sample set may be, for example, a sample image generated by performing image enhancement processing on the preset images, such as flipping the preset images in the picture library left and right or removing the color mean; secondly, the first convolutional neural network is trained by the sample set. Specifically, in step S212, the sample set is input into the first convolutional neural network, and the sample set is processed by multiple convolutional layers in the first convolutional neural network to obtain an output result in the fully connected layer; in step S213, the loss function of the output result is calculated, and the parameters of each layer of the first convolutional neural network are adjusted based on the loss function; finally, in step S214, the trained first convolutional neural network is saved for image retrieval.

[0073] Figure 3AA flowchart of an exemplary method 300 for extracting content features of an input image and obtaining content feature codes based on the content features according to an embodiment of the present disclosure is shown. Figure 3B Shows Figure 3A The network structure used in the method.

[0074] First, in step S301, content features of an input image are extracted.

[0075] Specifically, in step S3011, the target in the input image is located by a second convolutional neural network to obtain a positioning result.

[0076] According to different embodiments, the second convolutional neural network can realize target positioning based on different preset rules. For example, red can be designated as the target color, or the area in the center of the image can be designated as the target, or it can also be based on time domain rules and frequency domain rules, such as setting the salient area with the smallest entropy in the spectrum scale space (Spectrum Scale-Space) of the image after Gaussian kernel smoothing as the target, or setting the area with high contrast relative to the surrounding area of ​​the input image after filtering as the target. The embodiments of the present disclosure are not limited by the selection rules of the set targets.

[0077] The second convolutional neural network can be, for example, a deep convolutional neural network, a network formed by combining a neural network with an intersection-over-union algorithm, a non-maximum suppression algorithm, or a composite neural network formed by integrating the neural network with a fully connected network or other comprehensive processing network architecture. The embodiments of the present disclosure are not limited by the type of the second convolutional neural network.

[0078] The target in the input image may be one or more targets, and the embodiments of the present disclosure are not limited by the number of targets in the input image.

[0079] Furthermore, if Figure 3B As shown, for example, the image can be input to the input layer of the selected second convolutional neural network, and then processed by multiple convolutional layers in the second convolutional neural network structure, and the processing results are output to the global average pooling layer (GAP) of the second convolutional neural network. The global average pooling layer calculates the above processing results and further outputs the calculation results to the fully connected layer, and the fully connected layer performs classification and output, and finally obtains the positioning result of the target in the image at the output end of the second convolutional neural network.

[0080] The positioning result can be represented by a coordinate vector. For example, for each target in the image, the coordinates of the four corner points of a rectangular area including the target and having the smallest area can be used. For example, the positioning result is represented by the offset of the four corner points of the rectangular area relative to the center point of the input image. However, according to the embodiment of the present disclosure, the positioning result can also be a circular or irregular closed figure that surrounds the target and has the smallest area. The embodiment of the present disclosure does not limit the representation method of the positioning result.

[0081] After obtaining the positioning result, in step S3012, the image is cropped based on the positioning result to obtain a local area image, where the local area image at least includes the target.

[0082] For example, when the image includes only one target, the local area image includes the one target, which may be an image including the target. When the image includes multiple targets, such as three targets, the local area image is an image including the three targets.

[0083] The local area image may be, for example, a rectangular image, or based on the specific shape of the target, may also be a circular image or an irregular image. The embodiment of the present disclosure does not limit the shape and size of the local area.

[0084] Finally, in step S3013, the features of the local area image are extracted through the third convolutional neural network to obtain content features.

[0085] Specifically, the local area image is input into the input layer of the third convolutional neural network, and after being processed by the convolutional layer, the content features after image feature extraction can be obtained in the fully connected layer. The obtained content features can be multi-dimensional feature vectors, such as 2048-dimensional or 1024-dimensional. For example, it can be a 2048-dimensional content feature vector C T , which is specifically composed of (16.11, 10.23, 3.14, -1.64, ... 0.32). The disclosed embodiment is not limited by the dimension of the feature vector of the obtained content feature and its specific value.

[0086] The third convolutional neural network can be selected based on actual needs, for example, using a convolutional neural network algorithm of a deep residual network (Resnet) model, a convolutional neural network algorithm (R-CNN) based on a visual geometry group (VGG) model, or a convolutional neural network algorithm based on an Inception model, or using a composite neural network formed by integrating it with a fully connected network or other comprehensive processing network architecture. No restrictions are imposed on the type of the third convolutional neural network selected.

[0087] In some embodiments, the third convolutional neural network may have the same neural network structure as the first convolutional neural network, for example.

[0088] However, the embodiments of the present disclosure are not limited thereto. For example, the third convolutional neural network may also have a different network structure from the first convolutional neural network. In the embodiments of the present disclosure, no restriction is made on the network structure relationship between the third convolutional neural network and the first convolutional neural network.

[0089] By locating the target in the image and cropping the image based on the positioning result, a local area image with the smallest area including all the targets in the image is obtained, and further feature extraction is performed on the local area image. It is possible to extract content features in the image. When extracting content features, it is possible to effectively reduce the interference of attribute features of background areas and non-target areas in the image, which is conducive to obtaining content features with higher accuracy.

[0090] In addition, the training data in the existing image library can be used to train the third convolutional neural network.

[0091] Figure 3C An exemplary flow chart of training the second convolutional neural network 310 according to an embodiment of the present disclosure is shown; Figure 3D FIG. 3 is an exemplary flow chart of training the third convolutional neural network 320 according to an embodiment of the present disclosure. Referring to the above drawings, the training process of the above convolutional neural network can be described in more detail. Figure 3C For the second convolutional neural network, first, in step S311, a sample set is generated based on the preset images in the picture library. The sample set may be, for example, a sample image generated by performing image enhancement processing on the preset images, such as flipping the preset images in the picture library left and right or removing the color mean; secondly, the second convolutional neural network is trained by the sample set. Specifically, in step S312, the sample set is input into the second convolutional neural network, and the output result is obtained through processing and calculation of multiple convolutional layers, global average pooling layers and fully connected layers in the second convolutional neural network; in step S313, the loss function of the output result is calculated, and the parameters of each layer of the second convolutional neural network are adjusted based on the loss function; finally, in step S314, the trained second convolutional neural network is saved for subsequent image retrieval.

[0092] Reference Figure 3D, the training process of the third convolutional neural network can be described in more detail. First, in step S321, a sample set is generated based on the preset images in the picture library. The sample set may be, for example, a sample image generated by performing image enhancement processing on the preset images, such as flipping the preset images in the picture library left and right or removing the color mean; secondly, the third convolutional neural network is trained by the sample set. Specifically, in step S322, the sample set is input into the third convolutional neural network, and processed by multiple convolutional layers in the third convolutional neural network to obtain an output result in the fully connected layer. In step S323, the loss function of the output result is calculated, and the parameters of each layer of the third convolutional neural network are adjusted based on the loss function; finally, in step S324, the trained third convolutional neural network is saved for image retrieval.

[0093] For example, the picture library may be an artwork gallery on the Internet or a local gallery, and the preset images may be images captured in real time by a camera or images of fine art paintings preprocessed by a computer, etc. The embodiments of the present disclosure are not limited by the source and type of the picture library and the preset images.

[0094] The sample set generated by the preset image can, for example, directly use the preset image as a sample. However, the embodiments of the present disclosure are not limited thereto. In some embodiments, generating the sample set includes performing image enhancement processing on the preset image, and the enhancement processing includes, for example, selecting and translating images in an existing gallery to increase the sample size and diversity of the training data.

[0095] By performing image enhancement processing on the preset image, the data training amount and data diversity of the convolutional neural network can be further increased, and the performance of the trained convolutional neural network can be further enhanced, thereby improving the speed and accuracy of image retrieval.

[0096] After the content feature is obtained, in step S302, a content feature code is obtained based on the content feature. Specifically, in step S3021, the content feature is converted into a binary feature code with a second preset bit number to obtain a content feature code.

[0097] The second preset number of bits may be, for example, less than the dimension of the feature vector of the content feature, equal to the dimension of the feature vector of the content feature, or greater than the dimension of the feature vector of the content feature. For example, the second preset number of bits may be 512 bits, or may be 1024 bits. The embodiments of the present disclosure are not limited by the specific value of the second preset number of bits and its relationship with the dimension of the feature vector of the content feature.

[0098] The second preset number of bits may be the same as the first preset number of bits, for example, both are set to 512 bits, or they may be different, for example, the second preset number of bits for content feature encoding is 128, and the first preset number of bits for global feature encoding is 512. The embodiment of the present disclosure is not limited by the relationship between the first preset number of bits and the second preset number of bits.

[0099] By converting the extracted features into binary feature codes with a second preset number of bits, it helps to simplify calculations and improve retrieval speed in the subsequent retrieval process.

[0100] The conversion process of the binary feature coding can be achieved, for example, by a hash coding algorithm, that is, a hash conversion layer is set after the fully connected layer of the convolutional neural network, and the hash conversion layer can include, for example, a fully connected layer, an activation layer, a loss layer and a thresholding layer.

[0101] Furthermore, for example, the obtained content feature vector is a y-dimensional vector C T , the second preset number of bits of the content feature encoding is set to k, then firstly, the y-dimensional content feature vector C is converted into T Mapped to k-dimensional content mapping vector C S . Then, in the activation layer, the hyperbolic tangent function is used to map the continuous content to the vector C S Converted into content activation vector C Z , the content activation vector C Z The value of each sub-element in is constrained to be in the range of -1 to 1. Then, the content activation vector C Z The output is sent to the loss layer, where the target structure of the above content activation vector is optimized to compensate for the errors caused by discretizing the continuous value encoding.

[0102] In some embodiments, the error calculation in the loss layer can be implemented, for example, using an S-shaped growth curve function (Sigmoid), or can be implemented using a method that takes into account both cross entropy loss and quantization loss. The algorithm used for error calculation for content features can be the same as the algorithm used for error calculation for global features, or different algorithms can be selected based on their different features. The embodiments of the present disclosure are not limited by the specific method for error calculation for content features.

[0103] The content result vector C obtained by calculating the loss function R is further output to the thresholding layer, in which the content result vector C R Perform threshold binarization processing based on a preset threshold value for the content result vector C R Each sub-element in is thresholded and binarized. The specific formula is as follows:

[0104]

[0105] Among them C O _i is the content feature code C O The i-th sub-element in , and i = 1, ..., k, k is the second preset number of bits. When the content result vector C R When (-1,-1,-1,1,1,-1), based on the above formula, the final content feature code C O is (0,0,0,1,1,0).

[0106] By converting the extracted continuous content features into binary content feature codes with a second preset number of bits, it helps to simplify subsequent operations and improve the retrieval speed. Furthermore, by calculating the error, the feature loss caused by discretizing the continuous value code into a binary code is compensated, taking into account both the retrieval speed and the acquired feature accuracy.

[0107] Figure 4 A flowchart of an exemplary method 400 for obtaining an image retrieval result by comparing the composite feature code with a preset feature code library according to an embodiment of the present disclosure is shown.

[0108] Reference Figure 4 After the global feature code and the content feature code are combined to obtain a composite feature code, first, in step S401, the Hamming distance between the composite feature code and each preset feature code in the preset feature code library is calculated.

[0109] The preset feature code library includes at least one preset feature code, and the specific process of obtaining the preset feature code is similar to steps S101 to S103 in the image retrieval method 100. Specifically: first, for a preset image in the image library, the global features of the image are extracted, and the global feature code of the image is obtained based on the global features; then, the content features of the image are extracted, and the content feature code of the image is obtained based on the content features; finally, the global feature code of the image is spliced ​​with the content feature code of the image to obtain a composite feature code of the image, and the composite feature code is used as the preset feature code.

[0110] Based on this, the preset feature code and the composite feature code of the input image have the same length (the two vectors have the same dimension).

[0111] The Hamming distance refers to the number of different values ​​at corresponding positions of two codes of equal length or vectors of the same dimension. For example, if code A is 100 and code B is 101, the Hamming distance between codes A and B is 1.

[0112] Based on the above, the process of calculating the Hamming distance between the composite feature code and the preset feature code in the preset feature code library and obtaining the corresponding value can be described in more detail. For example, after processing, the composite feature code D of the input image is (1,1,1,0,0,1,0,1,0), and the first preset feature code G1 of the preset feature code library is (1,1,0,0,0,1,0,1,1), and the second preset feature code G2 of the preset feature code library is (0,0,0,1,0,1,0,1,0), then the Hamming distance between D and G1 is 2, and the Hamming distance between D and G2 is 4.

[0113] Furthermore, in step S402, an image retrieval result is determined based on the Hamming distance.

[0114] The above process of determining the image retrieval result can be described in more detail. For example, in some embodiments, based on the Hamming distance, the image corresponding to the preset feature code with the minimum Hamming distance is output. For example, when the first preset feature code G1 has the minimum Hamming distance, only the image corresponding to the first preset feature code G1 is output.

[0115] However, the present disclosure is not limited to this. For example, an image corresponding to a preset feature code whose Hamming distance is less than or equal to a preset threshold value can also be output. Specifically, the preset feature code is screened based on the preset threshold value of the Hamming distance, and an image corresponding to one or more preset feature codes whose Hamming distance is less than or equal to the preset threshold value is output.

[0116] The preset threshold value may be set to, for example, 5, or may be set to 10. The embodiments of the present disclosure are not limited by the specific value of the preset threshold value.

[0117] For example, when the preset threshold is 5, for the composite feature code D of the input image, it can be compared with each preset feature code in the preset feature code library and sorted from small to large based on the Hamming distance. If its Hamming distance with the first preset feature code G1 is 2, and the Hamming distance with the second preset feature code G2 is 4, and the Hamming distances with the remaining preset feature codes are all greater than 5, then the images corresponding to the first preset feature code G1 and the second preset feature code G2 can be output accordingly.

[0118] In some embodiments, when the number of output preset feature codes is less than the preset output number, the preset feature codes may be output from small to large according to the Hamming distance.

[0119] For example, when there are 7 preset feature codes in the preset feature code library and the number of preset outputs is 8, since the total number of preset feature codes is less than the number of preset outputs at this time, all the images corresponding to the 7 preset feature codes can be output; when there are 7 preset feature codes in the preset feature code library and their Hamming distances are sorted from smallest to largest, for example, G1 < G2 < G3 < G4 < G5 < G6 < G7, when the number of preset outputs is 4, then in the order of increasing Hamming distance, the images corresponding to the preset feature codes G1, G2, G3, and G4 are output.

[0120] The retrieval process is realized by calculating the Hamming distance. Compared with the method of realizing retrieval by calculating the spatial distance, its computational complexity is lower and the calculation speed is faster, which helps to improve the retrieval speed.

[0121] Figure 5 Fig. shows an exemplary block diagram of an image retrieval device 900 according to an embodiment of the present disclosure.

[0122] Such as Figure 5 The image retrieval device 900 shown includes a global feature processing module 910, a content feature processing module 920, a composite feature code generation module 930, and a feature comparison module 940, which can execute the image retrieval method as Figure 1 shown.

[0123] Among them, the global feature processing module 910 is configured to extract the global feature of the input image and obtain a global feature code based on the global feature. The content feature processing module 920 is configured to extract the content feature of the input image and obtain a content feature code based on the content feature. The composite feature code generation module 930 is configured to splice the global feature code and the content feature code to obtain a composite feature code of the input image. The feature comparison module 940 is configured to compare the composite feature code with the preset feature codes in the preset feature code library to obtain an image retrieval result.

[0124] The input image may be an image captured in real time by a camera or a video recording device, or may also be an image obtained in other ways in advance. The embodiments of the present disclosure are not limited by the source and acquisition method of the input image. For example, it may be an image directly captured by a road camera, a surveillance camera in a unmanned supermarket, etc., or may also be an image obtained after being pre-processed by a computer.

[0125] The process of comparing the composite feature code with the preset feature codes in the preset feature code library can be realized by calculating the spatial distance between the composite feature code and the preset feature code, or the corresponding data of the two can be compared and judged and a final comparison result can be generated based on each judgment result. In the embodiments of the present disclosure, the comparison method is not limited.

[0126] Image retrieval is achieved by extracting global features and content features from the input image, generating corresponding global feature codes and content feature codes, and concatenating the global feature codes and content feature codes to obtain composite feature codes. This takes into account the overall similarity of the images and the similarity of the core contents of the images, and solves the problem of poor content similarity and low accuracy of retrieval results, especially in image retrieval of images of art paintings. This achieves high-precision retrieval and further improves retrieval efficiency.

[0127] The global feature processing module 910 may further include: a global feature extraction module 911 and a global feature code generation module 912, which may execute the process shown in FIG. 2 to extract the global features of the input image and obtain the global feature code based on the global features.

[0128] The global feature extraction module 911 is configured to perform the operation of step S201 in FIG. 2 , extracting the features of the input image through the first convolutional neural network to obtain global features. For example, a convolutional neural network algorithm of a deep residual network (Resnet) model, a convolutional neural network algorithm (R-CNN) based on a visual geometry group (VGG) model, a convolutional neural network algorithm based on an Inception model can be used, or a composite neural network formed by integrating it with a fully connected network or other comprehensive processing network architecture can be used. No restrictions are imposed on the type of the first convolutional neural network selected.

[0129] Furthermore, the global feature code generation module 912 is configured to perform the operation of step S202 in FIG. 2 , converting the global feature into a binary feature code having a first preset number of bits to obtain a global feature code.

[0130] By converting the extracted continuous global features into binary feature codes with a first preset number of bits, it is helpful to simplify the operation and improve the retrieval speed in the subsequent retrieval process. Furthermore, through the error compensation process, the error introduced by discretizing the continuous value code is compensated, taking into account both speed and feature accuracy.

[0131] The content feature processing module 920 further includes: a target positioning module 921, a local area image generation module 922, a content feature extraction module 923 and a content feature code generation module 924, which can perform the following operations: Figure 3A The process shown extracts content features of an input image and obtains content feature encoding based on the content features.

[0132] The target positioning module 921 is configured to perform the following steps: Figure 3AIn the operation of step S3011, the target in the input image is located by the second convolutional neural network to obtain a positioning result. The target in the input image can be one or more, and the embodiment of the present disclosure is not limited by the number of targets in the input image.

[0133] The positioning result can be represented by a coordinate vector. For example, for each target in the image, the coordinates of the four corner points of a rectangular area that includes the target and has the smallest area can be used. For example, the positioning result is represented by the offset of the four corner points of the rectangular area relative to the center point of the input image. However, according to the embodiment of the present disclosure, the positioning result can also be an irregular closed figure that surrounds the target and has the smallest area. The target positioning result can also be represented by other criteria as needed. The embodiment of the present disclosure does not limit the representation method of the positioning result.

[0134] The local area image generation module 922 is configured to perform the following steps: Figure 3A In the operation of step S3012, the image is cropped based on the positioning result to obtain a local area image, and the local area image at least includes the target.

[0135] The local area image may be, for example, a rectangular image, or based on the specific shape of the target, may also be a circular image or an irregular image. The embodiment of the present disclosure does not limit the shape of the local area.

[0136] The content feature extraction module 923 is configured to perform the following steps: Figure 3A In the operation of step S3013, content features are extracted from the local area image through a third convolutional neural network.

[0137] The content feature code generation module 924 is configured to perform the following steps: Figure 3A In the operation of step S3021, the content feature is converted into a binary feature code having a second preset number of bits to obtain a content feature code.

[0138] By locating the target in the image and cropping the image based on the positioning result to obtain the local area image, and further extracting the features of the local area image, the content features in the image can be extracted. When extracting the content features, the interference of the attribute features of the background area and non-target area in the image can be effectively reduced, which is conducive to obtaining more accurate content features.

[0139] The feature comparison module 940 may further include: a Hamming distance calculation module 941 and an image output module 942, which may perform the following operations: Figure 4 In the process shown, the composite feature code is compared with the preset feature code in the preset feature code library to obtain the image retrieval result.

[0140] The Hamming distance calculation module 941 is configured to perform the following Figure 4 The operation of step S401 in the above step is to calculate the Hamming distance between the composite feature code and each preset feature code in the preset feature code library. The image output module 942 is configured to perform the following steps: Figure 4 The operation of step S402 is to determine the image retrieval result based on the Hamming distance.

[0141] In some embodiments, the preset feature code library includes at least one preset feature code, and wherein: for a preset image in the image library, the global features of the image are extracted, and the global feature code of the image is obtained based on the global features; the content features of the image are extracted, and the content feature code of the image is obtained based on the content features; the global feature code of the image is spliced ​​with the content feature code of the image to obtain a composite feature code of the image, and it is used as the preset feature code.

[0142] The retrieval process is implemented by calculating the Hamming distance. Compared with the method of implementing retrieval by calculating the spatial distance, the calculation amount is lower and the calculation speed is faster, which helps to improve the retrieval speed.

[0143] like Figure 6 The image retrieval device 950 shown can be implemented as one or more dedicated or general-purpose computer system modules or components, such as a personal computer, a laptop, a tablet computer, a mobile phone, a personal digital assistance (PDA) and any intelligent portable device. The image retrieval device 950 may include at least one processor 960 and a memory 970.

[0144] Wherein, the at least one processor is used to execute program instructions. The memory 970 can exist in the image retrieval device 950 in different forms of program storage units and data storage units, such as hard disks, read-only memories (ROMs), random access memories (RAMs), which can be used to store various data files used in the image retrieval process and / or execution of the processor, as well as possible program instructions executed by the processor. Although not shown in the figure, the image retrieval device 950 can also include an input / output component to support input / output data flows between the image retrieval device 950 and other components (such as image acquisition devices 980). The image retrieval device 950 can also send and receive information and data from the network through a communication port.

[0145] In some embodiments, when the computer-readable instructions stored in the memory 970 are executed by the processor 960, the image retrieval device 950 performs operations, the operations including: extracting global features of the input image and obtaining global feature codes based on the global features; extracting content features of the input image and obtaining content feature codes based on the content features; concatenating the global feature codes with the content feature codes to obtain a composite feature code of the input image; comparing the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result.

[0146] In some embodiments, in order to extract the global features of the input image and obtain the global feature code based on the global features, the operations performed by the image retrieval device 950 include: extracting the features of the input image through a first convolutional neural network to obtain the global features; and converting the global features into a binary feature code with a first preset number of bits to obtain the global feature code.

[0147] In some embodiments, extracting content features of an input image includes: locating a target in the input image through a second convolutional neural network to obtain a positioning result; cropping the image based on the positioning result to obtain a local area image, wherein the local area image at least includes the target; and extracting content features from the local area image through a third convolutional neural network.

[0148] In some embodiments, obtaining the content feature code based on the content feature includes: converting the content feature into a binary feature code having a second preset number of bits to obtain the content feature code.

[0149] In some embodiments, the operation of comparing the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result includes: calculating the Hamming distance between the composite feature code and each preset feature code in the preset feature code library; and determining the image retrieval result based on the Hamming distance.

[0150] In some embodiments, the image retrieval device 950 may receive images acquired from an image acquisition device external to the image retrieval device 950, and execute the image retrieval method described above on the received image data, thereby realizing the functions of the image retrieval apparatus described above.

[0151] Despite Figure 6 In the embodiment, the processor 960 and the memory 970 are presented as separate modules. It can be understood by those skilled in the art that the above device modules can be implemented as separate hardware devices or integrated into one or more hardware devices. As long as the principles described in the present disclosure can be implemented, the specific implementation methods of different hardware devices should not be used as factors limiting the scope of protection of the present disclosure.

[0152] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a computer, the above-mentioned method can be executed.

[0153] The program part of the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.

[0154] All or part of the software may sometimes communicate over a network, such as the Internet or other communications network. Such communications may load the software from one computer device or processor to another. For example: from a server or host computer of an image retrieval device to a hardware platform of a computer environment, or other computer environment that implements the system, or a system with similar functions related to providing information required for image retrieval. Therefore, another medium capable of transmitting software elements may also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, may also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media involved in the process of the processor executing any instructions.

[0155] The present application uses specific words to describe the embodiments of the present application. For example, "first / second embodiment", "one embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or multiple times in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0156] In addition, it will be appreciated by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present application may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0157] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0158] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. An image retrieval method, comprising: Extracting global features of an input image, and obtaining a global feature encoding based on the global features; Extracting content features of an input image, and obtaining a content feature code based on the content features; The global feature code and the content feature code are concatenated to obtain a composite feature code of the input image; Comparing the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result; The image retrieval result obtained by comparing the composite feature code with the preset feature code in the preset feature code library includes: Calculating the Hamming distance between the composite feature code and each preset feature code in the preset feature code library; and Determining an image retrieval result based on the Hamming distance; Wherein, the preset feature code library includes at least one preset feature code, and wherein: For a preset image in an image library, extracting a global feature of the preset image, and obtaining a global feature code of the preset image based on the global feature; Extracting content features of the preset image, and obtaining a content feature code of the preset image based on the content features; The global feature code of the preset image is concatenated with the content feature code of the preset image to obtain a composite feature code of the preset image, and the composite feature code is used as the preset feature code.

2. The image retrieval method according to claim 1, wherein: Extracting global features of the input image and obtaining global feature encoding based on the global features includes: The features of the input image are extracted through the first convolutional neural network to obtain global features; The global feature is converted into a binary feature code having a first preset number of bits to obtain a global feature code.

3. The image retrieval method according to claim 1, wherein: Extracting content features of the input image and obtaining content feature codes based on the content features includes: Through the second convolutional neural network, the target in the input image is located to obtain the positioning result; Cropping the image based on the positioning result to obtain a local area image, wherein the local area image at least includes the target; Extracting content features from the local area image by a third convolutional neural network; The content feature is converted into a binary feature code having a second preset number of bits to obtain a content feature code.

4. The image retrieval method as claimed in claim 2, wherein: Before extracting the features of the input image through the first convolutional neural network and obtaining the global features, it also includes: Generate a sample set based on preset images in the image library; Training the first convolutional neural network using a sample set; Save the first convolutional neural network after training.

5. The image retrieval method as claimed in claim 3, wherein: Through the second convolutional neural network, the target in the input image is located, and before obtaining the positioning result, it also includes: Generate a sample set based on preset images in the image library; Training the second convolutional neural network using the sample set; Save the trained second convolutional neural network.

6. The image retrieval method as claimed in claim 3, wherein: The features of the local area image are extracted by the third convolutional neural network, and before obtaining the content features, the following steps are also included: Generate a sample set based on preset images in the image library; Training the third convolutional neural network using a sample set; Save the trained third convolutional neural network.

7. An image retrieval device, comprising: A global feature processing module is configured to extract global features of an input image and obtain a global feature code based on the global features; A content feature processing module, configured to extract content features of an input image and obtain a content feature code based on the content features; A composite feature code generation module is configured to concatenate the global feature code with the content feature code to obtain a composite feature code of the input image; A feature comparison module is configured to compare the composite feature code with a preset feature code in a preset feature code library to obtain an image retrieval result; Wherein, the feature comparison module includes: A Hamming distance calculation module is configured to calculate the Hamming distance between the composite feature code and each preset feature code in the preset feature code library; and An image output module, configured to determine an image retrieval result based on the Hamming distance; Wherein, the preset feature code library includes at least one preset feature code, and wherein: For a preset image in an image library, extracting a global feature of the preset image, and obtaining a global feature code of the preset image based on the global feature; Extracting content features of the preset image, and obtaining a content feature code of the preset image based on the content features; The global feature code of the preset image is concatenated with the content feature code of the preset image to obtain a composite feature code of the preset image, and the composite feature code is used as the preset feature code.

8. The image retrieval device according to claim 7, wherein: The global feature processing module comprises: A global feature extraction module is configured to extract features of an input image through a first convolutional neural network to obtain a global feature; The global feature code generation module is configured to convert the global feature into a binary feature code with a first preset number of bits to obtain a global feature code.

9. The image retrieval device according to claim 7, wherein: The content feature processing module includes: The target positioning module is configured to locate the target in the input image through a second convolutional neural network to obtain a positioning result; A local area image generation module is configured to crop the image based on the positioning result to obtain a local area image, wherein the local area image at least includes the target; a content feature extraction module, configured to extract content features from the local area image through a third convolutional neural network; The content feature code generation module is configured to convert the content feature into a binary feature code having a second preset number of bits to obtain a content feature code.

10. An image retrieval device, wherein the device comprises a processor and a memory, the memory comprising computer-readable instructions, and the computer-readable instructions, when executed by the processor, cause the image retrieval device to perform the method of any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the instructions are executed by a computer, the method described in any one of claims 1 to 6 is performed.

Images