Image retrieval method, device, equipment and storage medium

Abstract

The embodiments of the present application provide an image retrieval method, device, equipment and storage medium, which are applied to the fields of financial technology and artificial intelligence. The method includes: obtaining multiple target images, wherein the target images do not include target elements; extracting local features of each of the multiple target images; respectively screening out the cluster center features closest to each of the local features from a preset cluster feature set, and using the cluster center features closest to the local features to obtain the encoding of the local features; obtaining the target features according to the encoding of the local features of each target image, and using the target features to perform image retrieval. The present application can improve the accuracy of image retrieval. The present application involves blockchain technology, such as being able to write multiple target images into a blockchain or read multiple target images from a blockchain.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of intelligent decision-making technology, and in particular to an image retrieval method, device, equipment and storage medium. Background Art

[0002] Image retrieval technology based on deep learning usually uses global feature retrieval methods. Among them, global feature retrieval methods include NetVLAD, RMAC, GEM, etc. The global feature retrieval method mainly performs image retrieval by extracting the global features of the image. However, this global retrieval method has low retrieval accuracy for some images with severe background occlusion and highly similar backgrounds. Summary of the invention

[0003] The embodiments of the present application provide an image retrieval method, apparatus, device and storage medium, which can improve the accuracy of image retrieval, especially improve the retrieval accuracy of images with severe background occlusion and highly similar backgrounds.

[0004] In a first aspect, an embodiment of the present application provides an image retrieval method, comprising:

[0005] Acquire multiple target images, wherein the target images do not include target elements;

[0006] Extracting local features of each target image from the plurality of target images;

[0007] From a preset cluster feature set, respectively select the cluster center feature closest to each of the local features, and use the cluster center feature closest to the local feature to obtain the encoding of the local feature;

[0008] Target features are acquired according to the encoding of the local features of each target image, and image retrieval is performed using the target features.

[0009] Optionally, extracting a local feature of each target image from the multiple target images includes:

[0010] Inputting the multiple target images into the trained convolutional neural network;

[0011] Extracting features of each target image from the plurality of target images using a trained convolutional neural network to obtain a feature map corresponding to each target image;

[0012] The grid units included in the feature map corresponding to each target image are determined as local features of each target image.

[0013] Optionally, the method further includes:

[0014] Acquire multiple sample images, wherein the sample images do not include the target element;

[0015] Constructing a training set using the multiple sample images, the training set comprising multiple sample image pairs, the multiple sample image pairs comprising a first sample image pair and a second sample image pair, the first sample image pair comprising a first sample image and a second sample image with a similar background to the first sample image, the second sample image pair comprising the first sample image and a third sample image with a background that is not similar to the first sample image;

[0016] The training set is input into an initial convolutional neural network to train the initial convolutional neural network to obtain a trained convolutional neural network.

[0017] Optionally, obtaining the encoding of the local feature by using the cluster center feature closest to the local feature includes:

[0018] Obtaining a codeword of a cluster center feature that is closest to the local feature; the codeword is a codeword in a preset codebook, the codebook includes a plurality of codewords, and each codeword corresponds to a cluster center feature;

[0019] The local feature is encoded and represented by using a codeword of a cluster center feature that is closest to the local feature to obtain an encoding of the local feature.

[0020] Optionally, the method further includes:

[0021] Using a clustering algorithm to perform clustering processing on the training set to obtain multiple cluster center features;

[0022] The codebook is constructed according to the multiple cluster center features.

[0023] Optionally, obtaining the encoding of the local feature by using the cluster center feature closest to the local feature includes:

[0024] Obtaining a distance value between the local feature and a cluster center feature that is closest to the local feature;

[0025] The encoding of the local feature is represented by using the distance value between the local feature and the cluster center feature closest to the local feature to obtain the encoding of the local feature.

[0026] Optionally, acquiring the target feature according to the encoding of the local feature of each target image includes:

[0027] Inputting the encoding of the local features of each target image into the trained Transformer network;

[0028] The trained Transformer network is used to obtain target features based on the encoding of local features of each target image.

[0029] In a second aspect, an embodiment of the present application provides an image retrieval device, comprising:

[0030] An acquisition module, used for acquiring a plurality of target images, wherein the target images do not include target elements;

[0031] An extraction module, used for extracting local features of each target image in the plurality of target images;

[0032] A screening module, used to screen out the cluster center feature closest to each of the local features from a preset cluster feature set, and obtain the encoding of the local feature using the cluster center feature closest to the local feature;

[0033] The acquisition module is further used to obtain the encoding of the local feature using the cluster center feature closest to the local feature, and to obtain the target feature according to the encoding of the local feature of each target image;

[0034] The retrieval module is used to perform image retrieval using the target features.

[0035] In a third aspect, an embodiment of the present application provides an image retrieval device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store computer program instructions, and the processor is configured to execute the program instructions to implement the method described in the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed by a processor, they are used to execute the method described in the first aspect.

[0037] To sum up, the image retrieval device can extract the local features of each target image among multiple target images, and filter out the cluster center features closest to the local features from the preset cluster feature set, so as to obtain the encoding of the local feature using the cluster center features closest to the local feature, and obtain the target features according to the encoding of the local features of each target image, and use the target features to perform image retrieval. Compared with the image retrieval method based on global features in the prior art, the present application can improve the image retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flowchart of an image retrieval method provided in an embodiment of the present application;

[0040] Figure 2 is a flowchart of another image retrieval method provided in an embodiment of the present application;

[0041] Figure 3 is a structural schematic diagram of an image retrieval device provided in an embodiment of the present application;

[0042] Figure 4 It is a structural schematic diagram of an image retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0044] See also Figure 1 , is a flow chart of an image retrieval method provided in an embodiment of the present application. The image retrieval method can be applied to an image retrieval device, which can be a device such as a server or an intelligent terminal that can implement an image retrieval function. Specifically, the method can include the following steps:

[0045] S101. Acquire multiple target images, where the target images do not include target elements.

[0046] The target element may be a human image, an animal image, or an object image, etc. In one embodiment, the target element may be an image that blocks a background image. The target image may be a background image.

[0047] In one embodiment, the image retrieval device can obtain multiple target images collected by the target device. The target device can be an electronic device with a camera, or an electronic device with an image download function. The electronic device here can be, for example, a smart phone, a laptop computer, a desktop computer, or other intelligent terminal.

[0048] In one embodiment, the image retrieval device may obtain multiple images captured by the target device, and when there is a first image including the target element in the multiple images, the target element included in the first image is cut out to obtain the cut out first image as the target image. The image retrieval device may determine the cut out first image and the images that do not include the target element in the multiple images as the multiple target images. The first image may be an image including the target element in the multiple images. Cutting out the target element may reduce the adverse effect of the target element on the accuracy of image retrieval.

[0049] In one embodiment, the image retrieval device may obtain multiple images captured by the target device. When there is an image in which the first image includes the target element among the multiple images, a mask may be added to the target element included in the first image to obtain the target image. The image retrieval device may determine the first image with the mask added and the images in the multiple images that do not include the target element as multiple target images. Adding a mask to the target element can reduce the adverse effect of the target element on the accuracy of image retrieval.

[0050] It should be noted that, in addition to using cutout processing or adding a mask to make an image not include the target element, other image processing methods can also be used to make an image not include the target element, which are not listed here one by one.

[0051] In one embodiment, the image retrieval device can obtain a target video and obtain multiple consecutive frames of the target video. When there is a second image including a target element in the multiple frames, the target element included in the second image is cut out to obtain a cut-out second image. The image retrieval device can determine the cut-out second image and the images that do not include the target element in the multiple frames as multiple target images. The target video can be a live video, or can be a recorded video, etc. The second image can refer to an image that includes the target element in the multiple frames.

[0052] In one embodiment, the multiple frames of images may be continuous multiple frames of images within a specified time range.

[0053] In one embodiment, the image retrieval device can obtain a target video and obtain multiple consecutive frames of the target video. When there is an image in which the second image includes the target element in the multiple frames, a mask can be added to the target element included in the second image to obtain the second image with the mask added. The image retrieval device can determine the second image with the mask added and the image in which the target element is not included in the multiple frames as multiple target images.

[0054] In one embodiment, the image retrieval device may read the target video from a database.

[0055] In one embodiment, the image retrieval device may obtain a target video captured by a target device.

[0056] In one embodiment, the target video may be a video of a target category. For example, when the present application is used for financial fraud identification, the target category may be a finance-related category such as financial management, investment, stock, shopping, lending, etc. Here, in other application scenarios, the target category may also be other categories, which are not limited here.

[0057] In one embodiment, the target video may also be a video of the target user. For example, when the present application is used for financial fraud identification, the target user may be a user with a record of financial fraud or a user reported to be suspected of financial fraud; or the target user may also be a user in the supervision user list.

[0058] S102: Extract local features of each target image in the multiple target images.

[0059] Each target image may have one or more local features.

[0060] In one embodiment, the image retrieval device may extract local features of each target image from a plurality of target images through an image feature extraction model.

[0061] S103: Select the cluster center feature closest to each local feature from the preset cluster feature set, and obtain the encoding of the local feature using the cluster center feature closest to the local feature.

[0062] In an embodiment of the present application, the image retrieval device can filter out the cluster center feature closest to each local feature from a preset cluster feature set. The image retrieval device can obtain the encoding of the local feature using the cluster center feature closest to the local feature. Among them, the cluster feature set may include multiple cluster center features. The cluster center feature is the feature corresponding to the cluster center. Compared with the encoding method based on the word frequency histogram, the encoding method adopted in the embodiment of the present application can retain the details of the local features to a greater extent, avoiding the problem of excessive information attenuation caused by the encoding method based on the word frequency histogram.

[0063] In one embodiment, the image retrieval device can obtain a preset codebook. The codebook includes multiple codewords. Each codeword corresponds to a cluster center feature. The codeword can be used to index the corresponding cluster center feature. The cluster center feature refers to the feature corresponding to the cluster center. In one embodiment, the cluster center feature corresponding to each codeword can be different. Afterwards, the image retrieval device can query the cluster feature set according to the codebook. Among them, the multiple cluster center features included in the cluster feature set can be the cluster center features corresponding to the multiple codewords included in the codebook.

[0064] In one embodiment, the image retrieval device can obtain a large number of images that do not include the target element, and use a clustering algorithm to cluster these images to obtain multiple cluster center features, thereby constructing the codebook according to the multiple cluster center features. For example, these images that do not include the target element can be a target scene, such as a background image of a financial fraud scene. The reason why these images do not include the target element may be that these images originally do not include the target element, or the images originally include the target element, and after the images are cut out or a mask is added, the images do not include the target element.

[0065] Specifically, the method in which the image retrieval device obtains the encoding of the local feature by using the cluster center feature closest to the local feature may be one of the following:

[0066] ① The image retrieval device obtains the codeword of the cluster center feature closest to the local feature, and uses the codeword of the cluster center feature closest to the local feature to encode the local feature to obtain the encoding of the local feature.

[0067] For example, assume that the preset codebook is a codebook corresponding to 8 cluster center features, such as [1 2 3 4 5 6 78]. Among them, 1 is the codeword of the first cluster center feature, 2 is the codeword of the second cluster center feature, 3 is the codeword of the third cluster center feature, 4 is the codeword of the fourth cluster center feature, 5 is the codeword of the fifth cluster center feature, 6 is the codeword of the feature in the sixth cluster, 7 is the codeword of the seventh cluster center feature, and 8 is the codeword of the eighth cluster center feature. Assume that the dimension of each cluster center feature is 512. Image 1 has 4 512-dimensional local features, and the 4 local features are closest to the cluster center features corresponding to [2 3 4 5] in the codebook. Therefore, it can be determined that the code corresponding to local feature 1 of image 1 is 2, the code corresponding to local feature 2 of image 1 is 3, the code corresponding to local feature 3 of image 1 is 4, and the code corresponding to local feature 4 of image 1 is 5. Assuming that image 2 has four 512-dimensional local features, and the four local features are closest to the cluster center features corresponding to [4 56 7] in the codebook, it can be determined that the code corresponding to local feature 1 of image 2 is 4, the code corresponding to local feature 2 of image 2 is 5, the code corresponding to local feature 3 of image 2 is 6, and the code corresponding to local feature 4 of image 2 is 7.

[0068] ② The image retrieval device obtains the distance value between the local feature and the cluster center feature closest to the local feature, and uses the distance value between the local feature and the cluster center feature closest to the local feature to represent the encoding of the local feature, thereby obtaining the encoding of the local feature.

[0069] In one embodiment, the cluster center feature closest to the local feature can be determined in the following manner: the image retrieval device calculates the distance value between the local feature and the cluster center feature corresponding to each codeword for the local feature of each target image, and determines the cluster center feature closest to the distance center feature from the cluster center features corresponding to each codeword based on the distance value between the local feature and the cluster center feature corresponding to each codeword. The embodiment of the present application can calculate the distance value by Euclidean distance or cosine similarity.

[0070] In one embodiment, after obtaining the encoding of the local features of each target image, the image retrieval device can construct a coding sequence using the encoding of the local features of each target image, so as to subsequently obtain the target features using the coding sequence. The process of constructing the coding sequence is the process of combining the encodings of the local features. The image retrieval device can combine the encodings of the local features of each target image according to the positional relationship between the local features of each target image (the positional relationship can be determined based on the positional information of the local features in the corresponding feature map) to obtain the coding sequence. Here, the image retrieval device does not need to pay attention to the positional relationship between the target images in the process of constructing the coding sequence.

[0071] For example, suppose that the code corresponding to local feature 1 of image 1 is 2, the code corresponding to local feature 2 of image 1 is 3, the code corresponding to local feature 3 of image 1 is 4, and the code corresponding to local feature 4 of image 1 is 5. The code corresponding to local feature 1 of image 2 is 4, the code corresponding to local feature 2 of image 2 is 5, the code corresponding to local feature 3 of image 2 is 6, and the code corresponding to local feature 4 of image 2 is 7. The image retrieval device can combine these codes according to the positional relationship of each local feature of image 1 and the positional relationship of each local feature of image 2, and obtain the code sequence [2 3 4 5 45 6 7]. If each codeword in the codebook corresponds to a 512-dimensional cluster center feature, then this code sequence corresponds to 8 512-dimensional cluster center features.

[0072] In one embodiment, in order to preserve the positional relationship between local features and improve the accuracy of subsequent image retrieval, the image retrieval device will not aggregate local features of a target image that have the same cluster center feature. For example, assume that the cluster center feature closest to local feature 1 of image 1 is the cluster center feature corresponding to codeword 2, the cluster center feature closest to local feature 2 of image 1 is the cluster center feature corresponding to codeword 2, the cluster center feature closest to local feature 3 of image 1 is the cluster center feature corresponding to codeword 3, and the cluster center feature closest to local feature 4 of image 1 is the cluster center feature corresponding to codeword 3. Although the cluster center feature of local feature 1 of image 1 is the same as the cluster center feature of local feature 2, and the cluster center feature of local feature 3 of image 1 is the same as the cluster center feature of local feature 4 of image 1, considering the positional relationship between local features, the embodiment of the present application will not retain only one of the local features of a target image and ignore other local features just because some local features of the target image correspond to the same cluster center feature. That is to say, in the embodiment of the present application, the image retrieval device will retain all the local features of the target image even if several local features of the target image correspond to the same cluster center feature.

[0073] S104: Acquire target features according to the encoding of the local features of each target image, and perform image retrieval using the target features.

[0074] In the embodiment of the present application, the image retrieval device can fuse the encoding of the local features of each target image to obtain the target features, and use the target features to perform image retrieval. When a target image has multiple local features, the encoding of the local features of the target image can be the encoding of each of the multiple local features of the target image.

[0075] In one embodiment, the image retrieval device can obtain the target features of each image in the gallery during the process of performing image retrieval using the target features, calculate the similarity between the target features obtained according to the local feature coding of each target image and the target features of each image, and sort the images according to the calculated similarity to obtain the sorted images, and output part or all of the sorted images. Among them, the method for obtaining the target features of each image in the gallery can refer to the method for obtaining the target features according to multiple target images, that is, it can refer to steps S201-S205, which will not be repeated here. In one embodiment, the gallery can store multiple images that do not include the target elements mentioned above, such as background images of the target scene.

[0076] For example, the gallery includes image a, image b... The image retrieval device can obtain target feature a of image a, target feature b of image b... Assume that the target feature obtained according to the encoding of the local features of each target image is target feature 1. The image retrieval device can calculate the similarity between target feature a and target feature 1, and the similarity between target feature b and target feature 1... The image retrieval device can sort each image according to the calculated similarity to obtain sorted images, and then output part of the sorted images or all of the images. For example, each image can be sorted in order of similarity from high to low to obtain sorted images, and then the top N images are selected from the sorted images, and the top N images are output, where N is an integer greater than or equal to 1.

[0077] visible, Figure 1 In the illustrated embodiment, the image retrieval device can extract local features of each target image, select the cluster center features closest to each local feature from a preset cluster feature set, and use the cluster center features closest to the local features to obtain the encoding of the local features, thereby obtaining the target features based on the encoding of the local features of each target image, and using the target features to perform image retrieval, thereby improving the image retrieval accuracy.

[0078] See also Figure 2 , is a flow chart of another image retrieval method provided in an embodiment of the present application. The method can be applied to the aforementioned image retrieval device. Specifically, the method may include the following steps:

[0079] S201. Acquire multiple target images, where the target images do not include target elements.

[0080] Among them, step S201 can refer to Figure 1 Step S101 of the embodiment is not described in detail here.

[0081] S202: Input the multiple target images into the trained convolutional neural network.

[0082] S203, extracting features of each target image in the multiple target images through the trained convolutional neural network to obtain a feature map corresponding to each target image.

[0083] S204: Determine the grid units included in the feature map corresponding to each target image as local features of each target image.

[0084] In step S202-step S204, the image retrieval device can input multiple target images into the trained convolutional neural network, and perform feature extraction on each of the multiple target images through the trained convolutional neural network to obtain a feature map corresponding to each target image, thereby determining the grid units included in the feature map corresponding to each target image as local features of each target image. The convolutional neural network referred to can be ResNet, VGG or MobileNet, etc. For situations where there are many similar backgrounds, such as different in-vehicle environments, due to their high overall similarity, they can only be distinguished by local background differences in the background, and the embodiment of the present application can better extract local background features through the trained convolutional neural network.

[0085] For example, suppose the feature map corresponding to image 1 is a 1*512*7*7 feature map, where 1 refers to 1 feature map, 512 refers to the dimension of each grid unit in this feature map, and 7*7 refers to the length and width of this feature map. Each grid unit in this map is the local feature referred to in the embodiment of the present application. This means that the image retrieval device can obtain 49 512-dimensional local features through the convolutional neural network.

[0086] In one embodiment, after inputting multiple target images into a trained convolutional neural network, the image retrieval device can perform full convolution processing on each of the multiple target images through the trained convolutional neural network to obtain a feature map corresponding to each target image. Among them, full convolution processing is the process of processing through multiple convolutional layers and multiple pooling layers. Here, the connection between multiple pooling layers and multiple convolutional layers can be: at least one convolutional layer is connected to a pooling layer, and at least one convolutional layer is connected after the pooling layer, and so on. In one embodiment, the features output by the last convolutional layer of the trained convolutional neural network can be used as local features of the target image.

[0087] In one embodiment, the trained convolutional neural network can be obtained in the following manner: an image retrieval device obtains a plurality of sample images, wherein the sample images do not include target elements; the image retrieval device uses the plurality of sample images to construct a training set, wherein the training set includes a plurality of sample image pairs, wherein the plurality of sample image pairs include a first sample image pair and a second sample image pair, wherein the first sample image pair includes a first sample image and a second sample image having a background similar to the first sample image, and the second sample image pair includes a first sample image and a third sample image having a background different from the first sample image; the training set is input into an initial convolutional neural network to train the initial convolutional neural network to obtain a trained convolutional neural network. The sample images may be, for example, sample images of target scenes or sample images of financial fraud scenes. Among them, a sample image having a similarity with the first sample image greater than or equal to a preset similarity may be the second sample image, and a sample image having a similarity with the first sample image less than a preset similarity may be the third sample image.

[0088] In one embodiment, the image retrieval device uses the training set to train the initial convolutional neural network model, and the process of obtaining the trained convolutional neural network model is as follows: the initial convolutional neural network model is trained using the training set and the labels indicating whether the sample images are similar carried by each sample image pair in the training set to obtain the trained convolutional neural network model. In one embodiment, the image retrieval device can use the training set and the labels indicating whether the sample images are similar carried by each sample image pair in the training set, and combine the metric learning method to train the initial convolutional neural network model to obtain the trained convolutional neural network model. Metric learning can learn the similarity between two sample images. Through metric learning, the feature representation distance between positive examples (sample images with similar backgrounds) can be made closer, and the feature representation distance between negative examples (sample images with dissimilar backgrounds) can be made farther.

[0089] In one embodiment, the image library mentioned in the embodiment of the present application may include some images of the training set mentioned above.

[0090] In one embodiment, the image retrieval device may use a clustering algorithm, such as a K-means algorithm, to perform clustering processing on the aforementioned training set to obtain a plurality of cluster center features, and construct the codebook according to the plurality of cluster center features.

[0091] S205 , selecting the cluster center feature closest to each of the local features from the preset cluster feature set, and obtaining the encoding of the local feature using the cluster center feature closest to the local feature.

[0092] Among them, step S205 can refer to Figure 1Step S103 of the embodiment is not described in detail here.

[0093] S206: Input the encoding of the local features of each target image into the trained Transformer network.

[0094] S207, obtaining target features according to the encoding process of the local features of each target image through the trained Transformer network, and performing image retrieval using the target features.

[0095] Since the Transformer technology is used to aggregate the encoding of local features, the target features of the embodiment of the present application can be used to characterize the local features of each target image, the contextual relationship between the local features of each target image (the positional relationship between the local features of a single target image), the global features of each target image, and the contextual relationship between two target images (the repetition and difference relationship between the local features of two images). This solution is particularly suitable for image retrieval of scenes with moving backgrounds, and its retrieval process can make full use of multi-frame background information, and its retrieval effect is higher.

[0096] In one embodiment, for using Figure 1 For the encoding obtained by the first method in the embodiment, the image retrieval device can specifically obtain the cluster center features corresponding to the encoding of the local features of each target image according to the encoding of the local features of each target image, and input the cluster center features corresponding to the encoding of the local features of each target image into the trained Transformer network, so as to obtain the target features by processing the cluster center features corresponding to the encoding of each local feature using the trained Transformer network. In one embodiment, the image retrieval device can also input the cluster center features corresponding to the encoding of the local features of each target image and the position information of the local features in the corresponding feature map into the trained Transformer network, and obtain the target features by processing the cluster center features corresponding to the encoding of the local features of each target image and the position information of the local features in the corresponding feature map using the trained Transformer network.

[0097] In one embodiment, for using Figure 1For the encoding obtained by the second method in the embodiment, the image retrieval device can input the encoding of the local features of each target image into the trained Transformer network, and use the trained Transformer network to process the encoding of each local feature to obtain the target feature. In one embodiment, the image retrieval device can input the encoding of the local features of each target image and the position information of the local features in the corresponding feature map into the trained Transformer network, and use the trained Transformer network to process the encoding of the local features of each target image and the position information of the local features in the corresponding feature map to obtain the target feature.

[0098] In one embodiment, the trained Transformer network can be obtained in the following manner: the image retrieval device extracts the local features of each sample image in the training set and obtains a preset codebook, which includes multiple codewords; the image retrieval device determines the cluster center feature closest to the local feature from the cluster center features corresponding to each codeword for the local features of each sample image, and obtains the encoding of the local feature using the cluster center feature closest to the local feature, thereby training the initial Transformer network according to the encoding of the local features of each sample image to obtain the trained Transformer network. The Transformer network can use the same training set as the convolutional neural network, or a different training set. The embodiment of the present application encodes the local features of the sample image, such as the dimension corresponding to the last convolution layer of a vgg network is 1*512*7*7, then there are 49 512-dimensional local features; since the scenes corresponding to the image background are various, the local features extracted based on the image will also be more diverse. If the local features that have not been encoded by the codebook are directly input to the Transformer network, the training difficulty will be very large and a lot of training data will be required. Instead, all local features are clustered and encoded through a codebook, such as a codebook of size 65536. This can reduce the diversity of Transformer input data and thus reduce the difficulty of training the Transformer.

[0099] It can be seen that the traditional method of extracting SIFT features from images and performing feature clustering retrieval through bow or vlad focuses on the local features of the image and cannot make good use of the global context information of the image. The application embodiment can combine the clustering generation codebook in traditional image retrieval with the Transformer network for image retrieval, which can not only use the Transformer network to extract target features to improve retrieval accuracy, but also use the traditional codebook representation method to reduce the difficulty of training the Transformer network and the requirement for a large amount of training data.

[0100] In one embodiment, the present application relates to blockchain technology, such as obtaining multiple target images from the blockchain, or writing multiple images into the blockchain, or obtaining a target video from the blockchain, or writing a target video into the blockchain, and so on.

[0101] visible, Figure 2 In the illustrated embodiment, the image retrieval device can extract local features through a convolutional neural network, and can obtain target features based on the encoding of the local features through a Transformer network for image retrieval. This process reduces the number of types of local features input into the Transformer network through encoding, which can improve the processing efficiency of the Transformer network, and obtains the target features through the Transformer network, which can effectively improve the image retrieval accuracy. In addition, due to the use of Transformer technology, the number of input frames does not need to be limited to a fixed length, and has relatively high flexibility and configurability.

[0102] In an application scenario, since the methods and means of financial fraud such as online video loans vary, the image retrieval scheme described in the embodiment of the present application can be used in scenarios where multiple people commit fraud in the same location and the same scene, such as a large number of business application locations are concentrated and the video background is the same when applying. The image retrieval scheme described in the embodiment of the present application can be used to pick out such images or videos with the same background or high background similarity, and to identify potential fraud risks in a timely manner. If it relies on human eyes for recognition, there are obviously problems such as human eye fatigue and low efficiency, and it is difficult to use for large-scale data sets.

[0103] See also Figure 3 , is a schematic diagram of the structure of an image retrieval device provided in an embodiment of the present application. The device can be applied to the image retrieval device mentioned above. Specifically, the image retrieval device may include:

[0104] The acquisition module 301 is used to acquire multiple target images, where the target images do not include target elements.

[0105] The extraction module 302 is used to extract local features of each target image in the multiple target images.

[0106] The screening module 303 is used to screen out the cluster center feature closest to each of the local features from the preset cluster feature set, and obtain the code of the local feature using the cluster center feature closest to the local feature.

[0107] The acquisition module 301 is further configured to obtain the encoding of the local feature using the cluster center feature closest to the local feature, and to obtain the target feature according to the encoding of the local feature of each target image.

[0108] The retrieval module 304 is used to perform image retrieval using the target features.

[0109] In an optional implementation, the extraction module 302 is specifically configured to:

[0110] Inputting the multiple target images into the trained convolutional neural network;

[0111] Extracting features of each target image from the plurality of target images using a trained convolutional neural network to obtain a feature map corresponding to each target image;

[0112] The grid units included in the feature map corresponding to each target image are determined as local features of each target image.

[0113] In an optional implementation, the image retrieval state further includes a training module 305 .

[0114] In an optional embodiment, the training module 305 is used to obtain multiple sample images, wherein the sample images do not include target elements; construct a training set using the multiple sample images, wherein the training set includes multiple sample image pairs, wherein the multiple sample image pairs include a first sample image pair and a second sample image pair, wherein the first sample image pair includes a first sample image and a second sample image having a background similar to the first sample image, and the second sample image pair includes a first sample image and a third sample image having a background not similar to the first sample image; and input the training set into an initial convolutional neural network to train the initial convolutional neural network to obtain a trained convolutional neural network.

[0115] In an optional implementation, the acquisition module 301 is specifically configured to:

[0116] Obtaining a codeword of a cluster center feature that is closest to the local feature; the codeword is a codeword in a preset codebook, the codebook includes a plurality of codewords, and each codeword corresponds to a cluster center feature;

[0117] The local feature is encoded and represented by using a codeword of a cluster center feature that is closest to the local feature to obtain an encoding of the local feature.

[0118] In an optional implementation, the image acquisition device further includes a codebook construction module 306.

[0119] In an optional implementation, the codebook construction module 306 is configured to perform clustering processing on the training set using a clustering algorithm to obtain a plurality of cluster center features; and construct the codebook according to the plurality of cluster center features.

[0120] In an optional implementation, the acquisition module 301 is specifically configured to:

[0121] Obtaining a distance value between the local feature and a cluster center feature that is closest to the local feature;

[0122] The encoding of the local feature is represented by using the distance value between the local feature and the cluster center feature closest to the local feature to obtain the encoding of the local feature.

[0123] In an optional implementation, the acquisition module 301 is specifically configured to:

[0124] Inputting the encoding of the local features of each target image into the trained Transformer network;

[0125] The trained Transformer network is used to obtain target features based on the encoding of local features of each target image.

[0126] visible, Figure 3 In the illustrated embodiment, the image retrieval device can extract local features of each target image, select the cluster center features closest to each local feature from a preset cluster feature set, and use the cluster center features closest to the local features to obtain the encoding of the local features, thereby obtaining the target features based on the encoding of the local features of each target image, and using the target features to perform image retrieval, thereby improving the image retrieval accuracy.

[0127] See also Figure 4 , is a schematic diagram of the structure of an image retrieval device provided in an embodiment of the present application. The image retrieval device described in this embodiment may include: one or more processors 1000 and a memory 2000. The processor 1000 and the memory 2000 may be connected via a bus or the like.

[0128] The processor 1000 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0129] The memory 2000 may be a high-speed RAM memory or a non-volatile memory, such as a disk memory. The memory 2000 is used to store a set of program codes, and the processor 1000 may call the program codes stored in the memory 2000. Specifically:

[0130] Processor 1000 is used to obtain multiple target images, wherein the target images do not include target elements; extract local features of each of the multiple target images; select cluster center features closest to each of the local features from a preset cluster feature set, and obtain the encoding of the local features using the cluster center features closest to the local features; obtain target features based on the encoding of the local features of each target image, and perform image retrieval using the target features.

[0131] In one embodiment, when extracting the local features of each target image in the plurality of target images, the processor 1000 is specifically configured to:

[0132] Inputting the multiple target images into the trained convolutional neural network;

[0133] Extracting features of each target image from the plurality of target images using a trained convolutional neural network to obtain a feature map corresponding to each target image;

[0134] The grid units included in the feature map corresponding to each target image are determined as local features of each target image.

[0135] In one embodiment, the processor 1000 is further configured to:

[0136] Acquire multiple sample images, wherein the sample images do not include the target element;

[0137] Constructing a training set using the multiple sample images, the training set comprising multiple sample image pairs, the multiple sample image pairs comprising a first sample image pair and a second sample image pair, the first sample image pair comprising a first sample image and a second sample image with a similar background to the first sample image, the second sample image pair comprising the first sample image and a third sample image with a background that is not similar to the first sample image;

[0138] The training set is input into an initial convolutional neural network to train the initial convolutional neural network to obtain a trained convolutional neural network.

[0139] In one embodiment, when obtaining the encoding of the local feature by using the cluster center feature closest to the local feature, the processor 1000 is specifically configured to:

[0140] Obtaining a codeword of a cluster center feature that is closest to the local feature; the codeword is a codeword in a preset codebook, the codebook includes a plurality of codewords, and each codeword corresponds to a cluster center feature;

[0141] The local feature is encoded and represented by using a codeword of a cluster center feature that is closest to the local feature to obtain an encoding of the local feature.

[0142] In one embodiment, the processor 1000 is further configured to:

[0143] Using a clustering algorithm to perform clustering processing on the training set to obtain multiple cluster center features;

[0144] The codebook is constructed according to the multiple cluster center features.

[0145] In one embodiment, when obtaining the encoding of the local feature by using the cluster center feature closest to the local feature, the processor 1000 is specifically configured to:

[0146] Obtaining a distance value between the local feature and a cluster center feature that is closest to the local feature;

[0147] The encoding of the local feature is represented by using the distance value between the local feature and the cluster center feature closest to the local feature to obtain the encoding of the local feature.

[0148] In one embodiment, when the target feature is acquired according to the encoding of the local feature of each target image, the processor 1000 is specifically configured to:

[0149] Inputting the encoding of the local features of each target image into the trained Transformer network;

[0150] The trained Transformer network is used to obtain target features based on the encoding of local features of each target image.

[0151] In a specific implementation, the processor 1000 described in the embodiment of the present application may execute Figure 1 Example Figure 2 The implementation method described in the embodiment can also execute the implementation method described in the embodiment of the present application, which will not be repeated here.

[0152] Each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of sampling hardware or sampling software functional modules.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the computer-readable storage medium can be volatile or non-volatile. For example, the computer storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The computer-readable storage medium can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, at least one application required for a function, etc.; the storage data area can store data created according to the use of the blockchain node, etc.

[0154] Among them, the blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0155] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiment and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image retrieval method, characterized in that: include: Acquire multiple target images, wherein the target images do not include target elements; the target image is a background image, and the target element is the image content that blocks the background image; the method for acquiring the target images is: acquire a target video, and acquire multiple consecutive frames of images of the target video within a specified time range; if there is a second image including the target element in the multiple frames, perform a cutout process on the target element included in the second image, and determine the cutout second image and the image that does not include the target element in the multiple frames as multiple target images, or add a mask to the target element included in the second image, and determine the masked second image and the image that does not include the target element in the multiple frames as multiple target images; the category of the target video is determined based on the current image retrieval scenario, and the target video is a video of a target user, and the target user is a marked user in the current image retrieval scenario; if the current image retrieval scenario is a financial fraud identification scenario, the category of the target video is a financial-related category, and the target user is a user with a record of financial fraud or a user reported to be suspected of financial fraud, or the target user is a user in the list of regulated users; Extracting local features of each target image from the plurality of target images; the local features of the target image are determined based on a convolutional neural network, and the convolutional neural network is trained by combining sample images in the current retrieval scenario and a metric learning method; From a preset cluster feature set, the cluster center feature closest to each of the local features is respectively screened out, and a codeword of the cluster center feature closest to the local feature is obtained; the codeword is a codeword in a preset codebook, the codebook includes a plurality of codewords, and each codeword corresponds to a cluster center feature; the local feature is encoded and represented by using the codeword of the cluster center feature closest to the local feature to obtain an encoding of the local feature; The encoding of the local features of each target image is input into the trained Transformer network, and the target features are obtained according to the encoding of the local features of each target image through the trained Transformer network, and the target features are used for image retrieval; wherein the target features are used to characterize the local features of each target image, the contextual relationship between the local features of each target image, the global features of each target image, and the contextual relationship between any two target images; the trained Transformer network is obtained in the following manner: the local features of each sample image in the training set are extracted, and a preset codebook is obtained, the codebook including multiple codewords; for the local features of each sample image, the cluster center feature closest to the local feature is determined from the cluster center features corresponding to each codeword, and the encoding of the local feature is obtained using the cluster center feature closest to the local feature, so as to train the initial Transformer network according to the encoding of the local features of each sample image to obtain the trained Transformer network.

2. The method according to claim 1, characterized in that The extracting the local features of each target image from the plurality of target images comprises: Inputting the multiple target images into the trained convolutional neural network; Extracting features of each target image from the plurality of target images using a trained convolutional neural network to obtain a feature map corresponding to each target image; The grid units included in the feature map corresponding to each target image are determined as local features of each target image.

3. The method according to claim 2, characterized in that The method further comprises: Acquire multiple sample images, wherein the sample images do not include the target element; Constructing a training set using the multiple sample images, the training set comprising multiple sample image pairs, the multiple sample image pairs comprising a first sample image pair and a second sample image pair, the first sample image pair comprising a first sample image and a second sample image with a similar background to the first sample image, the second sample image pair comprising the first sample image and a third sample image with a background that is not similar to the first sample image; The training set is input into an initial convolutional neural network to train the initial convolutional neural network to obtain a trained convolutional neural network.

4. The method according to claim 1, characterized in that: The method further comprises: Using a clustering algorithm to perform clustering processing on the training set to obtain multiple cluster center features; The codebook is constructed according to the multiple cluster center features.

5. An image retrieval device, characterized in that: include: An acquisition module is used to acquire multiple target images, wherein the target image does not include a target element; the target image is a background image, and the target element is an image content that blocks the background image; the target image is acquired by: acquiring a target video, and acquiring multiple consecutive frames of images of the target video within a specified time range; if there is a second image including the target element in the multiple frames, the target element included in the second image is cut out, and the cut-out second image and the image that does not include the target element in the multiple frames are determined as multiple target images, or a mask is added to the target element included in the second image, and the masked second image and the image that does not include the target element in the multiple frames are determined as multiple target images; the category of the target video is determined based on the current image retrieval scenario, and the target video is a video of a target user, and the target user is a marked user in the current image retrieval scenario; if the current image retrieval scenario is a financial fraud identification scenario, the category of the target video is a financial-related category, and the target user is a user with a financial fraud record or a user reported to be suspected of financial fraud, or the target user is a user in a list of regulated users; An extraction module, used for extracting local features of each target image in the plurality of target images; the local features of the target image are determined based on a convolutional neural network, and the convolutional neural network is trained by combining sample images in the current retrieval scenario and a metric learning method; A screening module is used to screen out the cluster center feature closest to each of the local features from a preset cluster feature set, and obtain a codeword of the cluster center feature closest to the local feature; the codeword is a codeword in a preset codebook, the codebook includes multiple codewords, and each codeword corresponds to a cluster center feature; the local feature is encoded using the codeword of the cluster center feature closest to the local feature to obtain the encoding of the local feature; The acquisition module is also used to input the encoding of the local features of each target image into the trained Transformer network, and obtain the target features according to the encoding processing of the local features of each target image through the trained Transformer network; wherein the target features are used to characterize the local features of each target image, the contextual relationship between the local features of each target image, the global features of each target image, and the contextual relationship between two target images; the trained Transformer network is obtained by: extracting the local features of each sample image in the training set, and obtaining a preset code book, which includes multiple code words; for the local features of each sample image, determining the cluster center feature closest to the local feature from the cluster center features corresponding to each code word, and obtaining the encoding of the local feature using the cluster center feature closest to the local feature, so as to train the initial Transformer network according to the encoding of the local features of each sample image to obtain the trained Transformer network; The retrieval module is used to perform image retrieval using the target features.

6. An image retrieval device, characterized in that: The invention comprises a processor and a memory, wherein the processor and the memory are connected to each other, wherein the memory is used to store computer program instructions, and the processor is configured to execute the program instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, they are used to execute the method according to any one of claims 1 to 4.

Images