Image processing method, image matching method, device and storage medium

Abstract

The present application discloses an image processing method, an image matching method, an image data processing device, an image matching device and a computer-readable storage medium. The image processing method comprises: determining an image feature of an input image; determining a plurality of local image features of the image feature; determining a plurality of local feature vectors corresponding to the plurality of local image features respectively; and determining a hash code of the input image based on the plurality of local feature vectors.

Description

Technical Field

[0001] The present disclosure relates to the field of image data processing, and in particular to an image processing method, an image matching method, a device, and a storage medium. Background Art

[0002] Image hashing technology can help with image search and matching. A good image hash code should have a locality-sensitive property, that is, the hash codes of similar images are also similar. Traditional image hashing technologies with locality-sensitive properties are divided into two types: data-independent random hashing algorithms, and hashing algorithms based on data and machine learning techniques. The current mainstream method is to generate a binary vector of hash codes for each image. Each bit in such a hash code condenses the information of the entire image. However, this hash encoding method is extremely disadvantageous for processing partial matching of images. Summary of the invention

[0003] According to one aspect of the present disclosure, an image processing method is proposed, comprising: determining image features of an input image; determining multiple local image features of the image features; determining multiple local feature vectors corresponding to the multiple local image features respectively; and determining a hash code of the input image based on the multiple local feature vectors.

[0004] In some embodiments, determining multiple local image features of the image feature includes: segmenting the image feature according to a preset size to determine multiple local areas of the image feature; and determining information of the image features corresponding to each local area of ​​the multiple local areas as local image features corresponding to the local area.

[0005] In some embodiments, the image features include feature maps of multiple channels, and the local image features corresponding to each local area include channel local features of the local area corresponding to the feature map of each channel in the feature maps of the multiple channels.

[0006] In some embodiments, determining multiple local feature vectors corresponding to the multiple local image features respectively includes: for each local image feature, pooling the channel local features of the corresponding local area in the feature map of each channel to determine the local feature vector corresponding to the local image feature.

[0007] In some embodiments, determining the hash code of the input image based on the multiple local feature vectors includes: mapping the multiple local feature vectors into multiple sub-hash codes of the same length respectively; and combining the multiple sub-hash codes to obtain the hash code of the input image.

[0008] In some embodiments, combining the multiple sub-hash codes to obtain the hash code of the input image includes: arranging the multiple sub-hash codes according to a spatial arrangement of the local regions corresponding to each sub-hash code to obtain the hash code of the input image.

[0009] In some embodiments, the hash code includes multiple layers, wherein the i-th layer in the hash code includes the i-th hash bit of each sub-hash code, and the i-th layer is a matrix formed by arranging the i-th hash bit of each sub-hash code according to the spatial arrangement of the local area corresponding to each sub-hash code, wherein i is an integer greater than or equal to 1 and less than or equal to K, and K is the length of each sub-hash code.

[0010] According to another aspect of the present application, an image matching method is provided for matching a first image and a second image, wherein the first image is represented as a first hash code, the first hash code includes multiple first layers, each first layer is a first matrix formed by multiple first hash bits, and the second image is represented as a second hash code, the second hash code includes multiple second layers, each second layer is a second matrix formed by multiple second hash bits, and wherein the number of first layers included in the first hash code and the number of second layers included in the second hash code are the same, and the sizes of the first matrix and the second matrix are the same, and the image matching method includes: determining the Hamming distance between the first hash code and the second hash code to determine the degree of matching between the first image and the second image.

[0011] In some embodiments, determining the Hamming distance between the first hash code and the second hash code includes: determining multiple layer Hamming distances using multiple first layers of the first hash code and multiple second layers of the corresponding second hash code, respectively; and determining the sum of the multiple layer Hamming distances as the Hamming distance between the first hash code and the second hash code.

[0012] In some embodiments, the size of the first matrix is ​​the same as the size of the second matrix, and determining the Hamming distance between each first layer in the first hash code and the second layer at the corresponding position in the second hash code includes: for each first matrix, dividing the first matrix with a predetermined matching size to determine a plurality of first local regions in the first matrix, for each second matrix, dividing the second matrix with the predetermined matching size to determine a plurality of second local regions in the second matrix, for each first local region, respectively determining a plurality of local Hamming distances between the first local region and each of the plurality of second local regions; and determining the Hamming distance between the first layer and the corresponding second layer according to the plurality of local Hamming distances.

[0013] According to another aspect of the present application, an image data processing device is provided, the device comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the processor executes the image processing method as described above.

[0014] According to another aspect of the present application, an image matching device is provided, the device comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the processor executes the image matching method as described above.

[0015] According to another aspect of the present application, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor executes the method as described above.

[0016] By using the image processing method and image matching method provided by the present disclosure, a hash code containing image spatial information can be generated for an image, and the spatial information of the hash code can be used to perform matching between images, thereby improving the accuracy of retrieval and matching of images for local matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. The following drawings are not deliberately scaled according to the actual size, and the focus is on illustrating the main purpose of the present disclosure.

[0018] Figure 1 A hashing algorithm based on locality-sensitive performance is demonstrated;

[0019] Figure 2A A model using a deep neural network as the main component of a hash function according to an embodiment of the present application is shown;

[0020] Figure 2B An example of a hash code generation process according to the present application is shown;

[0021] Figure 2C A schematic diagram showing two images having local similarities;

[0022] Figure 3 A schematic flow chart showing a method for processing image data according to an embodiment of the present application;

[0023] Figure 4A and Figure 4BA schematic diagram showing a method of determining local image features according to an embodiment of the present application is shown;

[0024] Figure 5A The schematic process of determining a hash code according to an embodiment of the present application is shown;

[0025] Figure 5B A schematic diagram showing the arrangement of multiple sub-hash codes according to an embodiment of the present application is shown;

[0026] Figure 6 A schematic flow chart of an image matching method according to an embodiment of the present application is shown;

[0027] Fig. 7A and Figure 7B A schematic diagram showing a process of determining a layer Hamming distance between a first layer in a first hash code and a second layer at a corresponding position in a second hash code according to an embodiment of the present application

[0028] Figure 8 A schematic block diagram of an image data processing device 800 according to an embodiment of the present application is shown;

[0029] Fig. 9 A schematic block diagram of an image matching device 900 according to an embodiment of the present application is shown;

[0030] Fig.10 An exemplary scene diagram showing an image data processing system according to an embodiment of the present application; and

[0031] Fig.11 A schematic diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Unless otherwise defined, the technical terms or scientific terms used herein shall have the usual meanings understood by persons of ordinary skill in the field to which the invention belongs. The words "first", "second" and similar terms used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0034] "Up", "down", "left", "right", etc. are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0035] With the advancement of computer vision technology, the expression of image visual features has become more refined. Unlike traditional features such as histogram of oriented gradients (HOG), the image features currently used to represent image information usually have higher dimensions. For example, the image features obtained based on FisherVector encoding are usually tens of thousands of dimensions. For another example, before passing through the final pooling layer, the image features obtained using deep neural networks (such as ResNet) usually have tens of thousands of dimensions. This means that greater storage complexity is required to represent a single image. For tasks such as image search, when processing large-scale image data sets, the cost of storing the visual features of the entire data set is very high, and it is difficult to read into the memory of a single server. During the operation of the algorithm, data input and output between the memory and the external storage must be repeated, increasing the response time of the algorithm.

[0036] Therefore, in the process of performing retrieval and matching operations on an image, the retrieval and matching operations can be conveniently performed by representing the image as a hash code, wherein the hash code can condense complex image information into simple data information with a predefined structure, such as binary data information.

[0037] Figure 1 A hash algorithm based on local sensitivity is presented. The hash algorithm associates an image with a binary hash code (or a string represented by any other characters) of a predetermined length through multiple hash functions. The Hamming distance between the hash codes can be used to represent the similarity between the associated images. For example, for two images with high similarity (such as similarity based on semantic similarity or similarity based on visual similarity), the Hamming distance between the hash codes of the two images is also small.

[0038] The Hamming distance used in the content disclosed in this application represents the number of positions where the symbols at corresponding positions of two strings of equal length (such as hash codes represented by 0, 1 or strings represented by any other characters) are different. For hash codes, when determining the Hamming distance between two hash codes of equal length, the symbols at each corresponding position in the two hash codes can be compared. If the symbols of the elements at the corresponding positions of the two hash codes are different (for example, one is 0 and the other is 1), then the Hamming distance between the two hash codes can be increased by one. By comparing the symbols of each corresponding position of the two hash codes in the above manner, the final Hamming distance between the two hash codes can be determined.

[0039] It is understandable that, without departing from the principles of the present disclosure, the Hamming distance used here can also be modified as long as it can reflect the similarity between the two hash codes. For example, the similarity between the two hash codes can also be represented by the number of positions with the same symbol at the corresponding positions of the two hash codes. In this case, the more bits with the same symbol, the higher the similarity of the two hash codes, that is, the higher the similarity between the contents represented by the two hash codes.

[0040] like Figure 1 As shown, the hash code of the picture "The Great Wall" can be expressed as "000101111010", the hash code of the picture "Forbidden City 1" can be expressed as "010011010110", and the hash code of the picture "Forbidden City 2" can be expressed as "110001010110". When determining the Hamming distance between the hash codes of the picture "The Great Wall" and the picture "Forbidden City 1", the Hamming distance between the two hash codes can be determined by calculating the number of positions with different symbols in the corresponding positions between the hash code "000101111010" and the hash code "010011010110". Figure 1 As shown, the signs of the 2nd, 4th, 5th, 7th, 9th, and 10th bits are different between the hash codes of the picture "The Great Wall" and the picture "Forbidden City 1", so the Hamming distance between the hash codes of the picture "The Great Wall" and the picture "Forbidden City 1" is 6. Similarly, it can be determined that the signs of the 1st and 5th bits are different between the picture "Forbidden City 1" and the picture "Forbidden City 2", that is, the Hamming distance between the picture "Forbidden City 1" and the picture "Forbidden City 2" is 2. Therefore, the similarity between the hash code of the picture "Forbidden City 1" and the hash code of the picture "Forbidden City 2" is higher, which means that the pictures "Forbidden City 1" and "Forbidden City 2" are closer.

[0041] The similarities between "Forbidden City 2" are even higher.

[0042] There are two common hashing algorithms: data-independent and data-driven. Data-independent refers to a hash function generated based on randomness. Data-driven refers to a hash code generated based on machine learning methods and a large amount of training data.

[0043] Figure 2A A model using a deep neural network as the main component of a hash function according to an embodiment of the present application is shown. Figure 2A As shown, a deep neural network can be used to extract the deep features of an image and a hash code for the deep features can be generated using a hash function layer. It is understandable that the hash function layer can be trained using a preset training data set.

[0044] The specific method of generating hash codes is not limited in the embodiments provided in this application. That is, a data-independent scheme can be used to generate hash codes for images, a data-driven scheme can be used to generate hash codes for images, and any other possible data processing method can be used to generate hash codes for images. As long as the generated hash codes can represent the image information in the image.

[0045] Figure 2B An example of a hash code generation process according to the present application is shown. Figure 2B As shown, a feature map of the input image can be determined. In some embodiments, the feature map of the input image may include at least one channel. The feature map of each channel in the at least one channel can be processed using a global average pooling method to obtain a feature vector representing the input image. Then, the feature vector can be processed using any of the aforementioned hash code generation methods to obtain a hash code representing the input image.

[0046] Understandably, using Figure 2B Each hash bit in the hash code generated by the process in contains the global information representing the image. Among them, the spatial information of the image (corresponding to the spatial information contained in the image) is erased by operations such as pooling. Therefore, if there is similar information with partial offset in two images, that is, the same or similar images are contained in different local areas, using Figure 2B The hash codes generated by the process shown in may not be able to express such local similarities. Figure 2B The image hash codes generated by the process in cannot reasonably cope with image search tasks with such partial offset characteristics. In other words, if there are locally similar areas in two images, using Figure 2B The hash codes generated in the illustrated method cannot represent the similarity of such image information.

[0047] Figure 2C A schematic diagram showing two images with local similarity is shown. Figure 2C The lower left corner of the left picture shows an image of the Forbidden City. Figure 2C The right picture shows a subway map, and the location of the Forbidden City is marked in the middle of the subway map. Figure 2C The local area in the right figure contains Figure 2C The local image in the lower left corner of the left image, but Figure 2C The global features of the two images shown in may be quite different, so Figure 2B The method shown in may not be recognized Figure 2C The similarity between two images in .

[0048] The image processing method provided by the present application is described below in conjunction with the embodiments provided by the present application.

[0049] Figure 3 A schematic flow chart of an image processing method according to an embodiment of the present application is shown.

[0050] like Figure 3 As shown, in step S302, image features of the input image may be determined.

[0051] In some embodiments, the image features of the input image may be extracted using a trained neural network. For example, a convolutional neural network may be used to process the input image to obtain the image features of the input image.

[0052] In other embodiments, the image features of the input image can be determined by calculating, for example, a directional gradient histogram of the image. The specific form of the image features of the input image is not limited here. In fact, those skilled in the art can use any possible method to achieve the determination of the image features for the input image.

[0053] In some embodiments, the image features of the input image may be represented in the form of a feature map, wherein the feature map may be represented in the form of a two-dimensional matrix. In some implementations, the image features of the input image may include feature maps of multiple channels, wherein the size of the feature map of each channel is the same.

[0054] In step S302, a plurality of local image features of the image feature may be determined.

[0055] In some embodiments, the image feature may be segmented at a preset size to determine a plurality of local regions of the image feature having a preset size. Then, information of the image feature corresponding to each local region of the plurality of local regions may be determined as a local image feature corresponding to the local region.

[0056] It can be understood that when the image features of the input image are composed of feature maps of multiple channels, the above-mentioned segmentation of the image features at a preset size refers to segmenting the feature map of each channel at a preset size.

[0057] Figure 4A and Figure 4B A schematic diagram of determining local image features according to an embodiment of the present application is shown.

[0058] like Figure 4A As shown, in one implementation, the image feature may be segmented into a plurality of non-overlapping local areas with a preset size. Figure 4A The principle of the present disclosure is described in the example that the size of the local area is 1 / 16 of the size of the feature map. However, it is understandable that those skilled in the art can set the size of the local area according to actual needs. For example, the size of the local area can also be any ratio of 1 / 4, 1 / 5, 1 / 6 of the size of the feature map.

[0059] like Figure 4B As shown, in another implementation, the image features may be segmented at a preset size and multiple partially overlapping local areas in the image features may be determined. Figure 4B As shown, the size of the local region is 4 / 9 of the feature map size. Figure 4B The local areas 1, 2, 3 and 4 shown in FIG. 1 have the same size, and there is an overlapping portion between any two local areas.

[0060] Return to reference Figure 3 ,use Figure 4A , Figure 4B The method shown in the figure can segment the image feature and determine multiple local image features of the image feature. When the image feature is represented as a feature map of multiple channels, the local image feature corresponding to each local area includes the channel local feature of the corresponding local area of ​​the feature map of each channel in the feature map of multiple channels of the image. That is, each local image feature also includes the local feature maps of multiple channels.

[0061] In step S306, a plurality of local feature vectors respectively corresponding to the plurality of local image features may be determined.

[0062] In some embodiments, for each local image feature, the channel local features of the corresponding local area in the feature map of each channel in the local image feature may be pooled to determine a local feature vector corresponding to the local image feature. The pooling described herein may include one or more of maximum pooling, minimum pooling, and average pooling.

[0063] In the case where the image feature includes feature maps of multiple channels, the local feature map of each channel can be pooled separately, and the pooling result of each channel is used as an element of the local feature vector. The number of elements of the local feature vector obtained in this way is the same as the number of channels of the feature map of the image feature.

[0064] In step S308, a hash code of the input image may be determined based on the multiple local feature vectors.

[0065] In some embodiments, the plurality of local feature vectors may be mapped into a plurality of sub-hash codes of the same length, respectively. For example, each local feature vector may be mapped into a sub-hash code of a predetermined length, such as a K-bit hash code, according to a predefined hash function. K is an integer greater than 1, and the value of K depends on the configuration of the specific hash function used.

[0066] The hash code of the input image is obtained by combining the multiple sub-hash codes. In one implementation, the multiple sub-hash codes may be arranged according to a spatial arrangement of corresponding local regions to obtain the hash code of the image.

[0067] Figure 5A FIG. 2 shows an exemplary process of determining a hash code according to an embodiment of the present application. Figure 5A As shown, feature extraction can be performed on the input image to determine image features representing image information of the input image, wherein the image features include feature maps of multiple channels. Then, the image features can be segmented at a predetermined size to obtain 16 local image features corresponding to 16 local regions. The 16 local regions described here can overlap or not overlap each other.

[0068] By pooling the local feature maps of each channel of the 16 local image features, 16 local feature vectors for the input image can be obtained.

[0069] Then, the 16 local feature vectors are processed separately in any of the above-mentioned ways to obtain a sub-hash code corresponding to each local feature vector, wherein each sub-hash code includes K hash bits, where K is a predefined positive integer.

[0070] The hash code of the image is obtained by arranging the multiple sub-hash codes according to the spatial arrangement of the corresponding local regions. Therefore, the hash code obtained by the method provided by the embodiment of the present application includes multiple layers, each of which is a matrix formed by multiple hash bits. Figure 5A As shown, the hash code finally generated includes K layers. For the i-th layer (where i is an integer greater than or equal to 1 and less than or equal to K) matrix, the matrix is ​​formed by arranging the i-th element of each sub-hash code, and the elements of each sub-hash code are arranged in accordance with the spatial arrangement of the local area corresponding to the sub-hash code.

[0071] Figure 5B FIG. 4 shows a schematic diagram of arranging multiple sub-hash codes according to an embodiment of the present application. Figure 5B As shown, it can be Figure 5B The local feature vector in the circle on the left is mapped into a hash code with K elements, where Figure 5B The local eigenvectors in the ellipse in the left figure can correspond to Figure 5B The hash bits with a gray background in the right figure of . It can be seen that by arranging the multiple sub-hash codes according to the spatial arrangement of the local regions corresponding to the local feature vectors, a hash code with multiple layers can be obtained, where the number of layers and the length of each sub-hash code are the same. Each layer may include hash bits of all sub-hash codes at corresponding positions. For example, for the i-th layer, the i-th layer hash code may be composed of the i-th element of each sub-hash code, where 1≤i≤K.

[0072] use Figure 5B In the manner shown in , the hash code of the input image can be determined by combining sub-hash codes determined according to the local feature vectors, wherein each sub-hash code is arranged according to the spatial position of the local region corresponding to the local feature vector, so that the final hash code can retain the information of the local image in the image and the spatial relationship between the local images at different positions.

[0073] The method provided by the present application can be used to obtain a hash code for representing input image information, such a hash code includes multiple layers, each layer is a two-dimensional matrix formed by multiple hash bits. One hash bit in each layer can be used to represent the image information of a local area of ​​the input image, and the hash bits of each layer can retain the spatial information of the input image. When two images have similar local images located at different positions, the hash bits of the local area in each layer of the hash code generated by the above method can reflect the similarity of the local images at the corresponding positions.

[0074] By using the image processing method provided in the present application, a higher-dimensional hash code for representing image information can be generated, thereby retaining the spatial domain information of the image in the generated hash code to improve the expression ability of the hash code for the local image.

[0075] Figure 6 A schematic flow chart of the image matching method provided in the present application is shown. Figure 6 The method 600 shown in FIG. 6 can be used to match a first image and a second image.

[0076] In step S602, you can use Figure 3 The method shown processes a first image to determine a first hash code for representing the first image, wherein the first hash code comprises a plurality of first layers, each of the first layers being a first matrix formed by a plurality of first hash bits.

[0077] In step S604, you can use Figure 3The method shown processes the second image to determine a second hash code for representing the second image. The second hash code includes a plurality of second layers, each second layer being a second matrix formed by a plurality of second hash bits. As described above, the first hash code and the second hash code have the same number of layers. And each first layer and the corresponding second layer have the same number of hash bits. In some embodiments, each first layer and the corresponding second layer can be matrices of the same size.

[0078] In step S606, the Hamming distance between the first hash code and the second hash code may be determined to determine the degree of match between the first image and the second image. As mentioned above, the smaller the Hamming distance between the first hash code and the second hash code, the higher the degree of match between the first image and the second image, that is, the higher the similarity between the first image and the second image.

[0079] In some embodiments, the Hamming distance between the first hash code and the second hash code may be determined by the layer Hamming distance between each first layer in the first hash code and the second layer at the corresponding position in the second hash code. For example, the sum of multiple layer Hamming distances determined using multiple first layers of the first hash code and multiple second layers of the corresponding second hash code may be determined as the Hamming distance between the first hash code and the second hash code.

[0080] In one implementation, the Hamming distance between the i-th first level in the first hash code and the i-th second level in the second hash code may be calculated as the layer Hamming distance, where 1≤i≤K.

[0081] In some embodiments, when determining the layer Hamming distance between the i-th first layer in the first hash code and the i-th second layer in the second hash code, the layer Hamming distance between the first layer and the second layer can be determined by determining the Hamming distance between local hash bits of a predetermined local size in the first layer and local hash bits of the same local size in the corresponding second layer.

[0082] Fig. 7A and Figure 7B A schematic diagram of a process of determining a layer Hamming distance between a first layer in a first hash code and a second layer at a corresponding position in a second hash code according to an embodiment of the present application is shown.

[0083] like Fig. 7A As shown, H1 represents a first layer in the first hash code, and H2 represents a second layer in the second hash code corresponding to H1. The first layer and the second layer are both 4*4 matrices composed of 16 hash bits.

[0084] According to the predetermined local size 3*3, four different local hash bits H1_1, H1_2, H1_3 and H1_4 can be determined in H1, wherein the sizes of H1_1, H1_2, H1_3 and H1_4 are all the predetermined local size 3*3 and are located at different positions in H1, thus containing information of different spatial regions in the first layer H1. Similarly, four different local hash bits H2_1, H2_2, H2_3 and H2_4 containing different spatial information can also be determined in H2.

[0085] use Figure 7B The process shown can determine the Hamming distance between each local hash bit in H2 and all local hash bits in H1. For example, the Hamming distance between H2_1 and H1_1, the Hamming distance between H2_1 and H1_2, the Hamming distance between H2_1 and H1_3, and the Hamming distance between H2_1 and H1_4 can be determined respectively, and the minimum Hamming distance between H2_1 and H1_1, H1_2, H1_3, and H1_4 is determined as the Hamming distance D(H1, H2_1) between H2_1 and H1.

[0086] Using the above process, the Hamming distances D(H1, H2_2), D(H1, H2_3) and D(H1, H2_4) between H2_2, H2_3, H2_4 and H1 can be similarly determined. Then, the minimum value of the Hamming distances between H2_1, H2_2, H2_3, H2_4 and H1 can be determined as the layer Hamming distance D(H1, H2) between H1 and H2.

[0087] The above method can be used to determine the layer Hamming distance between the first layer H1 and the second layer H2. Fig. 7A and Figure 7B The method shown in can respectively determine the layer Hamming distance between each first layer and the corresponding second layer between the first hash code and the second hash code, and can determine the sum of multiple layer Hamming distances determined by multiple first layers of the first hash code and multiple second layers of the corresponding second hash code as the Hamming distance between the first hash code and the second hash code.

[0088] Utilize the above method, because utilize this application Figure 3 The hash code determined by the method shown in can retain the spatial information of the image. Therefore, by comparing the Hamming distance between the local hash bits at different local positions in the corresponding layer of the hash code, the similarity between images with partial matches can be more accurately evaluated. Fig. 7A , 7BIn the image matching process shown in , when there is a partial match between the first image and the second image, the similarity between the first image and the second image can be determined by the Hamming distance between the local hash bits of different local areas.

[0089] For example, when similar image information exists in the upper left corner of the first image and the lower right corner of the second image, if the similarity between the hash codes of the global information is directly determined, it will be difficult to identify the similarity of the locally similar image information. However, using the image matching method provided by the present application, it can be determined that the local hash bits in the upper left part of the layer of the hash code and the local hash bits in the lower right part of the layer of another hash code have a higher similarity (for example, a smaller Hamming distance), and by determining the Hamming distance between the two hash codes with reference to the Hamming distance between such local hash bits, the locally similar image information can be identified.

[0090] Figure 8 FIG. 8 is a schematic block diagram of an image data processing device 800 according to an embodiment of the present application. Figure 8 As shown, the image data processing apparatus 800 may include an image feature determination unit 810 , a local image feature determination unit 820 , a local feature vector determination unit 830 , and a hash code determination unit 840 .

[0091] The image feature determination unit 810 may be configured to determine image features of an input image.

[0092] In some embodiments, the image features of the input image may be extracted using a trained neural network. For example, a convolutional neural network may be used to process the input image to obtain the image features of the input image.

[0093] In other embodiments, the image features of the input image can be determined by calculating, for example, a directional gradient histogram of the image. The specific form of the image features of the input image is not limited here. In fact, those skilled in the art can use any possible method to achieve the determination of the image features for the input image.

[0094] In some embodiments, the image features of the input image may be represented in the form of a feature map, wherein the feature map may be represented in the form of a two-dimensional matrix. In some implementations, the image features of the input image may include feature maps of multiple channels, wherein the size of the feature map of each channel is the same.

[0095] The local image feature determination unit 820 may be configured to determine a plurality of local image features of the image feature.

[0096] In some embodiments, the image feature may be segmented at a preset size to determine a plurality of local regions of the image feature having a preset size. Then, information of the image feature corresponding to each local region of the plurality of local regions may be determined as a local image feature corresponding to the local region.

[0097] It can be understood that when the image features of the input image are composed of feature maps of multiple channels, the above-mentioned segmentation of the image features at a preset size refers to segmenting the feature map of each channel at a preset size.

[0098] In some embodiments, the above-mentioned multiple local areas may overlap with each other or may not overlap with each other.

[0099] The local feature vector determining unit 830 may be configured to generate a plurality of local feature vectors respectively corresponding to the plurality of local image features.

[0100] In some embodiments, for each local image feature, the channel local features of the corresponding local area in the feature map of each channel in the local image feature may be pooled to determine a local feature vector corresponding to the local image feature. The pooling described herein may include one or more of maximum pooling, minimum pooling, and average pooling.

[0101] In the case where the image feature includes feature maps of multiple channels, the local feature map of each channel can be pooled separately, and the pooling result of each channel is used as an element of the local feature vector. The number of elements of the local feature vector obtained in this way is the same as the number of channels of the feature map of the image feature.

[0102] The hash code determining unit 840 may be configured to determine a hash code of the input image based on the plurality of local feature vectors.

[0103] In some embodiments, the plurality of local feature vectors may be mapped into a plurality of sub-hash codes of the same length, respectively. For example, each local feature vector may be mapped into a sub-hash code of a predetermined length, such as a K-bit hash code, according to a predefined hash function. K is an integer greater than 1, and the value of K depends on the configuration of the specific hash function used.

[0104] The hash code of the input image is obtained by combining the multiple sub-hash codes. In one implementation, the multiple sub-hash codes may be arranged according to a spatial arrangement of corresponding local regions to obtain the hash code of the image.

[0105] The method provided by the present application can be used to obtain a hash code for representing input image information, such a hash code includes multiple layers, each layer is a two-dimensional matrix formed by multiple hash bits. One hash bit in each layer can be used to represent the image information of a local area of ​​the input image, and the hash bit of each layer can retain the spatial domain information of the input image.

[0106] By using the image data processing device provided by the present application, a higher-dimensional hash code for representing image information can be generated, thereby retaining the spatial domain information of the image in the generated hash code to improve the expression ability of the hash code for the local image.

[0107] Fig. 9 FIG. 9 is a schematic block diagram of an image matching device 900 according to an embodiment of the present application. Fig. 9 As shown, the image matching device 900 may include an image data processing unit 910 and a matching unit 920 .

[0108] The image data processing unit 910 may be composed of Figure 8 The image data processing device shown in FIG. 9 is implemented. The image data processing unit 910 can be configured to use Figure 3 The method shown processes a first image to determine a first hash code for representing the first image, wherein the first hash code comprises a plurality of first layers, each of which is a first matrix formed by a plurality of first hash bits.

[0109] In order to achieve image matching for the first image and the second image, the hash code determination unit 910 may also be configured to use Figure 3 The method shown processes the second image to determine a second hash code for representing the second image. The second hash code includes a plurality of second layers, each second layer being a second matrix formed by a plurality of second hash bits. As described above, the first hash code and the second hash code have the same number of layers. And each first layer and the corresponding second layer have the same number of hash bits. In some embodiments, each first layer and the corresponding second layer can be matrices of the same size.

[0110] In some embodiments, the hash codes of the first image and the second image may be predetermined by using an independently arranged image data processing device. Fig. 9 The image data processing unit 910 and the matching unit 920 shown in the figure can be integrated into one device, or can be two independent units respectively provided.

[0111] The matching unit 920 may be configured to determine the Hamming distance between the first hash code and the second hash code to determine the degree of matching between the first image and the second image. As described above, the smaller the Hamming distance between the first hash code and the second hash code, the higher the degree of matching between the first image and the second image, that is, the higher the similarity between the first image and the second image.

[0112] In some embodiments, the Hamming distance between the first hash code and the second hash code may be determined by the layer Hamming distance between each first layer in the first hash code and the second layer at the corresponding position in the second hash code. For example, the sum of multiple layer Hamming distances determined using multiple first layers of the first hash code and multiple second layers of the corresponding second hash code may be determined as the Hamming distance between the first hash code and the second hash code.

[0113] In one implementation, the Hamming distance between the i-th first level in the first hash code and the i-th second level in the second hash code may be calculated as the layer Hamming distance, where 1≤i is less than or equal to K.

[0114] In some embodiments, when determining the layer Hamming distance between the i-th first layer in the first hash code and the i-th second layer in the second hash code, the layer Hamming distance between the first layer and the second layer can be determined by determining the Hamming distance between local hash bits of a predetermined local size in the first layer and local hash bits of the same local size in the corresponding second layer.

[0115] By using the image matching device provided by the present application, the similarity between images with partial matching can be evaluated more accurately. When there is a partial match between the first image and the second image, the similarity between the first image and the second image can be determined by the Hamming distance between the local hash bits of different local areas.

[0116] Fig.10 An exemplary scene diagram of an image data processing system according to the present application is shown. Fig.10 As shown, the system 1000 may include a user terminal 1010 , a network 1020 , a server 1030 , and a database 1040 .

[0117] The user terminal 1010 may be, for example, Fig.10 The computer 1010-1 and the mobile phone 1010-2 are shown in FIG. It is understandable that, in fact, the user terminal can be any other type of electronic device capable of performing data processing, which can include but is not limited to desktop computers, laptops, tablet computers, smart phones, smart home devices, wearable devices, vehicle-mounted electronic devices, monitoring devices, etc. The user terminal can also be any equipment equipped with electronic devices, such as vehicles, robots, etc.

[0118] The user terminal provided in the present application can be used to receive the image to be processed, and implement image data processing using the method provided in the present application. For example, the user terminal can collect the image to be processed by an image acquisition device (such as a camera, a video camera, etc.) provided on the user terminal. For another example, the user terminal can also receive the image to be processed from an independently provided image acquisition device. For another example, the user terminal can also receive the image to be processed from a server via a network. The image to be processed described here can be a separate image or a frame in a video.

[0119] In some embodiments, the image processing method and image matching method provided by the present application can be executed by a processing unit of a user terminal. In some implementations, the user terminal can execute the image processing method and image matching method provided by the present application by using an application built into the user terminal. In other implementations, the user terminal can execute the image processing method and image matching method provided by the present application by calling an application stored externally of the user terminal.

[0120] In other embodiments, the user terminal sends the received image to be processed to the server 1030 via the network 1020, and the server 1030 executes the image processing method and image matching method provided in this application. In some implementations, the server 1030 can use the application built into the server to execute the image processing method and image matching method provided in this application. In other implementations, the server 1030 can execute the image processing method and image matching method provided in this application by calling the application stored outside the server.

[0121] The network 1020 may be a single network or a combination of at least two different networks. For example, the network 1020 may include, but is not limited to, a local area network, a wide area network, a public network, a private network, etc., or a combination of several of them.

[0122] Server 1030 can be a single server or a server group, and each server in the group is connected via a wired or wireless network. A server group can be centralized, such as a data center, or distributed. Server 1030 can be local or remote.

[0123] The database 1040 may generally refer to a device with a storage function. The database 1030 is mainly used to store various data used, generated and output from the user terminal 1010 and the server 1030. The database 1040 may be local or remote. The database 1040 may include various memories, such as random access memory (RAM), read only memory (ROM), etc. The storage devices mentioned above are just some examples, and the storage devices that can be used by the system are not limited to these.

[0124] The database 1040 may be interconnected or communicated with the server 1030 or a part thereof via the network 1020, or directly interconnected or communicated with the server 1030, or a combination of the above two methods.

[0125] In some embodiments, the database 1050 may be an independent device. In other embodiments, the database 1050 may also be integrated in at least one of the user terminal 1010 and the server 1040. For example, the database 1050 may be set on the user terminal 1010 or on the server 1040. For another example, the database 1050 may also be distributed, with a portion of it being set on the user terminal 1010 and another portion being set on the server 1040.

[0126] In addition, the method or device according to the embodiment of the present application can also be used by Fig.11 The architecture of the computing device shown is implemented. Fig.11 The architecture of the computing device is shown. Fig.11 As shown, the computing device 1100 may include a bus 1110, one or at least two CPUs 1120, a read-only memory (ROM) 1130, a random access memory (RAM) 1140, a communication port 1150 connected to a network, an input / output component 1160, a hard disk 1170, etc. The storage device in the computing device 1100, such as the ROM 1130 or the hard disk 1170, may store various data or files used for processing and / or communication of the method for detecting a target in a video provided by the present application, as well as program instructions executed by the CPU. The computing device 1100 may also include a user interface 1180. Of course, Fig.11 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig.11 One or at least two components of the computing device shown.

[0127] The embodiments of the present application may also be implemented as a computer-readable storage medium. Computer-readable instructions are stored on a computer-readable storage medium according to an embodiment of the present application. When the computer-readable instructions are executed by a processor, the method according to the embodiment of the present application described with reference to the above figures may be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0128] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0129] In addition, as shown in the present application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0130] In addition, although the present application makes various references to certain units in the system according to embodiments of the present application, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0131] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0132] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0133] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. An image processing method, comprising: determining image features of an input image; determining a plurality of local image features of the image feature; Determine a plurality of local feature vectors respectively corresponding to the plurality of local image features; determining a hash code of the input image based on the plurality of local feature vectors; Wherein, determining the hash code of the input image based on the multiple local feature vectors comprises: Mapping the multiple local feature vectors into multiple sub-hash codes of the same length respectively; and combining the plurality of sub-hash codes to obtain a hash code of the input image; Combining the plurality of sub-hash codes to obtain a hash code of the input image comprises: Arranging the plurality of sub-hash codes according to a spatial arrangement of the local regions corresponding to each sub-hash code to obtain a hash code of the input image; The hash code includes multiple layers, wherein the i-th layer in the hash code includes the i-th hash bit of each sub-hash code, and the i-th layer is a matrix formed by arranging the i-th hash bit of each sub-hash code according to the spatial arrangement of the local area corresponding to each sub-hash code, wherein i is an integer greater than or equal to 1 and less than or equal to K, and K is the length of each sub-hash code.

2. The image processing method according to claim 1, wherein: The multiple local image features for determining the image feature include: Segmenting the image feature at a preset size to determine a plurality of local regions of the image feature; Information on image features corresponding to each local area among the multiple local areas is respectively determined as a local image feature corresponding to the local area.

3. The image processing method as described in claim 2, wherein the image features include feature maps of multiple channels, and the local image features corresponding to each local area include channel local features of the corresponding local area of ​​the feature map of each channel in the feature maps of the multiple channels.

4. The image processing method according to claim 3, wherein: Determining a plurality of local feature vectors respectively corresponding to the plurality of local image features comprises: For each local image feature, the channel local features of the corresponding local area in the feature map of each channel are pooled to determine a local feature vector corresponding to the local image feature.

5. An image matching method for matching a first image and a second image, wherein the first image is represented as a first hash code, the first hash code includes a plurality of first layers, each of the first layers is a first matrix formed by a plurality of first hash bits, and the second image is represented as a second hash code, the second hash code includes a plurality of second layers, each of the second layers is a second matrix formed by a plurality of second hash bits, and wherein the number of first layers included in the first hash code and the number of second layers included in the second hash code are the same, and the sizes of the first matrix and the second matrix are the same, the image matching method comprising: determining a Hamming distance between the first hash code and the second hash code to determine a degree of match between the first image and the second image; The size of the first matrix is ​​the same as the size of the second matrix, and determining the Hamming distance between each first level in the first hash code and the second level at a corresponding position in the second hash code comprises: For each first matrix, the first matrix is ​​segmented according to a predetermined matching size to determine a plurality of first local areas in the first matrix, For each second matrix, the second matrix is ​​segmented according to the predetermined matching size to determine a plurality of second local areas in the second matrix, For each first local area, respectively determine a plurality of local Hamming distances between the first local area and each second local area in the plurality of second local areas; and A Hamming distance between the first layer and a corresponding second layer is determined according to the multiple local Hamming distances.

6. The image matching method according to claim 5, wherein: Determining the Hamming distance between the first hash code and the second hash code includes: Determine a plurality of layer Hamming distances using a plurality of first layers of the first hash code and a plurality of second layers of the corresponding second hash code, respectively; and A sum of the plurality of layer Hamming distances is determined as a Hamming distance between the first hash code and the second hash code.

7. An image data processing device, comprising a memory and a processor, wherein the memory stores instructions, and when the processor is used to execute the instructions, the processor executes the image processing method according to any one of claims 1 to 4.

8. An image matching device, comprising a memory and a processor, wherein the memory stores instructions, and when the processor is used to execute the instructions, the processor executes the image matching method according to any one of claims 5 to 6.

9. A computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 4.

Images