Image processing method, device, equipment and storage medium

Abstract

An image processing method, apparatus, device and storage medium, the method comprising: obtaining an image to be processed, the image to be processed comprising a first image and a second image; processing the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model after adding a plurality of hidden layers to the initial model obtained by pre-training; based on the plurality of feature codes of the first image and the plurality of feature codes of the second image, determining the similarity between the first image and the second image. Through the embodiments of the present application, high-dimensional abstract features and low-dimensional detail features of an image can be extracted simultaneously, and the efficiency and accuracy of image similarity calculation can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of Internet technology, the amount of multimedia data has increased rapidly, especially image data as the most important information carrier. How to accurately query similar images in these massive images has become a research hotspot.

[0003] In the process of querying similar images, it is necessary to calculate the similarity of two images, which often depends on the quality of the image features that characterize the images. However, traditional feature extraction methods, such as scale-invariant feature transform (SIFT) features, color histogram features, and Histogram of Oriented Gradient (HOG) features, can only obtain the differences in image detail features. Although deep learning methods can obtain abstract features (such as general features of shape contours) or specific features (such as detailed features of color textures) of images, the extracted image features are relatively single, and too many parameters representing image features can easily lead to slow calculation speed. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method, apparatus, device and storage medium, which can simultaneously extract high-dimensional abstract features and low-dimensional detail features of an image and improve the efficiency and accuracy of image similarity calculation.

[0005] In one aspect, an embodiment of the present invention provides an image processing method, the method comprising:

[0006] Acquire an image to be processed, where the image to be processed includes a first image and a second image;

[0007] Processing the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model after adding a plurality of hidden layers to the initial model obtained by pre-training;

[0008] Based on the plurality of feature codes of the first image and the plurality of feature codes of the second image, a similarity between the first image and the second image is determined.

[0009] On the other hand, an embodiment of the present invention provides an image processing device, the device comprising:

[0010] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a first image and a second image;

[0011] a processing module, configured to process the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model after adding a plurality of hidden layers to the initial model obtained by pre-training;

[0012] The processing module is further used to determine the similarity between the first image and the second image based on multiple feature codes of the first image and multiple feature codes of the second image.

[0013] Accordingly, an embodiment of the present invention provides a computer device, which includes a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to execute the image processing method described in any of the above possible implementation methods.

[0014] Accordingly, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, and the processor executes the program involved in the image processing method described in any possible implementation manner.

[0015] Accordingly, an embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image processing method described in any possible implementation manner.

[0016] In an embodiment of the present invention, a first image and a second image are processed by a feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, thereby calculating similarity based on the multiple feature codes of the first image and the multiple feature codes of the second image, wherein each feature code is obtained by adding image features output by a hidden layer, and high-dimensional abstract features and low-dimensional detail features of the image can be extracted simultaneously. At the same time, in order to address the problem of slow calculation speed caused by directly comparing image features, the present invention can reduce parameters of image features by encoding the acquired image features, thereby improving the efficiency and accuracy of image similarity calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a model training provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of image feature extraction provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of obtaining image similarity provided in an embodiment of the present application;

[0023] Figure 6 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;

[0024] Figure 7 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that the descriptions of "first", "second", etc. involved in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0027] The image processing method in the embodiments of the present application can be applied to some common image retrieval tasks or image duplication removal tasks. Some application scenarios are listed below.

[0028] For example, the terminal device can send an image that needs to be compared for similarity to a server. After receiving the image sent by the terminal device, the server can input the image into the feature extraction model proposed in this application, extract multiple image features of the image, and hash encode the multiple image features to obtain the feature code of the image. The similarity is calculated using the feature code of the image and the feature code of the image stored in the image library, so that image deduplication or image retrieval tasks can be performed based on the similarity. For example, when the similarity between the image sent by the terminal and the image in the image library is greater than or equal to a preset value, it can be determined that the image is in the image library and is a duplicate image or a similar image in the image library. For example, if the image type sent by the terminal is "Corgi", all images containing "Corgi" are queried from the image library.

[0029] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application, and the image processing system includes a terminal device 101 and a server 102. The terminal device 101 is connected to the server 102 via a network, for example, via a wireless network connection. The terminal device 101 is also called a terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent or user device. The terminal device can be a smart TV, a handheld device with wireless communication function (such as a smart phone, a tablet computer), a computing device (such as a personal computer (PC), a vehicle-mounted device, a wearable device or other smart device, but is not limited thereto.

[0030] The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0031] It can be understood that the image processing method of this embodiment can be executed on a terminal device, can be executed on a server, or can be executed jointly by the terminal device and the server.

[0032] In one embodiment, take the execution of the image processing method by server 102 as an example. Server 102 obtains the first image and the second image through the network, and processes the first image and the second image through the feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, thereby calculating the similarity based on the multiple feature codes of the first image and the multiple feature codes of the second image, wherein each feature code is obtained by adding image features output by a hidden layer. Therefore, the image features obtained by this method can cover both high-dimensional abstract features and low-dimensional detail features. At the same time, in order to solve the problem of slow calculation speed caused by directly comparing image features, the present application can reduce the parameters of image features by encoding the obtained image features, thereby improving the efficiency and accuracy of image similarity calculation.

[0033] It can be understood that the architecture diagram of the system described in the embodiment of the present application is for more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0034] In one embodiment, Figure 2 As shown, the embodiment of the present application is based on Figure 1 An image processing method provided by an image processing system. Figure 1 The server 102 mentioned in is taken as an example. Figure 2 The method of the embodiment of the present application is described.

[0035] S201 , obtaining an image to be processed, where the image to be processed includes a first image and a second image.

[0036] In one embodiment, the server can obtain the image to be processed from various electronic devices in communication connection through a wired connection or a wireless connection. The image to be processed may include a first image and a second image, and may also include a third image, a fourth image, etc., which is not limited in this application. The image to be processed may be image data acquired by an image acquisition device, or image data pre-stored in an electronic device, other storage devices associated with the electronic device, or in the cloud, or a picture frame obtained by extracting frames from video data. The image to be processed may be an RGB image or a YUV image, etc., and this application does not limit the format of the image to be processed.

[0037] S202. Process the first image and the second image using a feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, wherein the feature extraction model is obtained by training the initial model after adding multiple hidden layers to the pre-trained initial model.

[0038] In one embodiment, the server inputs the first image and the second image into a feature extraction model, thereby using the multiple newly added hidden layers included in the feature extraction model to output multiple image features of the first image and multiple image features of the second image, thereby determining multiple feature codes of the first image and multiple feature codes of the second image based on the multiple image features of the first image and the multiple image features of the second image. The image features output by the newly added hidden layer can be vectors or feature maps (matrices), which is not limited in this application. Since the image features are obtained by adding a hidden layer, the obtained image features can represent the detailed features of the image while representing the abstract features of the image.

[0039] In one embodiment, before the image to be processed is input into the feature extraction model, it is necessary to perform scale normalization on the image to be processed that is input into the feature extraction model, and scale the image to be processed to a fixed size, such as a size of 224×224, etc. This application does not limit this.

[0040] S203: Determine a similarity between the first image and the second image based on multiple feature codes of the first image and multiple feature codes of the second image.

[0041] In one embodiment, after obtaining multiple feature codes of the first image and multiple feature codes of the second image, the similarity between the first image and the second image may be determined according to the distance between the multiple feature codes of the first image and the multiple feature codes of the second image.

[0042] In an embodiment of the present application, the first image and the second image are processed by a feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, thereby calculating similarity based on the multiple feature codes of the first image and the multiple feature codes of the second image, wherein each feature code is obtained by the image features output by the hidden layer. Therefore, the image features obtained by this method can cover both high-dimensional abstract features and low-dimensional detail features. At the same time, in order to solve the problem of slow calculation speed caused by directly comparing image features, the present application can reduce the parameters of image features by encoding the obtained image features, thereby improving the efficiency and accuracy of image similarity calculation.

[0043] In one embodiment, the first image and the second image are processed using a feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, including: processing the first image and the second image using a feature extraction model to obtain multiple image features of the first image and multiple image features of the second image output by the multiple hidden layers newly added in the feature extraction model; hash encoding the multiple image features of the first image respectively to obtain multiple feature codes of the first image; hash encoding the multiple image features of the second image respectively to obtain multiple feature codes of the second image.

[0044] In one embodiment, the first image and the second image are input into a feature extraction model, and multiple image features of the first image and multiple image features of the second image are obtained through image features output by multiple hidden layers newly added in the feature extraction model.

[0045] Furthermore, multiple image features of the first image are hash-encoded to obtain multiple feature codes of the first image; multiple image features of the second image are hash-encoded to obtain multiple feature codes of the second image. Hash coding is to map each image feature to a binary code, such as the form of [01010110…], and the image features can be mapped to 64 bits, 128 bits, 256 bits, etc., which is not limited in this application. Hash coding can adopt locality sensitive hashing (LSH), iterative quantization hashing (ITQ), spectral hashing (SH), graph-based hashing (Hashing with Graphs), etc., which is not limited in this application.

[0046] In one embodiment, before the first image and the second image are processed using the feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, the method also includes: using the first training sample set and the category label of each training sample included in the first training sample set to pre-train the original model to obtain an initial model; embedding multiple hidden layers at multiple specified positions of the neural network in the initial model, and the multiple hidden layers are not adjacent to each other; using the second training sample set and the category label of each training sample included in the second training sample set, training the initial model embedded in the hidden layer to obtain a feature extraction model.

[0047] In one embodiment, before calculating the similarity between the first image and the second image, a feature extraction model needs to be established. First, the original model is pre-trained using the first training sample set and the category label of each training sample included in the first training sample set to obtain an initial model.

[0048] Specifically, the first training sample set is an image sample set including category labels, such as ImageNet dataset, COCO dataset, etc., which is not limited in this application. The neural network in the original model can be a VGG, AlexNet or ResNet-101 network, etc., which is not limited in this application. Taking the original model as VGG-16 as an example, as shown in Table 1, the model structure of the VGG-16 network is recorded.

[0049] Table 1 Model structure of VGG-16 network

[0050]

[0051]

[0052] As shown in Table 1 above, the VGG-16 network does not use different convolution and pooling layers at different stages of the network, but only uses convolution layers of size 3×3 and maximum pooling layers of size 2×2. The stride of the convolution layer is 1, and the stride of the pooling layer is 2. Two 3×3 convolution layers are used in sequence, and their receptive field is essentially 5×5. Similarly, the receptive field of three 3×3 convolution layers is 7×7. The benefit of using a set of 3×3 convolution layers instead of a single 5×5 or 7×7 layer is that more nonlinear activation functions are used, thereby increasing the discriminative power of the VGG network while reducing parameters. The number of convolutional and pooling layers doubles after each stage (64>128>256>512), and this uniformity of layer arrangement increases the simplicity of the network.

[0053] In one embodiment, see Figure 3In the first step of training, the VGG-16 network and output layer in the original model are pre-trained using the first training sample set (such as the ImageNet dataset) to obtain an initial model, and the output layer mainly outputs the predicted category of the image in the ImageNet dataset. Before the second step of training, multiple hidden layers are embedded at multiple specified positions of the neural network of the initial model, wherein the multiple hidden layers are not adjacent to each other, that is, they are not adjacent to each other. Then, the initial model is trained using the second training sample set and the category label of each training sample included in the second training sample set to obtain a feature extraction model. Among them, the second training sample set is an image dataset related to the task. For example, when performing similarity calculation, the second training sample set is mainly an image dataset including various animals; when it is an image retrieval task, the second training sample set is mainly an image retrieval library included in the image retrieval task, and the pictures in the image retrieval library are used for the training of the initial model. Among them, the output layer performs a classification task, and the output layer can use a softmax function for multi-classification as an activation function.

[0054] In one embodiment, it is necessary to perform scale normalization processing on the images in the first training sample set and the second training sample set, and scale the images to a fixed size, such as a size of 224×224, etc., which is not limited in the present application.

[0055] In one embodiment, using the initial model obtained after the first step of training, starting from the last fully connected layer in the neural network of the initial model, a new hidden layer is embedded every m (m is a positive integer) layers, and a total of n (n is a positive integer) hidden layers are inserted. This application does not limit the number of m and n, and can be designed according to the number of layers of the neural network in the initial model. Among them, the number of neurons in each hidden layer is the same as the eigenvalues ​​in the image features and the number of feature codes obtained by hashing the image features. This application takes the number of neurons in each hidden layer as 128 as an example for explanation.

[0056] In one embodiment, the images in the second training sample set can be subjected to image enhancement processing, such as adding one or more of Gaussian noise, salt and pepper noise, cropping, rotation, watermarking, hue conversion, upside down flipping, left-right flipping, and adding a black frame, so that the feature extraction model has the ability to resist attacking images, such as changing the size, direction, cropping, and adding a watermark to the image.

[0057] In the embodiment of the present application, since the neural network has powerful understanding and abstraction capabilities, by embedding multiple hidden layers in the neural network, the extracted image features can take into account both high-dimensional abstract features and low-dimensional detail features.

[0058] In one embodiment, the multiple image features of the first image are hash-encoded respectively to obtain multiple feature codes of the first image, including: for any image feature among the multiple image features of the first image, compressing the any image feature to obtain a corresponding multiple feature values; determining an image feature mean of the any image feature according to the multiple feature values; obtaining a difference between each feature value among the multiple feature values ​​and the image feature mean; determining a binary series according to the difference, and using the binary series as the feature code of the any image feature.

[0059] In one embodiment, Figure 4 As shown, the first image is input as an input image into the feature extraction model, and multiple image features of the first image are hash-coded respectively to obtain feature coding of each of the multiple image features of the first image. Hash coding is to map image features to binary coding, and image features can be mapped to 64 bits, 128 bits, 256 bits, etc., which is not limited in this application. Hash coding can adopt locality sensitive hashing (LSH), iterative quantization hashing (ITQ), spectral hashing (SH), graph-based hashing (Hashing with Graphs), etc., which is not limited in this application.

[0060] In one embodiment, when hash coding is performed on any image feature among the multiple image features of the first image, the acquired image features need to be compressed first to remove redundant information and reduce the amount of data. The present application does not limit the specific implementation method of the compression process. In a feasible embodiment, the image features can be compressed using DCT discrete cosine transform to obtain multiple eigenvalues ​​corresponding to the image features. The number of eigenvalues ​​is equal to the number of neurons in the hidden layer. When the number of neurons is 128, each image feature corresponds to 128 eigenvalues.

[0061] In one embodiment, an image feature mean of any image feature is determined based on multiple feature values, and the difference between each feature value in the multiple feature values ​​and the image feature mean is obtained, thereby determining a binary sequence based on the difference, and using the binary sequence as a feature code for any image feature.

[0062] Specifically, the mean value of any image feature is AVG n , then AVG n= the sum of multiple eigenvalues ​​corresponding to any image feature n / the number of multiple eigenvalues ​​(e.g. 128), where n represents the nth hidden layer added. A binary sequence is determined based on the difference between each eigenvalue in the multiple eigenvalues ​​corresponding to any image feature and the image feature mean of each image feature, and the binary sequence is used as the feature encoding of any image feature. If the multiple eigenvalues ​​of any image feature are [X1, X2, …, Xi …, Xn], then the difference between the i-th eigenvalue and the image feature mean of any image feature is X i -AVG n , when X i -AVG n Greater than a preset value, such as X i -AVG n >0, then the i-th element in the binary sequence is recorded as 1; or when X i -AVG n ≤0, the i-th element in the binary sequence is recorded as 0. The binary sequence is determined by the difference between each eigenvalue of multiple eigenvalues ​​[X1, X2, …, Xi …, Xn] of any image feature and the mean value of the image feature, thereby obtaining the feature code of any image, for example: the feature code is [01010110 …].

[0063] In one embodiment, the multiple image features of the second image are hash-encoded respectively to obtain multiple feature codes of the second image, including: for any image feature among the multiple image features of the second image, compressing the any image feature to obtain a corresponding multiple feature values; determining an image feature mean of the any image feature according to the multiple feature values; obtaining a difference between each feature value among the multiple feature values ​​and the image feature mean; determining a binary series according to the difference, and using the binary series as the feature code of the any image feature.

[0064] In one embodiment, Figure 4As shown, the second image is input as an input image into the feature extraction model, and multiple image features of the second image are hash-coded respectively to obtain a feature code of each of the multiple image features of the second image. When performing hash coding, the acquired image features need to be compressed first to remove redundant information and reduce the amount of data. The present application does not limit the specific implementation method of the compression process. In a feasible embodiment, the DCT discrete cosine transform can be used to compress the image features to obtain multiple eigenvalues ​​corresponding to the image features. The number of eigenvalues ​​is equal to the number of neurons in the hidden layer. When the number of neurons is 128, each image feature corresponds to 128 eigenvalues. Furthermore, the image feature mean of any image feature is determined based on the multiple eigenvalues, and the difference between each eigenvalue in the multiple eigenvalues ​​and the image feature mean is obtained, thereby determining a binary sequence based on the difference, and using the binary sequence as the feature code of any image feature.

[0065] In one embodiment, determining the binary sequence based on the difference includes: judging whether the difference between each eigenvalue and the image feature mean is greater than a preset value; if so, setting the element corresponding to each eigenvalue in the binary sequence to 1; if not, setting the element corresponding to each eigenvalue in the binary sequence to 0.

[0066] In one embodiment, when determining a binary sequence based on the difference between each eigenvalue of multiple eigenvalues ​​corresponding to any image feature and the image feature mean of any image feature, it is necessary to determine whether the difference between each eigenvalue and the image feature mean is greater than a preset value. If so, the element corresponding to each eigenvalue in the binary sequence is set to 1; if not, the element corresponding to each eigenvalue in the binary sequence is set to 0.

[0067] For example, if the multiple eigenvalues ​​of any image feature are [X1,X2,…,Xi…,Xn], then the difference between the i-th eigenvalue and the image feature mean of any image feature is X i -AVG n , when X i -AVG n Greater than a preset value, such as X i -AVG n >0, then the i-th element in the binary sequence is recorded as 1; or when X i -AVG n ≤0, then the i-th element in the binary sequence is recorded as 0.

[0068] In one embodiment, determining the similarity between the first image and the second image based on multiple feature codes of the first image and multiple feature codes of the second image includes: obtaining the distance between each feature code in the multiple feature codes of the first image and a corresponding feature code in the multiple feature codes of the second image; and determining the similarity between the first image and the second image based on the distance between each feature code and the corresponding feature code.

[0069] In one embodiment, when the server obtains the distance between each feature code in the multiple feature codes of the first image and the corresponding feature code in the multiple feature codes of the second image, it can determine the target hidden layer corresponding to any feature code in the multiple hidden layers newly added to the feature extraction model for any feature code in the multiple feature codes of the first image, thereby obtaining the feature code corresponding to the target hidden layer in the multiple feature codes of the second image, and calculating the distance between any feature code and the feature code corresponding to the target hidden layer. The embodiment of the present application takes the calculation of the Hamming distance as an example, and the Hamming distance calculation formula is: Where i=0,1,...,n-1, x[i] is the i-th element of feature code x, y[i] is the i-th element of feature code y, represents XOR, x, y are the feature codes corresponding to any feature code and the target hidden layer respectively.

[0070] In one embodiment, Figure 5 As shown in FIG. 1 , when determining the similarity between the first image and the second image according to the distance between each feature code and the corresponding feature code, the distance between each feature code and the feature code corresponding to the target hidden layer can be standardized, such as zero mean standardization, then the Hamming distance d(x, y) after standardization is Z = (d(x, y)-μ) / σ, μ and σ are respectively When there are n new hidden layers, the similarity SIM between the first image and the second image is AVG(Z n ), wherein AVG is an averaging function. When the similarity SIM is larger, the similarity between the first image and the second image is higher. Conversely, when the similarity SIM is smaller, the similarity between the first image and the second image is lower.

[0071] In the embodiment of the present application, by performing hash coding on image features, similarity can be calculated based on feature coding, which can effectively reduce the data size, thereby improving the efficiency and accuracy of image similarity calculation.

[0072] In one embodiment, the step of obtaining the distance between each feature code among the multiple feature codes of the first image and a corresponding feature code among the multiple feature codes of the second image includes: determining, for any feature code among the multiple feature codes of the first image, a target hidden layer corresponding to the any feature code among the multiple hidden layers newly added to the feature extraction model; obtaining the feature code corresponding to the target hidden layer among the multiple feature codes of the second image; and calculating the distance between the any feature code and the feature code corresponding to the target hidden layer.

[0073] In one embodiment, for any feature code among the multiple feature codes of the first image, determine that any feature code corresponds to the target hidden layer in the feature extraction model, and calculate the distance between any feature code and the feature code corresponding to the target hidden layer. This embodiment of the application takes the calculation of the Hamming distance as an example, and the Hamming distance calculation formula is: Where i=0,1,...,n-1, x[i] is the i-th element of feature code x, y[i] is the i-th element of feature code y, represents XOR, x, y are the feature codes corresponding to any feature code and the target hidden layer respectively.

[0074] like Figure 6 As shown, Figure 6 : is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application, the device comprising:

[0075] An acquisition module 601 is used to acquire an image to be processed, where the image to be processed includes a first image and a second image;

[0076] A processing module 602 is used to process the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model after adding a plurality of hidden layers to the initial model obtained by pre-training;

[0077] The processing module 602 is further configured to determine a similarity between the first image and the second image based on a plurality of feature codes of the first image and a plurality of feature codes of the second image.

[0078] In one embodiment, the processing module 602 is specifically configured to:

[0079] Processing the first image and the second image using a feature extraction model to obtain a plurality of image features of the first image and a plurality of image features of the second image output by the plurality of hidden layers newly added in the feature extraction model;

[0080] Performing hash coding on multiple image features of the first image respectively to obtain multiple feature codes of the first image;

[0081] Hash encoding is performed on multiple image features of the second image respectively to obtain multiple feature codes of the second image.

[0082] In one embodiment, the processing module 602 is specifically configured to:

[0083] Using the first training sample set and the category label of each training sample included in the first training sample set, pre-training the original model to obtain an initial model;

[0084] Embedding multiple hidden layers at multiple specified positions of the neural network in the initial model, wherein the multiple hidden layers are not adjacent to each other;

[0085] The initial model embedded in the hidden layer is trained using the second training sample set and the category label of each training sample included in the second training sample set to obtain a feature extraction model.

[0086] In one embodiment, the processing module 602 is specifically configured to:

[0087] For any image feature among the plurality of image features of the first image, compress the any image feature to obtain a plurality of corresponding feature values;

[0088] Determine an image feature mean value of any one of the image features according to the multiple feature values;

[0089] Obtaining a difference between each eigenvalue of the plurality of eigenvalues ​​and a mean value of the image feature;

[0090] A binary number sequence is determined according to the difference, and the binary number sequence is used as a feature code of any one of the image features.

[0091] In one embodiment, the processing module 602 is specifically configured to:

[0092] Determine whether the difference between each characteristic value and the image characteristic mean is greater than a preset value;

[0093] If yes, then set the element corresponding to each eigenvalue in the binary sequence to 1;

[0094] If not, the element corresponding to each eigenvalue in the binary sequence is set to 0.

[0095] In one embodiment, the processing module 602 is specifically configured to:

[0096] Acquire a distance between each feature code in a plurality of feature codes of the first image and a corresponding feature code in a plurality of feature codes of the second image;

[0097] The similarity between the first image and the second image is determined according to the distance between each feature code and the corresponding feature code.

[0098] In one embodiment, the processing module 602 is specifically configured to:

[0099] For any feature code among the plurality of feature codes of the first image, determine a target hidden layer corresponding to the any feature code among the plurality of hidden layers newly added to the feature extraction model;

[0100] Obtaining a feature code corresponding to the target hidden layer from among a plurality of feature codes of the second image;

[0101] Calculate the distance between any one of the feature codes and the feature code corresponding to the target hidden layer.

[0102] Through the embodiments of the present application, the first image and the second image are processed by a feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, thereby calculating the similarity based on the multiple feature codes of the first image and the multiple feature codes of the second image, wherein each feature code is obtained by the image features output by the hidden layer, so the image features obtained by this method can cover both high-dimensional abstract features and low-dimensional detail features. At the same time, in order to solve the problem of slow calculation speed caused by directly comparing image features, the present application can reduce the parameters of the image features by encoding the obtained image features, thereby improving the efficiency and accuracy of image similarity calculation.

[0103] like Figure 7 As shown, Figure 7 is a schematic diagram of a computer device provided in an embodiment of the present application, the internal structure of the device is as follows Figure 7 As shown, it includes: one or more processors 701, memory 702, and communication interface 703. The processor 701, memory 702, and communication interface 703 may be connected via a bus 704 or other means. The embodiment of the present application takes the connection via the bus 704 as an example.

[0104] Among them, the processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off instructions sent by the user to the computer device, and control the computer device to perform power on and off operations; for another example, the CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 703 can optionally include a standard wired interface, a wireless interface (such as Wi-Fi, a mobile communication interface, etc.), which is controlled by the processor 701 to send and receive data. The memory 702 (Memory) is a memory device in the computer device for storing programs and data. It can be understood that the memory 702 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 702 provides a storage space, which stores the operating system of the computer device, which may include but is not limited to: Windows system, Linux system, etc., and this application does not limit this.

[0105] In one embodiment, the processor 701 is specifically configured to:

[0106] Acquire an image to be processed, where the image to be processed includes a first image and a second image;

[0107] Processing the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model after adding a plurality of hidden layers to the initial model obtained by pre-training;

[0108] Based on the plurality of feature codes of the first image and the plurality of feature codes of the second image, a similarity between the first image and the second image is determined.

[0109] In one embodiment, the processor 701 is specifically configured to:

[0110] Processing the first image and the second image using a feature extraction model to obtain a plurality of image features of the first image and a plurality of image features of the second image output by the plurality of hidden layers newly added in the feature extraction model;

[0111] Performing hash coding on multiple image features of the first image respectively to obtain multiple feature codes of the first image;

[0112] Hash encoding is performed on multiple image features of the second image respectively to obtain multiple feature codes of the second image.

[0113] In one embodiment, the processor 701 is specifically configured to:

[0114] Using the first training sample set and the category label of each training sample included in the first training sample set, pre-training the original model to obtain an initial model;

[0115] Embedding multiple hidden layers at multiple specified positions of the neural network in the initial model, wherein the multiple hidden layers are not adjacent to each other;

[0116] The initial model embedded in the hidden layer is trained using the second training sample set and the category label of each training sample included in the second training sample set to obtain a feature extraction model.

[0117] In one embodiment, the processor 701 is specifically configured to:

[0118] For any image feature among the plurality of image features of the first image, compress the any image feature to obtain a plurality of corresponding feature values;

[0119] Determine an image feature mean value of any one of the image features according to the multiple feature values;

[0120] Obtaining a difference between each eigenvalue of the plurality of eigenvalues ​​and a mean value of the image feature;

[0121] A binary number sequence is determined according to the difference, and the binary number sequence is used as a feature code of any one of the image features.

[0122] In one embodiment, the processor 701 is specifically configured to:

[0123] Determine whether the difference between each characteristic value and the image characteristic mean is greater than a preset value;

[0124] If yes, then set the element corresponding to each eigenvalue in the binary sequence to 1;

[0125] If not, the element corresponding to each eigenvalue in the binary sequence is set to 0.

[0126] In one embodiment, the processor 701 is specifically configured to:

[0127] Acquire a distance between each feature code in a plurality of feature codes of the first image and a corresponding feature code in a plurality of feature codes of the second image;

[0128] The similarity between the first image and the second image is determined according to the distance between each feature code and the corresponding feature code.

[0129] In one embodiment, the processor 701 is specifically configured to:

[0130] For any feature code among the plurality of feature codes of the first image, determine a target hidden layer corresponding to the any feature code among the plurality of hidden layers newly added to the feature extraction model;

[0131] Obtaining a feature code corresponding to the target hidden layer from among a plurality of feature codes of the second image;

[0132] Calculate the distance between any one of the feature codes and the feature code corresponding to the target hidden layer.

[0133] Through the embodiments of the present application, the first image and the second image are processed by a feature extraction model to obtain multiple feature codes of the first image and multiple feature codes of the second image, thereby calculating the similarity based on the multiple feature codes of the first image and the multiple feature codes of the second image, wherein each feature code is obtained by inputting the image features into the hidden layer, so the image features obtained by this method can cover both high-dimensional abstract features and low-dimensional detail features. At the same time, in order to solve the problem of slow calculation speed caused by directly comparing image features, the present application can reduce the parameters of the image features by encoding the obtained image features, thereby improving the efficiency and accuracy of image similarity calculation.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the process of the above-mentioned embodiment of the image processing method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0135] One or more embodiments of the present application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps performed in the embodiments of the above-mentioned methods.

[0136] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, where the image to be processed includes a first image and a second image; The first image and the second image are processed by using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model with the newly added hidden layers by using a second training sample set after adding a plurality of hidden layers to the initial model obtained by pre-training, the initial model is obtained by pre-training the original model by using a first training sample data set, the second training sample set is an image data set for similarity calculation of a specific type, and the plurality of feature codes are obtained by image features output by the newly added plurality of hidden layers; Determining the similarity between the first image and the second image based on the multiple feature codes of the first image and the multiple feature codes of the second image includes: obtaining the distance between each feature code in the multiple feature codes of the first image and the corresponding feature code in the multiple feature codes of the second image; determining the similarity between the first image and the second image according to the distance between each feature code and the corresponding feature code; The method of obtaining the distance between each feature code among the multiple feature codes of the first image and the corresponding feature code among the multiple feature codes of the second image includes: for any feature code among the multiple feature codes of the first image, determining a target hidden layer corresponding to the any feature code among the multiple hidden layers newly added to the feature extraction model; obtaining the feature code corresponding to the target hidden layer among the multiple feature codes of the second image; and calculating the distance between the any feature code and the feature code corresponding to the target hidden layer.

2. The method according to claim 1, characterized in that Processing the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image includes: Processing the first image and the second image using a feature extraction model to obtain a plurality of image features of the first image and a plurality of image features of the second image output by the plurality of hidden layers newly added in the feature extraction model; Performing hash coding on multiple image features of the first image respectively to obtain multiple feature codes of the first image; Hash encoding is performed on multiple image features of the second image respectively to obtain multiple feature codes of the second image.

3. The method according to claim 1 or 2, characterized in that: Before the first image and the second image are processed by using the feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, the method further includes: Using the first training sample set and the category label of each training sample included in the first training sample set, pre-training the original model to obtain an initial model; Embedding multiple hidden layers at multiple specified positions of the neural network in the initial model, wherein the multiple hidden layers are not adjacent to each other; The initial model embedded in the hidden layer is trained using the second training sample set and the category label of each training sample included in the second training sample set to obtain a feature extraction model.

4. The method according to claim 2, characterized in that: The performing hash coding on the plurality of image features of the first image respectively to obtain the plurality of feature codes of the first image includes: For any image feature among the plurality of image features of the first image, compress the any image feature to obtain a plurality of corresponding feature values; Determine an image feature mean value of any one of the image features according to the multiple feature values; Obtaining a difference between each eigenvalue of the plurality of eigenvalues ​​and a mean value of the image feature; A binary number sequence is determined according to the difference, and the binary number sequence is used as a feature code of any one of the image features.

5. The method according to claim 4, characterized in that Determining a binary number sequence according to the difference comprises: Determine whether the difference between each characteristic value and the image characteristic mean is greater than a preset value; If yes, then set the element corresponding to each eigenvalue in the binary sequence to 1; If not, the element corresponding to each eigenvalue in the binary sequence is set to 0.

6. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a first image and a second image; A processing module, used for processing the first image and the second image using a feature extraction model to obtain a plurality of feature codes of the first image and a plurality of feature codes of the second image, wherein the feature extraction model is obtained by training the initial model with the newly added hidden layers by a second training sample set after adding a plurality of hidden layers to the initial model obtained by pre-training, the initial model is obtained by pre-training the original model by using the first training sample data set, the second training sample set is an image data set for similarity calculation of a specific type, and the plurality of feature codes are obtained by image features output by the newly added plurality of hidden layers; The processing module is further used to determine the similarity between the first image and the second image based on the multiple feature codes of the first image and the multiple feature codes of the second image, including: obtaining the distance between each feature code in the multiple feature codes of the first image and the corresponding feature code in the multiple feature codes of the second image; determining the similarity between the first image and the second image according to the distance between each feature code and the corresponding feature code; In terms of obtaining the distance between each feature code in the multiple feature codes of the first image and the corresponding feature code in the multiple feature codes of the second image, the processing module is specifically used to: determine, for any feature code in the multiple feature codes of the first image, a target hidden layer corresponding to the any feature code in the multiple hidden layers newly added to the feature extraction model; obtain the feature code corresponding to the target hidden layer in the multiple feature codes of the second image; and calculate the distance between any feature code and the feature code corresponding to the target hidden layer.

7. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a set of program codes, and the processor calls the program codes stored in the memory to execute the image processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images