A multi-scale feature extraction method, system, device and storage medium for strengthening image similarity matching

Abstract

The application discloses a kind of multi-scale feature extraction method, system, equipment and storage medium of reinforced image similarity matching, belong to machine learning field, solve the problem of low precision and efficiency of complex image similarity matching;Including: the color normalization of received image is normalized and using interpolation method or image edge is reduced to obtain preprocessed image;Preprocessed image and target image are simultaneously input into twin neural network, and after neck network is convolved, the multi-layer feature map corresponding to each image is obtained and residual connection is carried out;Fusion feature map of semantic information and spatial information fusion is obtained using PAN;SPP is introduced to generate multi-layer long width depth fixed standard feature map, the weight sharing of full connection layer is realized;According to the efficiency weight of task machine, the standard feature map is spliced and input into full connection layer, according to the task type, the corresponding output layer network is selected, and the prediction result is obtained.The application improves the precision and efficiency of complex image similarity matching by combining SPP and PAN.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine learning, and relates to a multi-scale feature extraction method and system for reinforcing image similarity matching, a device and a storage medium. BACKGROUND

[0002] In the field of computer vision, image similarity matching is an important basic technology, which is widely used in image retrieval, target detection, image classification and image stitching and other fields.

[0003] The core of image similarity matching lies in effectively extracting and comparing features in images to determine the similarity of two images. With the sharp increase in the amount of image data and the diversification of application scenarios, traditional image similarity matching methods gradually fail to meet the actual needs in terms of accuracy and efficiency. Therefore, how to design an efficient and accurate image similarity matching method has become one of the current research hotspots and difficulties.

[0004] With the rise of deep learning technology, convolutional neural networks (CNN) have shown great ability in image feature extraction and similarity matching. Through multiple convolution and pooling operations, CNN can extract features of images and train on large-scale image datasets to obtain more discriminative feature representations.

[0005] Existing image similarity matching methods mainly rely on feature extraction technology, which converts images into feature vectors and judges the similarity of images through the similarity measurement of feature vectors. Traditional feature extraction methods mainly include pixel-based feature extraction methods, statistical-based feature extraction methods and local feature-based descriptor feature extraction methods such as SIFT, SURF, etc. Traditional feature extraction methods solve the problem of image similarity matching to some extent, but when dealing with complex images with non-unique target subjects, the algorithm cannot be directly transplanted into deep learning models due to its interpretability, and cannot adapt to different scales and rotation changes. Therefore, traditional feature extraction methods cannot meet the efficient and accurate needs of complex images in image similarity matching.

[0006] However, except for Unet and other end-to-end models for generating images, the topmost model output layer is a fully connected layer, which limits the image size, and the pictures generated in actual production are not limited. Therefore, simple image stretching may cause damage to the pixels. And if only a single scale feature extraction method is used to extract the multi-scale changes of the target in the image, there are still certain limitations.

[0007] For example, in the prior art, the patent application with publication number CN117911731A mentions a method based on multi-scale feature fusion, which combines the attention mechanism in the Transformer model, performs multi-scale fusion through batch matrix multiplication Bmm, generates K, Q and V features of a group of images to be matched, and extracts image features at different scales and fuses them through a multi-attention mechanism, taking Q features as the main features and K and V features as key features. This method improves the extraction of multi-scale features. However, the attention computer mechanism method has certain complexity in the feature fusion process, and has high demand for computing resources. Although multi-head attention mechanism can be applied to improve efficiency, it will also lead to further increase in demand for computing resources.

[0008] Therefore, how to design a multi-scale feature extraction technology suitable for image similarity matching to improve the accuracy and efficiency of matching is an important problem to be solved in the current technical field. SUMMARY

[0009] To solve the above technical problems, the present application provides a multi-scale feature extraction method for strengthening image similarity matching, system, device and storage medium, which introduces two technologies of Path Aggregation Network (PAN) and Spatial Pyramid Pooling (SPP), through multi-scale extraction and aggregation of image features, enhances the robustness and accuracy of image processing algorithm, solves the problem that traditional feature extraction methods often fail to capture enough detailed information when facing image scale changes, complex backgrounds and object deformation, etc., resulting in very limited image similarity matching in the subsequent process, and cannot well sense the image stretching phenomenon, and the problem of high complexity and large demand for computing resources in the existing multi-scale feature extraction method in the feature fusion process, aiming to propose a method with strong universality, taking convolutional neural network as the core, strengthening image multi-scale information extraction and improving the accuracy and efficiency of image similarity matching.

[0010] The object of the present application is realized by the following technical solutions:

[0011] The present application provides a multi-scale feature extraction method for strengthening image similarity matching, which comprises:

[0012] Step 1, color normalization of the received image to obtain a normalized image, and using interpolation method or image edge reduction method to pre-process the edge of the normalized image to obtain a pre-processed image;

[0013] Step two, the pretreated image and the target image are input into the twin neural network at the same time, and the neck network is used for convolution to obtain a plurality of layers of feature maps corresponding to each image; the residual block is used to perform residual connection on the initial position of each layer of feature maps and the end position of the upper layer of feature maps;

[0014] Step three, the feature pyramid network (FPN) in the path aggregation network (PAN) is used to transmit the semantic information of the high layer feature map of each image after residual connection to the bottom layer feature map, and then transmit the spatial information of the bottom layer feature map to the high layer feature map, so that the semantic information and spatial information of each layer of feature maps are fused to obtain a fusion feature map;

[0015] Step four, the spatial pyramid pooling (SPP) is introduced, the width and height of the output of each layer of fusion feature maps are fixed through the three-channel dynamic maximum pooling network (dynamics max pooling layer), and then the depth of the output is fixed through the 1*1 convolution kernel to generate a plurality of layers of fixed length, width and depth of the specification feature map, so as to realize the weight sharing of the full connection layer;

[0016] Step five, according to the efficiency weight of the task machine, the specification feature map is spliced and input into the full connection layer, and according to the type of task, the corresponding output layer network is selected to obtain the similarity of the pretreated image and the target image, and the final prediction result is obtained.

[0017] In step one, the method for normalizing the received image to obtain a normalized image comprises:

[0018] The R, G and B channel pixel mean and standard deviation of the images divided into a training set in the target library are calculated respectively;

[0019] Based on the pixel mean and standard deviation, the received image is color normalized to obtain a normalized image:

[0020]

[0021] Wherein, X is a pixel point of the normalized image, x is a pixel point of the received image, is the pixel mean; and σ is the standard deviation.

[0022] In step one, the method for edge preprocessing of the normalized image to obtain a pretreated image comprises:

[0023] Setting the length and width threshold of the image edge;

[0024] If the length-width of the normalized image is less than the length-width threshold, an interpolation method of expanding the edge of the normalized image by pixel point 0 is used until the set length-width threshold is reached, and the expansion is stopped, to obtain the preprocessed image.

[0025] If the length-width of the normalized image is greater than the length-width threshold, an image edge cutting method of cutting the edge pixel points of the normalized image is used until the set length-width threshold is reached, and the cutting is stopped, to obtain the preprocessed image.

[0026] In step two, the preprocessed image and the target image are simultaneously input into the twin neural network, and the neck network is convolved to obtain the corresponding multi-layer feature map of each image.

[0027] The preprocessed image and the target image searched in the target library are simultaneously input into the twin neural network, and the neck network is convolved to obtain the corresponding five-layer feature map of each image.

[0028] In step four, the three-channel dynamic maximum pooling network (dynamics max pooling layer) can be replaced by an inverse convolution layer.

[0029] In step five, the method for splicing the normalized feature map according to the efficiency weight of the task machine includes:

[0030] According to the efficiency weight of the task machine, the normalized feature map of each channel is flattened into a one-dimensional vector according to the channel order, and the one-dimensional vectors of all channels are depth-spliced; or the depth-splicing is performed in a stacking manner.

[0031] In step five, after the normalized feature map is spliced according to the efficiency weight of the task machine, the spliced normalized feature map is input into a full connection layer, a corresponding output layer network is selected according to the task type, the similarity between the preprocessed image and the target image is obtained, and the final prediction result is obtained.

[0032] The spliced normalized feature map is input into a 1024 full connection layer, the output of the preprocessed image and the target image is subjected to Manhattan distance alignment absolute difference, a single neuron sigmod classification output layer network is selected according to the task type, the similarity between the preprocessed image and the target image is obtained, and the final prediction result is obtained.

[0033] The application also provides a multi-scale feature extraction system for strengthening image similarity matching, comprising:

[0034] A preprocessing module is configured to normalize the color of the received image to obtain a normalized image, and to preprocess the edge of the normalized image by using an interpolation method or an image edge cutting method to obtain a preprocessed image.

[0035] The feature extraction module is used to simultaneously input the preprocessed image and the target image into the Siamese neural network, and then perform convolution through the Neck network to obtain multi-layer feature maps corresponding to each image; residual blocks are used to perform residual connections between the initial position of each layer feature map and the end position of the upper layer feature map.

[0036] The fusion module utilizes the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN) to transmit the semantic information of the high-level feature maps of each image after residual connection to the low-level feature maps, and then transmits the spatial information of the low-level feature maps to the high-level feature maps, so that the semantic and spatial information of each layer of feature maps are fused to obtain a fused feature map.

[0037] The specification module is used to introduce Spatial Pyramid Pooling (SPP), which fixes the width and height of the output of each fused feature map through a three-channel dynamic maxpooling layer, and then fixes the depth of the output through a 1*1 convolutional kernel, generating multiple specification feature maps with fixed length, width and depth, thereby realizing weight sharing of fully connected layers.

[0038] The similarity acquisition module is used to concatenate the standardized feature maps according to the efficiency weight of the task machine, input them into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and obtain the final prediction result.

[0039] The present invention also provides a multi-scale feature extraction device for enhancing image similarity matching, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above method.

[0040] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0041] This invention provides a multi-scale feature extraction method, system, device, and storage medium for enhancing image similarity matching, addressing the following problems:

[0042] 1. In the field of deep learning, feature extraction methods struggle to guarantee matching accuracy and robustness when dealing with complex images, different scales, and rotational changes, and when facing the recognition of objects of different scales. Furthermore, they cannot adapt to images of different sizes.

[0043] 2. Existing multi-scale feature extraction methods suffer from high complexity and large computational resource requirements during feature fusion, which affects the efficiency of practical applications.

[0044] The beneficial effects of this invention are:

[0045] PAN's feature extraction integrates the semantic information of high-level feature maps and the spatial information of low-level feature maps, enabling the invention to maintain stable performance even when facing different resolutions and complex scenes.

[0046] The fusion of SPP and PAN perfectly mitigates the problem of significantly increased model complexity and consequently decreased model prediction and training efficiency caused by PAN networks. It effectively reduces the computational cost of the model, enabling efficient image similarity recognition even in environments with limited computing resources.

[0047] This technology is not only applicable to similarity recognition tasks, but can also be extended to other deep learning tasks that require processing multi-scale information, such as object detection and image classification, and has a high degree of applicability.

[0048] The combination of SPP and PAN significantly enhances the model's image processing capabilities at different resolutions, reduces the fluctuation in recognition accuracy caused by changes in image resolution, and effectively improves the model's stability and robustness. Attached Figure Description

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0050] Figure 1 This is a schematic diagram of a multi-scale feature extraction method for enhancing image similarity matching, using ResNetV50 as an example, provided in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of an image being convolved and residually connected by a Neck network, as provided in an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram illustrating the fusion of semantic and spatial information of each layer's feature map using the FPN provided in this embodiment of the invention.

[0053] Figure 4 This is a schematic diagram of a standard feature map with fixed length, width and depth generated by SPP according to an embodiment of the present invention.

[0054] Figure 5 This is a schematic diagram illustrating the similarity between a preprocessed image and a target image, provided in an embodiment of the present invention.

[0055] Figure 6The online city-level platform is applied to the technical solution disclosed by the present application.

[0056] Figure 7 The training comparison schematic diagram is provided by the embodiment of the present application.

[0057] Figure 8 The accuracy change schematic diagram of not using the technical solution disclosed by the present application is provided by the embodiment of the present application.

[0058] Figure 9 The efficiency schematic diagram of not using the technical solution disclosed by the present application is provided by the embodiment of the present application.

[0059] Figure 10 The efficiency schematic diagram of using the technical solution disclosed by the present application is provided by the embodiment of the present application.

[0060] wherein, Figures 2 to 5 is Figure 1 An enlarged view of each step in the method. DETAILED DESCRIPTION

[0061] Embodiment one

[0062] The embodiment one of the present application provides a multi-scale feature extraction method for strengthening image similarity matching, as shown in the figure, the method comprises the following steps: Figure 1

[0063] Step one, normalizing the color of the received image to obtain a normalized image, and using an interpolation method or an image edge reduction method to pre-process the edge of the normalized image to obtain a pre-processed image;

[0064] The strengthening of image similarity matching work is often affected by different shooting devices, or light, etc. The result is deviated, in order to effectively eliminate the color difference under the condition of light change, camera setting and different shooting, the image channel pixel is normalized, and the image pixel is converted into the change of pixel. In addition, the traditional image preprocessing only compresses or stretches the image in proportion, which also causes the problem of scale deformation. From the perspective of image similarity matching, the scale invariance of the image needs to be preserved in the image preprocessing stage, therefore, the interpolation method is adopted to expand the image and the image edge reduction method is adopted to clip the image, so as to ensure the stability of the length and width.

[0065] Step two, inputting the pre-processed image and the target image into a twin neural network at the same time, performing convolution through a neck network to obtain a multi-layer feature map corresponding to each image; using a residual block to perform residual connection on the initial position of each layer feature map and the end position of the upper layer feature map;

[0066] ​The similarity comparison between two images requires the extraction of their features in a deep neural network. To ensure that the extracted features are from the same level, machine learning (ResNet) is injected into the twin neural network, and the weights of two identical ResNet network layers are shared. This ensures that the observation surface of a group of images is the same when comparing two images. The ResNet model shared on HuggingFace is used for image classification. We only want to extract image features through ResNet, so we need to remove the top layer in the original ResNet. Then freeze the non-terminal two layers of neurons for image feature extraction. Because the model is easily overfit to a small number of unstable task objects when comparing image similarity, this also effectively prevents the model from overfitting during training.

[0067] Step three, use the feature pyramid network (Feature Pyramid Network, FPN) in the path aggregation network (Path Aggregation Network, PAN) to transmit the semantic information of the high-level feature map of each image after residual connection to the bottom layer feature map, and then transmit the spatial information of the bottom layer feature map to the high-level feature map, so that the semantic information and spatial information of each layer of feature map are fused to obtain the fused feature map;

[0068] In deep learning models, especially in image processing tasks, as the convolutional layers continue to deepen, the spatial information of the image is gradually replaced by semantic information, resulting in a lack of comprehensive understanding of the image by the model in the final prediction output. In order to better capture and utilize the multi-level information in the image, Path Aggregation Network (PAN) is an effective structure that can strengthen the spatial feature information of the bottom layer while maintaining the high-level semantic information.

[0069] Specifically, PAN uses the Feature Pyramid Network (FPN) design to pass high-level semantic information to the bottom layer feature map. This transmission method ensures that the bottom layer feature map not only has the original high-resolution spatial information, but also incorporates high-level semantic features, thereby improving the overall feature map expression ability.

[0070] To further enhance the model's capture of image details, PAN also transmits the bottom layer feature map to the high layer. This bidirectional transmission mechanism allows each layer of feature map to retain the corresponding level of information and integrate multi-level semantic and spatial information. This multi-level information fusion allows the model to more comprehensively consider important features in the image when making the final prediction, improving the prediction accuracy and robustness of the model.

[0071] By introducing PAN, the model can effectively pass information between different levels, so that the feature map of each layer not only retains spatial details but also has rich semantic information. The advantage of this is that the model can better understand the image content when processing complex images, thereby improving the accuracy of tasks such as classification, detection, or segmentation. This structure not only enhances the expression ability of the model but also reduces the prediction error caused by information loss, making the model perform better in practical applications.

[0072] In step two, it is mentioned that during the continuous convolution process, the spatial information and semantic information of the image are continuously fused, which may cause the output prediction of the last fully connected layer to be not comprehensive enough, such as Figure 3 As shown in the figure, the present application uses the feature pyramid network (FPN) in the path aggregation network (PAN) to transmit the semantic information of the high-level feature map corresponding to each image to the bottom layer feature map. In order to further strengthen the feature map of the high-level semantic strong image, the spatial information of the bottom layer feature map is continuously transmitted to the high-level feature map, so that the semantic information and spatial information of each layer of feature map are fused. In this way, each feature map not only has high-level semantic information but also has strong spatial information.

[0073] Step four, introduce spatial pyramid pooling (SPP), fix the width and height of the output of each fused feature map through a three-channel dynamic max pooling layer, and then fix the depth through a 1*1 convolution kernel to generate a multi-layer fixed width, height, and depth specification feature map, thereby realizing weight sharing of the fully connected layer; as Figure 4 shown;

[0074] Because the output of each feature map is different, if different fully connected layers are added to them, it will cause a large increase in computational load. In order to avoid this defect, spatial pyramid pooling (SPP) is used to change the width of each output through a three-channel dynamic max pooling layer. Finally, through a 1*1 point-by-point convolution, their depth is standardized. Form multiple feature maps with the same width, height, and depth. Realize weight sharing of the fully connected layer.

[0075] After the Pan network, multiple feature maps to be predicted will appear, and these features all need to be connected with the fully connected network to obtain the similarity score between the final group of images. If multiple different fully connected layers are directly used for multiple feature maps, the model parameter size will increase significantly, which does not conform to the original design of the convolutional neural network. Therefore, a Spatial pyramid Pooling is introduced in the model to generate a uniform size feature map using a dynamics max pooling layer, and then a 1*1 convolution kernel is used to generate a fixed depth feature map. To adapt to feature maps of different input sizes. Subsequently, the fully connected layer can be assimilated, and the parameters of the fully connected network are shared in the prediction of each feature map. This is also very necessary in image similarity prediction, because the comparison of images refers to the comparison of one picture with tens of thousands or even millions of pictures. The large model size will cause the model design to be unable to be applied in actual scenarios. This is from the perspective of image similarity prediction. If the network has high accuracy requirements but low FPS requirements, the dynamics max pooling layer can be replaced by an inverse convolution layer to ensure that the sizes of the model feature maps are the same. In addition, SPP combines multi-level feature information, and the model can maintain stable performance when facing inputs of different resolutions, thereby improving the overall robustness.

[0076] Step five, after splicing the normalized feature maps according to the efficiency weight of the task machine, input them into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity of the preprocessed image and the target image, and obtain the final prediction result;

[0077] In step four, the weight sharing of the fully connected layer is realized. At this time, the multiple feature maps can be aligned and added or stacked and spliced according to the efficiency weight of the machine. Because the change of the convolutional layer is slightly insufficient compared with the fully connected layer, as shown in the following formula: Figure 5 After splicing, input them into a 1024 fully connected layer first to strengthen the nonlinear expression ability of the network, then perform absolute value difference (Manhattan norm) on the outputs of the two pictures, and finally select the output layer network according to the task type. The above is used for judging whether the pictures are similar or not, so a single neuron sigmod classification output layer is directly used after each picture is processed by the same shared weight neural network (twin neural network) to distinguish whether the images are similar images.

[0078] In step one, the received image is color normalized to obtain a normalized image. The method includes:

[0079] The R, G, and B channel pixel mean and standard deviation of the images divided into a training set in the target library are calculated respectively;

[0080] Based on the pixel mean and standard deviation, the received image color is normalized to obtain a normalized image:

[0081]

[0082] where X is the pixel point of the normalized image, x is the pixel point of the received image, is the pixel mean; and σ is the standard deviation.

[0083] In image similarity matching, the brightness and color difference of the image often leads to fluctuations in the matching result. In order to reduce the influence of these external factors on the matching accuracy, color normalization is a common preprocessing method. By normalizing the RGB channels of the image respectively, the color difference under different shooting conditions, such as light changes, camera settings, and the like, can be effectively eliminated, so that the core features of the image are more prominent and consistent, thereby improving the accuracy of similarity matching. For each channel (R, G, B), the pixel value mean and standard deviation (σ) of the entire training data set are calculated, and then all pixel values are normalized Through the normalization of the mean and standard deviation, the brightness and color difference of the image will be smoothed, so that the image still has a high similarity under different light and shooting conditions, which effectively reduces the interference of the external environment on the image similarity matching. In addition, the input features of the image are more standardized, and the stability and robustness of the model when processing different images are also improved. Especially in deep learning models, the normalized input can speed up the convergence speed of the model, reduce the training time, and improve the matching effect in practical applications.

[0084] In step one, the edge of the normalized image is preprocessed using an interpolation method or an image edge reduction method to obtain a preprocessed image, and the method comprises:

[0085] Setting a length-width threshold for the image edge;

[0086] If the length-width of the normalized image is less than the length-width threshold, then an interpolation method is used to expand the edge of the normalized image by pixel point 0, and the expansion is stopped when the set length-width threshold is reached to obtain a preprocessed image;

[0087] If the length-width of the normalized image is greater than the length-width threshold, then an image edge reduction method is used to crop the edge pixel points of the normalized image, and the cropping is stopped when the set length-width threshold is reached to obtain a preprocessed image;

[0088] The traditional pre-processing image directly stretches or compresses the image in equal proportion, which destroys the scale invariant feature of the neural network in machine learning, the object in the image changes in scale, and the subsequent image space information is destroyed, so the image is not directly stretched, but interpolation or image edge reduction method. In the present application, the length and width threshold of the image edge needs to be set, such as 224*224; 1, the edge of the pixel point 0 is expanded for the image smaller than 224*224; 2, the edge deletion method is carried out for the image exceeding the length and width threshold range, that is, the edge pixel point is cropped. This pre-processing scheme is from the image similarity comparison direction, so it does not pay too much attention to the edge information of the image. Compared with the center point information, the edge information has relatively less information entropy.

[0089] In step two, the pre-processed image and the target image are simultaneously input into the twin neural network, and the method for obtaining the multi-layer feature map corresponding to each image after convolution by the neck network includes:

[0090] The pre-processed image and the target image searched in the target library are simultaneously input into the twin neural network, and the image width is continuously reduced by 1 / 2 and the depth is strengthened by 2 times after convolution by the neck network, so as to obtain five-layer feature maps corresponding to each image.

[0091] As shown in Figure 2 The pre-processed image and the target image are simultaneously input into the twin neural network, and the image width is continuously reduced by 1 / 2 and the depth is strengthened by 2 times after convolution by the neck network (1*1, depth, 0 padding, Relu convolution layer; 3*3 depth*2 1 padding, Relu convolution layer; 1*1 depth*2 0 padding Relu convolution layer). Five feature maps are obtained (64*112*112; 256*56*56; 1024*14*14; 2048*7*7), the semantic information of which is continuously strengthened from left to right, and the spatial information is continuously fused. Because the network layer is too deep, the residual block is used to connect the initial and the end of each feature map, which prevents the neurons of the deep network from being confused. The traditional residual network ResNet network directly uses the end 2048*7*7 feature map for prediction because it has the strongest semantic information. However, this also leads to the neglect of spatial information of the image. In order to consider enough spatial information when comparing two images, the two images need to be fused in step three after entering the network.

[0092] In step four, the three-channel dynamic max pooling layer can be replaced by a deconvolution layer.

[0093] In step five, the method for splicing the normalized feature maps according to the efficiency weight of the task machine comprises:

[0094] According to the efficiency weight of the task machine, the normalized feature maps of each channel are flattened into one-dimensional vectors in channel order, and the one-dimensional vectors of all channels are spliced in depth; or the one-dimensional vectors of all channels are spliced in depth in a stacking manner.

[0095] In the last stage of the deep learning model, after multiple convolution and pooling operations, the input image is converted into a high-dimensional feature map. These feature maps contain rich information of the image, including spatial features and semantic features. However, in order to pass these information to the fully connected layer for final classification or similarity calculation, the present application needs to flatten the high-dimensional feature map into a one-dimensional vector. The feature maps of each channel are unfolded into one-dimensional vectors in channel order, and the vectors of all channels are spliced together. Through this operation, the feature map in two-dimensional space is converted into a long vector, which retains all the information in the feature map and provides input for the subsequent fully connected layer. This flattening operation simplifies the structure of the model and facilitates the operation in similarity calculation.

[0096] In step five, after the normalized feature maps are spliced according to the efficiency weight of the task machine, the spliced normalized feature maps are input into a fully connected layer, and the similarity between the preprocessed image and the target image is obtained by selecting a corresponding output layer network according to the type of the task, to obtain the final prediction result.

[0097] The spliced normalized feature maps are input into a fully connected layer with 1024 neurons, the output of the preprocessed image and the target image are compared by Manhattan distance, and the similarity between the preprocessed image and the target image is obtained by selecting a single neuron sigmod classification output layer network according to the type of the task, to obtain the final prediction result.

[0098] After the feature maps are flattened into one-dimensional vectors, the next step is to further process these features through the fully connected layer to generate the final prediction result. First, because the non-linear representation ability of the convolutional neural network is weaker than that of the fully connected layer, a fully connected layer with 1024 neurons is added to strengthen its non-linear ability. Finally, a result fully connected layer is followed, which mainly maps the high-dimensional feature vector to the target space, so as to realize classification, regression or similarity calculation. According to the type of the task, a suitable activation function is selected, because the prediction of similarity belongs to classification and the prediction result is between 0 and 1, therefore a single neuron design mode is adopted, and the neuron adopts sigmoid activation function.

[0099] Application examples

[0100] The government subsidizes the service personnel for the old, in order to ensure the authenticity and effectiveness of the service, the staff need to collect and upload the on-site images before, during and after the service for audit.

[0101] But because the number of service orders is huge, it is very time-consuming to manually review the images; and because the illegal operation is simple and low-cost, it leads to the emergence of a large number of fake work orders, such as: repeatedly uploading images, taking multiple pictures at a time and uploading them in different time periods, or uploading a small number of ps images to meet the service order demand.

[0102] In order to solve the above problems, the technical scheme provided by the present application is actually applied. First, all uploaded images are extracted by the deep learning model provided by the present application to remove the top layer, and stored in the vector database to form a large service order image library. Because the vector data matching scheme provided by the vector database is the cos distance, and the cos distance will retain more pictures when screening, combined with the efficient retrieval capability of the vector database, the cos distance of a large number of pictures is initially screened, and a small number of high similarity pictures are screened out. Similarity matching prediction is used to assist the service personnel to complete the authenticity detection under the condition of a small amount of discrimination. According to the technical scheme provided by the present application and the recall ability of the vector database, the similarity prediction of more than 5w images can be completed each time, and the resource demand is controlled at single nividia3080Ti. Make small and medium-sized companies able to apply machine learning model to effectively improve the work efficiency of staff. The specific application examples are as follows:

[0103] The technical scheme disclosed by the present application is used on an online city-level platform.

[0104] As shown in Figure 6 , the right side is the received image, and the left side is the target image searched by the target library.

[0105] Based on nividia 3080Ti training comparison;

[0106] Basic settings: learning rate is 0.0001, optimizer is dynamic learning rate optimizer, data set contains 4000 groups of negative example groups and 3000 groups of positive example groups. 0.1 dropout is used. Figure 7 In the middle, the left side is the same example comparison group without using this scheme, and the right side is the effect group after using it.

[0107] As shown in Figure 8 , the correct rate change graph without using this scheme: Learning_rate=0.0001, gradient optimizer is Adam. The parameter initialization is xavier.

[0108] Based on the speed comparison of NVidia 3080Ti, the increase of the spatial feature pyramid, and the comprehensive prediction of multiple outputs after sharing weights do not slow down the efficiency too much. The efficiency when not used is as shown in Figure 9 ; the efficiency after use is as shown in Figure 10 .

[0109] Each prediction is twice because the first prediction is a certain cache used by PyTorch in use, and the second is directly using the cache.

[0110] In summary, the multi-scale feature extraction method for strengthening image similarity matching provided by the embodiment one solves the following problems:

[0111] 1. In the field of deep learning, the feature extraction method is difficult to guarantee the matching accuracy and robustness when dealing with complex images, different scales and rotation changes, and recognizing different scale objects, and cannot adapt to different sizes of images.

[0112] 2. The existing multi-scale feature extraction method has the problems of high complexity and large computing resource demand in the feature fusion process, which affects the efficiency of actual application.

[0113] The beneficial effects of the embodiment of the application are:

[0114] The PAN extracts and fuses the semantic information of the high-level feature map and the spatial information of the bottom layer feature map, so that the application can still maintain stable performance when facing different resolutions and complex scenes.

[0115] The fusion of SPP and PAN perfectly reduces the problem of large model complexity and the resulting large decline in model prediction and training efficiency caused by the PAN network. The computing cost of the model is effectively reduced, so that in the environment of limited computing resources, efficient image similarity recognition can still be realized.

[0116] This technology is not only suitable for similarity recognition tasks, but also can be extended to other deep learning tasks that need to process multi-scale information, such as target detection, image classification, etc., and has high application universality.

[0117] Due to the combination of SPP and PAN, the image processing capability of the model under different resolutions has been significantly enhanced, reducing the identification accuracy fluctuation caused by the change of image resolution, and effectively improving the stability and robustness of the model.

[0118] Embodiment two

[0119] The embodiment two of the application provides a multi-scale feature extraction system for strengthening image similarity matching, comprising:

[0120] a preprocessing module configured to normalize colors of the received image to obtain a normalized image, and to perform edge preprocessing on the normalized image using an interpolation method or an image edge reduction method to obtain a preprocessed image;

[0121] a feature extraction module configured to input the preprocessed image and the target image into a twin neural network simultaneously, perform convolution on the preprocessed image and the target image through a neck network to obtain a plurality of layer feature maps corresponding to each image, and perform residual connection on an initial position of each layer feature map and an end position of an upper layer feature map using a residual block;

[0122] a fusion module configured to transmit semantic information of a high layer feature map of each image after the residual connection to a bottom layer feature map through a feature pyramid network (FPN) in a path aggregation network (PAN), and then transmit spatial information of the bottom layer feature map to the high layer feature map, so that semantic information and spatial information of each layer feature map are fused to obtain a fused feature map;

[0123] a normalization module configured to introduce spatial pyramid pooling (SPP) to fix a width and a height of an output of each layer fused feature map through a three-channel dynamic max pooling layer, and then fix a depth of the output through a 1*1 convolution kernel to generate a plurality of layer fixed-length-width-depth normalized feature maps, so as to realize weight sharing of a full connection layer;

[0124] a similarity obtaining module configured to splice the normalized feature maps according to efficiency weights of task machines, input the normalized feature maps to a full connection layer, select a corresponding output layer network according to a task type, obtain a similarity between the preprocessed image and the target image, and obtain a final prediction result.

[0125] Embodiment three

[0126] Embodiment three of the present application provides a multi-scale feature extraction device for strengthening image similarity matching, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method provided in embodiment one when executing the computer program.

[0127] Embodiment four

[0128] Embodiment four of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method provided in embodiment one.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and computer readable storage medium can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0130] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for multi-scale feature extraction for enhanced image similarity matching, the method comprising: The method comprises: Step one, color normalization of the received image to obtain a normalized image, and using an interpolation method or an image edge reduction method to pre-process the edge of the normalized image to obtain a pre-processed image; Step two, inputting the pre-processed image and the target image into a twin neural network at the same time, performing convolution through a neck network to obtain a plurality of layer feature maps corresponding to each image; using a residual block to perform residual connection on the initial position of each layer feature map and the end position of the upper layer feature map; Step three, using a feature pyramid network (FPN) in a path aggregation network (PAN) to transmit the semantic information of the high layer feature map of each image after residual connection to the bottom layer feature map, and then transmitting the spatial information of the bottom layer feature map to the high layer feature map, so that the semantic information and the spatial information of each layer feature map are fused to obtain a fused feature map; Step four, introducing spatial pyramid pooling (SPP) to fix the width and height of the output of each layer fused feature map through a three-channel dynamic maximum pooling network (dynamics max pooling layer), and then fixing the depth of the output through a 1*1 convolution kernel to generate a plurality of layer fixed width, height and depth specification feature maps, so as to realize the weight sharing of the full connection layer; Step five, after splicing the specification feature maps according to the efficiency weight of the task machine, inputting them into the full connection layer, selecting the corresponding output layer network according to the task type, obtaining the similarity of the pre-processed image and the target image, and obtaining the final prediction result.

2. The method of claim 1, wherein, In step one, the method for color normalization of the received image to obtain a normalized image comprises: Respectively calculating the R, G and B channel pixel mean and standard deviation of the images divided into a training set in the target library; Based on the pixel mean and standard deviation, the received image is color normalized to obtain a normalized image: where X is a pixel of the normalized image, x is a pixel of the received image, x - is the pixel mean; and σ is the standard deviation.

3. The method of claim 2, wherein, In step one, the method for using an interpolation method or an image edge reduction method to pre-process the edge of the normalized image to obtain a pre-processed image comprises: Setting the length and width threshold of the image edge; If the length and width of the normalized image are less than the length and width threshold, using the interpolation method of expanding the edge of the normalized image by 0 pixels to reach the set length and width threshold, and stopping the expansion to obtain the pre-processed image; If the length and width of the normalized image are greater than the length and width threshold, using the image edge reduction method of cutting the edge pixels of the normalized image to reach the set length and width threshold, and stopping the cutting to obtain the pre-processed image.

4. The method of claim 3, wherein, In step two, the method for inputting the pre-processed image and the target image into a twin neural network at the same time, performing convolution through a neck network, and obtaining a plurality of layer feature maps corresponding to each image comprises: Inputting the pre-processed image and the target image searched in the target library into the twin neural network at the same time, performing convolution through the neck network, constantly reducing the image width by 1 / 2, and intensifying the depth by 2 times to obtain five layer feature maps corresponding to each image.

5. The method of claim 1, wherein, In step four, the three-channel dynamic max pooling layer is replaced by a reverse convolution layer.

6. The method of claim 1, wherein, In step five, the method for obtaining the final prediction result includes: inputting the spliced normalized feature map into a 1024 full connection layer, performing Manhattan distance alignment absolute value difference on the outputs of the preprocessed image and the target image, selecting a single neuron sigmod classification output layer network according to the task type, obtaining the similarity between the preprocessed image and the target image, and obtaining the final prediction result.

7. The method of claim 1 or 6, wherein, In step five, the method for splicing the normalized feature map according to the efficiency weight of the task machine includes: According to the efficiency weight of the task machine, the normalized feature map of each channel is flattened into a one-dimensional vector in the order of the channel, and the one-dimensional vectors of all channels are deeply spliced; or the deep splicing is performed in a stacking manner.

8. A multi-scale feature extraction system that enhances image similarity matching, characterized by, The method includes: a preprocessing module configured to normalize the color of the received image to obtain a normalized image, and to preprocess the edge of the normalized image using an interpolation method or an image edge disassembly method to obtain a preprocessed image; a feature extraction module configured to input the preprocessed image and a target image into a twin neural network, perform convolution through a neck network to obtain a plurality of layer feature maps corresponding to each image, and use a residual block to perform residual connection on the initial position of each layer feature map and the end position of the upper layer feature map; a fusion module configured to use a feature pyramid network (FPN) in a path aggregation network (PAN) to transmit semantic information of a high layer feature map of each image after residual connection to a bottom layer feature map, and then transmit spatial information of the bottom layer feature map to the high layer feature map, so that the semantic information and the spatial information of each layer feature map are fused to obtain a fused feature map; a normalization module configured to introduce a spatial pyramid pooling (SPP) to fix the width and height of the output of each layer fused feature map through a three-channel dynamic max pooling network, and then fix the depth of the output through a 1*1 convolution kernel to generate a plurality of layer normalized feature maps with fixed length, width and depth, so as to realize weight sharing of a full connection layer; a similarity obtaining module configured to splice the normalized feature map according to the efficiency weight of the task machine, input the spliced normalized feature map into a full connection layer, select a corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and obtain the final prediction result.

9. A multi-scale feature extraction apparatus for enhancing image similarity matching, characterized by, The device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program. The device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images