Multi-scale feature extraction method, system and device for enhancing image similarity matching and storage medium

Abstract

The invention discloses a multi-scale feature extraction method, system and device for enhancing image similarity matching and a storage medium, belongs to the field of machine learning, and solves the problem that the similarity matching precision and efficiency of complex images are low. Comprising the following steps: normalizing the color of a received image and obtaining a pre-processed image by using an interpolation method or an image edge splitting method; inputting the pre-processed image and the target image into a twin neural network at the same time, performing convolution through a check network to obtain a multi-layer feature map corresponding to each image, and performing residual connection; using PAN to obtain a fusion feature map of semantic information and spatial information fusion; sPP is introduced to generate a plurality of layers of standard feature maps with fixed length, width and depth, and weight sharing of a full connection layer is realized; and according to the efficiency weight of a task machine, splicing the standard feature maps, inputting the spliced standard feature maps to a full connection layer, and selecting a corresponding output layer network according to a task type to obtain a prediction result. According to the method, the precision and efficiency of similarity matching of the complex images are improved through combination of the SPP and the PAN.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning and relates to a multi-scale feature extraction method, system, device and storage medium for enhancing image similarity matching. Background Art

[0002] In the field of computer vision, image similarity matching is an important basic technology, which is widely used in many fields such as image retrieval, target detection, image classification and image stitching.

[0003] The core of image similarity matching is to effectively extract and compare features in images to determine the similarity between two images. With the rapid increase in image data and the diversification of application scenarios, traditional image similarity matching methods are gradually unable to meet actual needs in terms of accuracy and efficiency. Therefore, how to design an efficient and accurate image similarity matching method has become one of the hot spots and difficulties in current research.

[0004] With the rise of deep learning technology, convolutional neural networks (CNNs) have shown powerful capabilities in image feature extraction and similarity matching. Through multi-layer convolution and pooling operations, CNNs can extract image features and train them on large-scale image datasets to obtain more discriminative feature representations.

[0005] Existing image similarity matching methods mainly rely on feature extraction technology, which converts images into feature vectors and judges the similarity of images by measuring the similarity of feature vectors. Traditional feature extraction methods mainly include pixel-based feature extraction methods, statistical-based feature extraction methods, and feature extraction methods based on local feature descriptors, such as SIFT and SURF. Traditional feature extraction methods have solved the problem of image similarity matching to a certain extent, but when dealing with complex images where the target subject is not unique, because their interpretable algorithms cannot be directly transplanted into deep learning models and cannot adapt to different scales and rotation changes, traditional feature extraction methods cannot meet the requirements of efficient and accurate image similarity matching for complex images.

[0006] However, except for end-to-end models such as Unet for generating images, the top-level model output layers are all fully connected layers, which restrict the image size, while the images produced in actual production are not restricted, so simple image stretching will cause pixel damage. And if only a single-scale feature extraction method is used to extract multi-scale objects in the image, there are still certain limitations.

[0007] For example, in the prior art, a patent application with publication number CN117911731A mentions a method based on multi-scale feature fusion. This method combines the attention mechanism in the Transformer model and performs multi-scale fusion through batch matrix multiplication Bmm to generate a set of K, Q and V features of the image to be matched; through the multi-attention mechanism, the Q feature is used as the backbone feature, and the K and V features are fused as key features to extract image features at different scales and fuse them. This method improves the extraction of multi-scale features. However, the method of the attention calculation mechanism has a certain complexity in the feature fusion process and has a high demand for computing resources. Although the multi-head attention mechanism can be applied to improve efficiency, this will also lead to a further increase in the demand for computing resources.

[0008] Therefore, how to design a multi-scale feature extraction technology suitable for image similarity matching to improve the matching accuracy and efficiency is an important problem that needs to be solved in the current technical field. Summary of the invention

[0009] In order to solve the above technical problems, the present application provides a multi-scale feature extraction method, system, device and storage medium for enhancing image similarity matching. This solution introduces two technologies, path aggregation network (PAN) and spatial feature pyramid pooling (SPP). By extracting and aggregating image features at multiple scales, the robustness and accuracy of the image processing algorithm are enhanced. This solves the problem that traditional feature extraction methods are often difficult to capture sufficient detail information when facing problems such as image scale changes, complex backgrounds and object deformations, resulting in subsequent image similarity matching being very limited and unable to well sense phenomena such as image stretching. In addition, the existing multi-scale feature extraction methods have the problems of high complexity and large computing resource requirements in the feature fusion process. This invention aims to propose a method with strong versatility, centered on convolutional neural networks, to enhance image multi-scale information extraction and improve the accuracy and efficiency of image similarity matching.

[0010] The purpose of the present invention is specifically achieved through the following technical solutions:

[0011] The present invention provides a multi-scale feature extraction method for enhancing image similarity matching, the method comprising:

[0012] Step 1: normalize the color of the received image to obtain a normalized image, and preprocess the edge of the normalized image using an interpolation method or an image edge subtraction method to obtain a preprocessed image;

[0013] Step 2: Input the preprocessed image and the target image into the twin neural network at the same time, and perform convolution through the neck network to obtain the multi-layer feature map corresponding to each image; use the residual block to perform residual connection between the initial position of each layer of the feature map and the terminal position of the upper layer feature map;

[0014] Step 3: Using the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN), the semantic information of the high-level feature map of each image after residual connection is transferred to the low-level feature map, and then the spatial information of the low-level feature map is transferred to the high-level feature map, so that the semantic information and spatial information of each layer of feature map are fused to obtain a fused feature map;

[0015] Step 4: Introduce spatial pyramid pooling (SPP), fix the width and height of the output of each layer of fusion feature map through a three-channel dynamics max pooling layer, and then fix the output depth through a 1*1 convolution kernel to generate multiple layers of standard feature maps with fixed length, width and depth, thereby realizing weight sharing of the fully connected layer;

[0016] Step 5: After splicing the standard feature map according to the efficiency weight of the task machine, input it into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and get the final prediction result.

[0017] In step 1, the color of the received image is normalized to obtain a normalized image, including:

[0018] Calculate the pixel mean and standard deviation of the R, G, and B channels of the images in the target library divided into the training set respectively;

[0019] Based on the pixel mean and standard deviation, the received image color is normalized to obtain a normalized image:

[0020]

[0021] Among them, X is the pixel of the normalized image, x is the pixel of the received image, is the pixel mean; σ is the standard deviation.

[0022] In step 1, the normalized image edge is preprocessed using an interpolation method or an image edge subtraction method, and the method for obtaining the preprocessed image includes:

[0023] Set the length and width threshold of the image edge;

[0024] If the length and width of the normalized image are less than the length and width threshold, the edge of the normalized image is expanded to pixel 0, and the expansion is stopped after reaching the set length and width threshold, to obtain the preprocessed image;

[0025] If the length and width of the normalized image are greater than the length and width threshold, the edge pixels of the normalized image are cropped and the image edge subtraction method is used to stop cropping when the set length and width threshold is reached to obtain a preprocessed image.

[0026] In step 2, the preprocessed image and the target image are simultaneously input into the twin neural network, and convolution is performed through the neck network to obtain a multi-layer feature map corresponding to each image. The method includes:

[0027] The preprocessed image and the target image retrieved from the target library are simultaneously input into the twin neural network, and convolved through the neck network to continuously reduce the image width by 1 / 2 and enhance the depth by 2 times, so as to obtain a five-layer feature map corresponding to each image.

[0028] In step 4, the three-channel dynamics max pooling layer can be replaced with a reverse convolutional layer.

[0029] In step 5, the method of splicing the standard feature map according to the efficiency weight of the task machine includes:

[0030] According to the efficiency weight of the task machine, the canonical feature map of each channel is flattened into a one-dimensional vector in the channel order, and the one-dimensional vectors of all channels are deeply spliced; or they are deeply spliced ​​in a stacking manner.

[0031] In step 5, after splicing the standard feature map according to the efficiency weight of the task machine, the map is input into the fully connected layer, and the corresponding output layer network is selected according to the task type to obtain the similarity between the preprocessed image and the target image. The method for obtaining the final prediction result includes:

[0032] The spliced ​​canonical feature map is input into a 1024 fully connected layer, and the output of the preprocessed image and the target image are aligned and the absolute value difference is calculated through the Manhattan distance. The classification output layer network of the single neuron sigmoid is selected according to the task type to obtain the similarity between the preprocessed image and the target image, and the final prediction result is obtained.

[0033] The present invention also provides a multi-scale feature extraction system for enhancing image similarity matching, comprising:

[0034] A preprocessing module is used to normalize the color of the received image to obtain a normalized image, and to preprocess the edge of the normalized image using an interpolation method or an image edge subtraction method to obtain a preprocessed image;

[0035] The feature extraction module is used to input the preprocessed image and the target image into the twin neural network at the same time, and convolve them through the neck network to obtain the multi-layer feature map corresponding to each image; the residual block is used to perform residual connection between the initial position of each layer of the feature map and the terminal position of the upper layer feature map;

[0036] The fusion module is used to use the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN) to transfer the semantic information of the high-level feature map of each image after residual connection to the bottom-level feature map, and then transfer the spatial information of the bottom-level feature map to the high-level feature map, so that the semantic information and spatial information of each layer of the feature map are fused to obtain a fused feature map;

[0037] The standardization module is used to introduce spatial pyramid pooling (SPP), fix the width and height of the output of each layer of fusion feature map through a three-channel dynamics maxpooling layer, and then fix the output depth through a 1*1 convolution kernel to generate multiple layers of standard feature maps with fixed length, width and depth, thereby realizing weight sharing of the fully connected layer;

[0038] The similarity acquisition module is used to splice the standard feature map according to the efficiency weight of the task machine, input it into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and get the final prediction result.

[0039] The present invention also provides a multi-scale feature extraction device for enhancing image similarity matching, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0040] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0041] The present invention provides a multi-scale feature extraction method, system, device and storage medium for enhancing image similarity matching, which solves the following problems:

[0042] 1. In the field of deep learning, feature extraction methods have difficulty ensuring matching accuracy and robustness when dealing with complex images, different scales and rotation changes, and when facing the recognition of objects of different scales, and cannot adapt to images of different sizes.

[0043] 2. The existing multi-scale feature extraction methods have problems of high complexity and large computing resource requirements in the feature fusion process, which affects the efficiency of practical applications.

[0044] The beneficial effects of the present invention are:

[0045] The feature extraction of PAN integrates the semantic information of high-level feature maps and the spatial information of low-level feature maps, so that the present invention can still maintain stable performance when facing different resolutions and complex scenes.

[0046] The integration of SPP and PAN perfectly reduces the problem that the PAN network brings about a significant increase in model complexity, which leads to a significant decrease in model prediction efficiency and training efficiency. It effectively reduces the computational cost of the model, making it possible to achieve efficient image similarity recognition in an environment with limited computing resources.

[0047] This technology is not only suitable for similarity recognition tasks, but can also be extended to other deep learning tasks that require processing multi-scale information, such as target detection, image classification, etc., and has a wide range of applications.

[0048] Due to the combination of SPP and PAN, the model's image processing capabilities at different resolutions have been significantly enhanced, reducing the fluctuations in recognition accuracy caused by changes in image resolution and effectively improving the stability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention is further described in detail below based on the accompanying drawings and embodiments.

[0050] Figure 1 It is a schematic diagram of a multi-scale feature extraction method for enhancing image similarity matching using resnetv50 as an example provided by an embodiment of the present invention.

[0051] Figure 2 It is a schematic diagram of an image provided by an embodiment of the present invention that is convolved through a neck network and residually connected.

[0052] Figure 3 It is a schematic diagram of the FPN provided by an embodiment of the present invention fusing the semantic information and spatial information of each layer of feature graph.

[0053] Figure 4 It is a schematic diagram of a standard feature map of multiple layers with fixed length, width and depth generated by SPP provided in an embodiment of the present invention.

[0054] Figure 5 It is a schematic diagram of obtaining the similarity between a preprocessed image and a target image provided by an embodiment of the present invention.

[0055] Figure 6It is a schematic diagram of applying the technical solution disclosed in the present invention provided by the embodiment of the present invention on an online city-level platform.

[0056] Figure 7 It is a training comparison diagram provided by an embodiment of the present invention.

[0057] Figure 8 It is a schematic diagram of the change in accuracy provided by an embodiment of the present invention without using the technical solution disclosed in the present invention.

[0058] Fig. 9 It is a schematic diagram of the efficiency provided by an embodiment of the present invention without using the technical solution disclosed in the present invention.

[0059] Fig.10 It is a schematic diagram of the efficiency of the technical solution disclosed in the present invention provided in an embodiment of the present invention.

[0060] in, Figures 2 to 5 yes Figure 1 Enlarged views of each step in the process. DETAILED DESCRIPTION

[0061] Embodiment 1

[0062] Embodiment 1 of the present invention provides a multi-scale feature extraction method for enhancing image similarity matching, such as Figure 1 As shown, the method includes:

[0063] Step 1: normalize the color of the received image to obtain a normalized image, and preprocess the edge of the normalized image using an interpolation method or an image edge subtraction method to obtain a preprocessed image;

[0064] The work of strengthening image similarity matching is often affected by different shooting equipment or lighting, which may lead to deviations in the results. In order to effectively eliminate lighting changes, camera settings, and color differences under different shooting conditions, the image channel pixels are normalized and the image pixels are converted into pixel changes. In addition, traditional image preprocessing only compresses or proportionally stretches the image, which also causes the problem of scale deformation. From the perspective of image similarity matching, the scale of the image needs to be kept unchanged in the image preprocessing stage. Therefore, the embodiment of the present invention uses interpolation to expand the image and image edge subtraction to crop the image. To ensure the stability of the length and width.

[0065] Step 2: Input the preprocessed image and the target image into the twin neural network at the same time, and perform convolution through the neck network to obtain the multi-layer feature map corresponding to each image; use the residual block to perform residual connection between the initial position of each layer of the feature map and the terminal position of the upper layer feature map;

[0066] To compare the similarity of two images, we need to extract their features in the deep neural network. In order to ensure that the features they extract are from the same level, we inject machine learning (ResNet) into the twin neural network and share weights between the two identical ResNet network layers. This ensures that the observation surface of a group of images is the same when comparing two images. The ResNet model shared on HuggingFace is used for image classification. But we just want to extract image features through ResNet, so we need to remove the top layer in the original ResNet. Then freeze the neurons of the two non-terminal layers of image feature extraction. Because when comparing image similarity, the model is easily overfitted by a small number of non-fixed task objects, this can also effectively prevent the model from overfitting during the training process.

[0067] Step 3: Using the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN), the semantic information of the high-level feature map of each image after residual connection is transferred to the low-level feature map, and then the spatial information of the low-level feature map is transferred to the high-level feature map, so that the semantic information and spatial information of each layer of feature map are fused to obtain a fused feature map;

[0068] In deep learning models, especially in image processing tasks, as the convolutional layers go deeper, the spatial information of the image will gradually be replaced by semantic information, resulting in the model's incomplete understanding of the image in the final prediction output. In order to better capture and utilize the multi-level information in the image, Path Aggregation Network (PAN) is an effective structure that can strengthen the underlying spatial feature information while maintaining high-level semantic information.

[0069] Specifically, PAN transfers high-level semantic information to the underlying feature map through the design of Feature Pyramid Network (FPN). This transfer method ensures that the underlying feature map not only has the original high-resolution spatial information, but also incorporates high-level semantic features, thereby improving the overall expressiveness of the feature map.

[0070] In order to further enhance the model's ability to capture image details, PAN will also pass the underlying feature maps to higher layers step by step. This two-way transmission mechanism allows the feature maps of each layer to not only retain the information of the corresponding layer, but also integrate multi-level semantic and spatial information. This multi-level information fusion allows the model to more comprehensively consider the important features in the image when making the final prediction, improving the accuracy of the prediction and the robustness of the model.

[0071] By introducing PAN, the model can effectively transfer information between different layers, so that the feature map of each layer retains spatial details while also having rich semantic information. The benefit of this is that the model can understand the image content more comprehensively when processing complex images, thereby improving the accuracy of tasks such as classification, detection or segmentation. This structure not only enhances the expressive power of the model, but also reduces the prediction error caused by information loss, making the model more stable and reliable in practical applications.

[0072] As mentioned in step 2, in the continuous convolution process, the spatial information and semantic information of the image are constantly melted, which will lead to the problem of insufficient consideration when predicting the output of the final fully connected layer, such as Figure 3 As shown in the figure, the present invention uses the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN) to transfer the semantic information of the high-level feature map corresponding to each image to the underlying feature map. In order to further strengthen the feature map of the high-level semantically strong image, the spatial information of the underlying feature map is further transmitted to the high-level feature map, so that the semantic information and spatial information of each layer of feature map are fused. In this way, each feature map not only has high-level semantic information but also has strong spatial information.

[0073] Step 4: Introduce spatial pyramid pooling (SPP), fix the width and height of the output of each layer of fusion feature map through a three-channel dynamics max pooling layer, and then fix the depth through a 1*1 convolution kernel to generate multiple layers of standard feature maps with fixed length, width and depth, thereby realizing weight sharing of the fully connected layer; Figure 4 As shown;

[0074] Because the output of each feature map is different, if different fully connected layers are added to them, the amount of calculation will increase significantly. In order to avoid this defect, spatial pyramid pooling is used to change the width of each output through a three-channel dynamic maximum pooling network. Finally, their depths are normalized through 1*1 point-by-point convolution. Multiple feature maps with exactly the same width, height, and depth are formed. Weight sharing of the fully connected layer is achieved.

[0075] After passing through the Pan network, there will be multiple feature maps to be predicted. These features need to be connected to the fully connected network to obtain the final similarity score between a set of images. If multiple different fully connected layers are used directly for multiple feature maps, the scale of model parameters will increase significantly, which is not in line with the original design of the convolutional neural network. Therefore, Spatial pyramid Pooling is introduced in the model to use the dynamics max pooling layer to generate feature maps of uniform size, and then use the 1*1 convolution kernel to generate feature maps of fixed depth. To adapt to feature maps of different input sizes. The fully connected layer can be assimilated later, and the parameters of the fully connected network can be shared in the prediction of each feature map. This is also very necessary in image similarity prediction, because image comparison refers to the comparison of one picture with 100,000 or even millions of pictures. Too large a model scale will make the design of the model unable to be applied in actual scenarios. This is from the perspective of image similarity prediction. If the FPS requirement is not high, but the accuracy requirement is extremely high, the dynamics max pooling layer can be replaced with a reverse convolution layer to make the size of each model feature map the same. In addition, SPP combines multi-level feature information, and the model can maintain stable performance when facing inputs of different resolutions, thereby improving the overall robustness.

[0076] Step 5: After splicing the standard feature map according to the efficiency weight of the task machine, input it into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and get the final prediction result;

[0077] In step 4, the weight sharing of the fully connected layer is realized. At this time, multiple feature maps can be added and spliced ​​according to the efficiency weight of their own machine, or stacked for deep splicing. Because the changes in the convolutional layer are slightly less than those in the fully connected layer, such as Figure 5 As shown in the figure, after the splicing is completed, it is first input into a 1024 fully connected layer to enhance the nonlinear expression ability of the network, and then the absolute value difference (Manhattan norm) of the output of the two images is calculated, and finally the network of the output layer is selected according to the task type. The above is a binary classification used to determine whether the images are similar. Therefore, after each image is in the same shared weight neural network (twin neural network), a single neuron sigmoid classification output layer is directly used to identify whether the image is a similar image.

[0078] In step 1, the color of the received image is normalized to obtain a normalized image, including:

[0079] Calculate the pixel mean and standard deviation of the R, G, and B channels of the images in the target library divided into the training set respectively;

[0080] Based on the pixel mean and standard deviation, the received image color is normalized to obtain a normalized image:

[0081]

[0082] Among them, X is the pixel of the normalized image, x is the pixel of the received image, is the pixel mean; σ is the standard deviation;

[0083] In image similarity matching, the brightness and color differences of images often lead to fluctuations in matching results. In order to reduce the impact of these external factors on matching accuracy, color normalization is a commonly used preprocessing method. By normalizing the RGB channels of the image separately, the color differences caused by lighting changes, camera settings, and different shooting conditions can be effectively eliminated, making the core features of the image more prominent and consistent, thereby improving the accuracy of similarity matching. For each channel (R, G, B), calculate the mean pixel value in the entire training data set And standard deviation (σ), then normalize all pixel values By normalizing the mean and standard deviation, the brightness and color differences of the image will be smoothed, so that the image still has a high similarity under different lighting and shooting conditions, which effectively reduces the interference of the external environment on the image similarity matching. In addition, the input features of the image are more standardized, and the stability and robustness of the model when processing different images are also improved. Especially in deep learning models, the normalized input can speed up the convergence of the model, reduce training time, and improve the matching effect in practical applications.

[0084] In step 1, the normalized image edge is preprocessed using an interpolation method or an image edge subtraction method, and the method for obtaining the preprocessed image includes:

[0085] Set the length and width threshold of the image edge;

[0086] If the length and width of the normalized image are less than the length and width threshold, the edge of the normalized image is expanded to pixel 0, and the expansion is stopped after reaching the set length and width threshold, to obtain the preprocessed image;

[0087] If the length and width of the normalized image are greater than the length and width threshold, the edge pixels of the normalized image are cropped, and the image edge subtraction method is used to stop cropping when the set length and width threshold is reached to obtain a preprocessed image;

[0088] Traditional image preprocessing will directly stretch or proportionally compress the image, which will destroy the scale-invariant feature of the neural network in machine learning. The objects in the image will change in scale, which will damage the image spatial information that is subsequently maintained. Therefore, the image is not stretched directly, but interpolation or image edge subtraction is used. In the present invention, it is necessary to set the length and width threshold of the image edge, such as 224*224; 1. For images smaller than 224*224, the edge of pixel 0 is expanded; 2. For images that exceed the length and width threshold range, the edge deletion method is performed, that is, cropping is performed from the edge pixels. This preprocessing scheme starts from the direction of image similarity comparison, so it does not focus too much on the edge information of the image. Compared with the center point information, the information entropy of the edge information is relatively less.

[0089] In step 2, the preprocessed image and the target image are simultaneously input into the twin neural network, and convolution is performed through the neck network to obtain a multi-layer feature map corresponding to each image. The method includes:

[0090] The preprocessed image and the target image retrieved from the target library are simultaneously input into the twin neural network, and convolved through the neck network to continuously reduce the image width by 1 / 2 and enhance the depth by 2 times, so as to obtain a five-layer feature map corresponding to each image.

[0091] like Figure 2 As shown, the embodiment of the present invention simultaneously inputs the preprocessed image and the target image into the twin neural network, and performs convolution through the neck network (1*1, depth, 0padding, Relu convolution layer; 3*3depth*2 1padding, Relu convolution layer; 1*1depth*2 0padding Relu convolution layer) to continuously reduce the width by 1 / 2 and enhance the depth by 2 times. The five feature maps obtained are (64*112*112; 256*56*56; 1024*14*14; 2048*7*7), and the semantic information of this feature map is continuously enhanced from left to right, while the spatial information is also continuously melted. Because the network layer is too deep, it fits the design of the ResNet network itself, and uses residual blocks to perform residual connections on the initial and previous ends of each feature map to prevent confusion of neurons in an overly deep network. The traditional residual network ResNet network directly uses the terminal 2048*7*7 feature map for prediction because it has the strongest semantic information. However, this also leads to the neglect of spatial information in the image. In order to consider sufficient spatial information when comparing two images, the two images need to be fused with semantic information and spatial information in step 3 after entering the network.

[0092] In step 4, the three-channel dynamics max pooling layer can be replaced with a reverse convolutional layer.

[0093] In step 5, the method of splicing the standard feature map according to the efficiency weight of the task machine includes:

[0094] According to the efficiency weight of the task machine, the canonical feature map of each channel is flattened into a one-dimensional vector in the channel order, and the one-dimensional vectors of all channels are deeply spliced; or they are deeply spliced ​​in a stacking manner.

[0095] In the final stage of the deep learning model, after multiple layers of convolution and pooling operations, the input image is converted into a high-dimensional feature map. These feature maps contain rich information about the image, including spatial features and semantic features. However, in order to pass this information to the fully connected layer for final classification or similarity calculation, the present invention needs to flatten the high-dimensional feature map into a one-dimensional vector. The feature map of each channel is expanded into a one-dimensional vector in the order of the channel, and the vectors of all channels are spliced ​​together. Through this operation, the feature map originally in the two-dimensional space is converted into a long vector, which retains all the information in the feature map and provides input for the subsequent fully connected layer. This flattening operation simplifies the structure of the model and is convenient for operation during similarity calculation.

[0096] In step 5, after splicing the standard feature map according to the efficiency weight of the task machine, the map is input into the fully connected layer, and the corresponding output layer network is selected according to the task type to obtain the similarity between the preprocessed image and the target image. The method for obtaining the final prediction result includes:

[0097] The spliced ​​canonical feature map is input into a 1024 fully connected layer, and the output of the preprocessed image and the target image are aligned and the absolute value difference is calculated through the Manhattan distance. The classification output layer network of the single neuron sigmoid is selected according to the task type to obtain the similarity between the preprocessed image and the target image, and the final prediction result is obtained.

[0098] After the feature map is flattened into a one-dimensional vector, the next step is to further process these features through the fully connected layer to generate the final prediction result. First, because the nonlinear representation ability of the convolutional neural network is weaker than that of the fully connected layer, a 1024 fully connected layer is added to enhance its nonlinear ability. Finally, a result fully connected layer is added, whose main task is to map the high-dimensional feature vector to the target space, thereby realizing classification, regression or similarity calculation. Select the appropriate activation function according to the task type. Because the prediction of similarity belongs to classification and the prediction of the result is between 0-1, a single neuron design mode is adopted, and the neuron uses the sigmoid activation function.

[0099] Application Examples

[0100] The government provides financial subsidies to people who provide services for the elderly. To ensure that the services are authentic and effective, staff are required to collect on-site images before, during and after the service and upload them for review.

[0101] However, due to the large number of service orders, manual review of images is very time-consuming; and because illegal operations are simple and low-cost, a large number of fake work orders have emerged, such as: repeatedly uploading images, taking multiple pictures at a time and uploading them in different time periods, or uploading a small number of PS images to meet service order needs.

[0102] In order to solve the above problems, the technical solution provided by the present application was put into practical application. First, all the uploaded images are subjected to feature extraction through the deep learning model provided by the present invention that removes the top layer, and are stored in a vector database to form a huge service work order image library. Since the vector data matching solution provided by the vector database is the cos distance, and the cos distance will retain more pictures during screening, the efficient retrieval capability of the vector database is combined to perform an initial screening of a large number of pictures using the cos distance, and then a small number of highly similar pictures that have been screened are predicted for similarity matching, to assist service personnel in completing authenticity detection with a small amount of identification work. According to the technical solution provided by the present invention and the recall capability of the vector database, the similarity prediction of images far exceeding 50,000 can be completed each time, and the resource demand is controlled within a single nividia3080Ti. It enables small and medium-sized companies to apply machine learning models to effectively improve the work efficiency of their staff. Specific application examples are as follows:

[0103] The technical solution disclosed in the present invention is applied on an online city-level platform.

[0104] like Figure 6 As shown, the right side is the received image, and the left side is the target image searched by the target library.

[0105] Training comparison based on nividia 3080Ti;

[0106] Basic settings: learning rate is 0.0001, optimizer is dynamic learning rate optimizer, the dataset contains 4000 sets of negative examples and 3000 sets of positive examples. Dropout of 0.1 is used. Figure 7 In the figure, the left picture is the comparison group of the same case without using this scheme, and the right picture is the effect group after using it.

[0107] like Figure 8 As shown in the figure, the accuracy change diagram without using this solution: Learning_rate = 0.0001, the gradient optimizer is Adam, and the parameters are initialized to xavier.

[0108] Speed ​​comparison based on NVidia 3080Ti, adding spatial feature pyramid and comprehensive prediction of multiple outputs after sharing weights did not slow down the efficiency too much. The efficiency when not used is as follows Fig. 9 As shown; the efficiency after use is as Fig.10 shown.

[0109] Each prediction is made twice because PyTorch caches the first prediction and uses the cache directly for the second prediction.

[0110] In summary, the multi-scale feature extraction method for enhancing image similarity matching provided in the first embodiment of the present invention solves the following problems:

[0111] 1. In the field of deep learning, feature extraction methods have difficulty ensuring matching accuracy and robustness when dealing with complex images, different scales and rotation changes, and when facing the recognition of objects of different scales, and cannot adapt to images of different sizes.

[0112] 2. The existing multi-scale feature extraction methods have problems of high complexity and large computing resource requirements in the feature fusion process, which affects the efficiency of practical applications.

[0113] The beneficial effects of the embodiments of the present invention are:

[0114] The feature extraction of PAN integrates the semantic information of high-level feature maps and the spatial information of low-level feature maps, so that the present invention can still maintain stable performance when facing different resolutions and complex scenes.

[0115] The integration of SPP and PAN perfectly reduces the problem that the PAN network brings about a significant increase in model complexity, which leads to a significant decrease in model prediction efficiency and training efficiency. It effectively reduces the computational cost of the model, making it possible to achieve efficient image similarity recognition in an environment with limited computing resources.

[0116] This technology is not only suitable for similarity recognition tasks, but can also be extended to other deep learning tasks that require processing multi-scale information, such as target detection, image classification, etc., and has a wide range of applications.

[0117] Due to the combination of SPP and PAN, the model's image processing capabilities at different resolutions have been significantly enhanced, reducing the fluctuations in recognition accuracy caused by changes in image resolution and effectively improving the stability and robustness of the model.

[0118] Embodiment 2

[0119] Embodiment 2 of the present invention provides a multi-scale feature extraction system for enhancing image similarity matching, including:

[0120] A preprocessing module is used to normalize the color of the received image to obtain a normalized image, and to preprocess the edge of the normalized image using an interpolation method or an image edge subtraction method to obtain a preprocessed image;

[0121] The feature extraction module is used to input the preprocessed image and the target image into the twin neural network at the same time, and convolve them through the neck network to obtain the multi-layer feature map corresponding to each image; the residual block is used to perform residual connection between the initial position of each layer of the feature map and the terminal position of the upper layer feature map;

[0122] The fusion module is used to use the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN) to transfer the semantic information of the high-level feature map of each image after residual connection to the bottom-level feature map, and then transfer the spatial information of the bottom-level feature map to the high-level feature map, so that the semantic information and spatial information of each layer of the feature map are fused to obtain a fused feature map;

[0123] The standardization module is used to introduce spatial pyramid pooling (SPP), fix the width and height of the output of each layer of fusion feature map through a three-channel dynamics maxpooling layer, and then fix the output depth through a 1*1 convolution kernel to generate multiple layers of standard feature maps with fixed length, width and depth, thereby realizing weight sharing of the fully connected layer;

[0124] The similarity acquisition module is used to splice the standard feature map according to the efficiency weight of the task machine, input it into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and get the final prediction result.

[0125] Embodiment 3

[0126] Embodiment 3 of the present invention provides a multi-scale feature extraction device for enhancing image similarity matching, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method provided in the above embodiment 1 when executing the computer program.

[0127] Embodiment 4

[0128] Embodiment 4 of the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the method provided in embodiment 1 above when the computer program is executed by a processor.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and computer-readable storage media can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-scale feature extraction method for enhancing image similarity matching, characterized in that: The method includes: Step 1: normalize the color of the received image to obtain a normalized image, and preprocess the edge of the normalized image using an interpolation method or an image edge subtraction method to obtain a preprocessed image; Step 2: Input the preprocessed image and the target image into the twin neural network at the same time, and perform convolution through the neck network to obtain the multi-layer feature map corresponding to each image; use the residual block to perform residual connection between the initial position of each layer of the feature map and the terminal position of the upper layer feature map; Step 3: Using the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN), the semantic information of the high-level feature map of each image after residual connection is transferred to the low-level feature map, and then the spatial information of the low-level feature map is transferred to the high-level feature map, so that the semantic information and spatial information of each layer of feature map are fused to obtain a fused feature map; Step 4: Introduce spatial pyramid pooling (SPP), fix the width and height of the output of each layer of fusion feature map through a three-channel dynamics max pooling layer, and then fix the output depth through a 1*1 convolution kernel to generate multiple layers of standard feature maps with fixed length, width and depth, thereby realizing weight sharing of the fully connected layer; Step 5: After splicing the standard feature map according to the efficiency weight of the task machine, input it into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and get the final prediction result.

2. The method according to claim 1, characterized in that In step 1, the color of the received image is normalized to obtain a normalized image, including: Calculate the pixel mean and standard deviation of the R, G, and B channels of the images in the target library divided into the training set respectively; Based on the pixel mean and standard deviation, the received image color is normalized to obtain a normalized image: Among them, X is the pixel of the normalized image, x is the pixel of the received image, and x - is the pixel mean; σ is the standard deviation.

3. The method according to claim 2, characterized in that In step 1, the normalized image edge is preprocessed using an interpolation method or an image edge subtraction method, and the method for obtaining the preprocessed image includes: Set the length and width threshold of the image edge; If the length and width of the normalized image are less than the length and width threshold, the edge of the normalized image is expanded to pixel 0, and the expansion is stopped after reaching the set length and width threshold, to obtain the preprocessed image; If the length and width of the normalized image are greater than the length and width threshold, the edge pixels of the normalized image are cropped and the image edge subtraction method is used to stop cropping when the set length and width threshold is reached to obtain a preprocessed image.

4. The method according to claim 3, characterized in that In step 2, the preprocessed image and the target image are simultaneously input into the twin neural network, and convolution is performed through the neck network to obtain a multi-layer feature map corresponding to each image. The method includes: The preprocessed image and the target image retrieved from the target library are simultaneously input into the twin neural network, and convolved through the neck network to continuously reduce the image width by 1 / 2 and enhance the depth by 2 times, so as to obtain a five-layer feature map corresponding to each image.

5. The method according to claim 1, characterized in that Step 4 also includes replacing the three-channel dynamics max pooling layer with a reverse convolutional layer.

6. The method according to claim 1, characterized in that In step 5, after splicing the standard feature map according to the efficiency weight of the task machine, the map is input into the fully connected layer, and the corresponding output layer network is selected according to the task type to obtain the similarity between the preprocessed image and the target image. The method for obtaining the final prediction result includes: The spliced ​​canonical feature map is input into a 1024 fully connected layer, and the output of the preprocessed image and the target image are aligned and the absolute value difference is calculated through the Manhattan distance. The classification output layer network of the single neuron sigmoid is selected according to the task type to obtain the similarity between the preprocessed image and the target image, and the final prediction result is obtained.

7. The method according to claim 1 or 6, characterized in that In step 5, the method of splicing the standard feature map according to the efficiency weight of the task machine includes: According to the efficiency weight of the task machine, the canonical feature map of each channel is flattened into a one-dimensional vector in the channel order, and the one-dimensional vectors of all channels are deeply spliced; or they are deeply spliced ​​in a stacking manner.

8. A multi-scale feature extraction system for enhancing image similarity matching, characterized in that: include: A preprocessing module is used to normalize the color of the received image to obtain a normalized image, and to preprocess the edge of the normalized image using an interpolation method or an image edge subtraction method to obtain a preprocessed image; The feature extraction module is used to input the preprocessed image and the target image into the twin neural network at the same time, and convolve them through the neck network to obtain the multi-layer feature map corresponding to each image; the residual block is used to perform residual connection between the initial position of each layer of the feature map and the terminal position of the upper layer feature map; The fusion module is used to use the Feature Pyramid Network (FPN) in the Path Aggregation Network (PAN) to transfer the semantic information of the high-level feature map of each image after residual connection to the bottom-level feature map, and then transfer the spatial information of the bottom-level feature map to the high-level feature map, so that the semantic information and spatial information of each layer of the feature map are fused to obtain a fused feature map; The standard module is used to introduce spatial pyramid pooling (SPP), fix the width and height of the output of each layer of fusion feature map through a three-channel dynamics max pooling layer, and then fix the output depth through a 1*1 convolution kernel to generate multiple layers of standard feature maps with fixed length, width and depth, thereby realizing weight sharing of the fully connected layer; The similarity acquisition module is used to splice the standard feature map according to the efficiency weight of the task machine, input it into the fully connected layer, select the corresponding output layer network according to the task type, obtain the similarity between the preprocessed image and the target image, and get the final prediction result.

9. A multi-scale feature extraction device for enhancing image similarity matching, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images