Image recognition method, device, equipment and storage medium using adaptive multi-scale feature fusion

Abstract

The present application discloses an image recognition method, device, equipment and storage medium for adaptive multi-scale feature fusion, and relates to the field of computer vision technology. The present application comprises: performing edge detection on an image to be recognized to obtain image partitions of the image to be recognized, and generating corresponding feature recognition image blocks based on the image partitions; performing feature recognition on the feature recognition image blocks to obtain candidate features of the feature recognition image blocks; determining feature labels of the candidate features respectively, classifying the feature labels to obtain classification results, and determining features to be fused according to the classification results; performing feature fusion on the features to be fused based on an image recognition model to obtain target image features, and obtaining image recognition results according to the target image features, so as to identify and obtain key feature information in the image and improve image recognition quality.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image recognition method, device, equipment and storage medium for adaptive multi-scale feature fusion. Background Art

[0002] Currently, when performing image recognition, a single-scale method is usually used for image processing. In the single-scale image recognition process, only a fixed scale in the image can be used for feature extraction. When single-scale feature extraction is performed on the image, the image depth information will be lost. In addition, when performing image recognition based on a single-scale method, especially when detecting small targets, due to the small size of the target, low contrast or little difference from the background, the recognition result deviates from the expectation, and the key feature information in the image cannot be identified, thereby reducing the image recognition quality.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide an image recognition method, device, equipment and storage medium with adaptive multi-scale feature fusion, aiming to solve the technical problem in the prior art that all features of an image cannot be obtained by using single-scale features, thereby reducing the quality of image recognition.

[0005] To achieve the above objectives, the present application proposes an image recognition method using adaptive multi-scale feature fusion, the image recognition method using adaptive multi-scale feature fusion comprising:

[0006] Performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature recognition image blocks based on the image partitions;

[0007] Performing feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block;

[0008] Determine the feature labels of the candidate features respectively, classify the feature labels to obtain classification results, and determine the features to be fused according to the classification results;

[0009] The features to be fused are fused based on the image recognition model to obtain target image features, and image recognition results are obtained according to the target image features.

[0010] In one embodiment, the step of performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature recognition image blocks based on the image partitions respectively includes:

[0011] The image to be identified is subjected to denoising and gray-scale processing through a filter to obtain a preprocessed image;

[0012] Calculating the gradient magnitude and gradient direction of the preprocessed image;

[0013] Determine edge information based on the gradient amplitude and the gradient direction, determine a main body area according to the edge information; and obtain a plurality of image partitions according to the main body area;

[0014] Performing feature recognition on the plurality of image partitions to obtain image partition features, and calculating feature similarity for the image partition features;

[0015] The image partitions are reorganized based on the feature similarity to obtain feature recognition image blocks.

[0016] In one embodiment, the step of performing feature recognition on the plurality of image partitions to obtain image partition features and calculating feature similarity of the image partition features comprises:

[0017] Performing feature recognition on the image partition to determine a plurality of feature points of the image partition;

[0018] Extracting description vectors from the plurality of feature points, and converting the description vectors into standard vectors;

[0019] Determining the characteristic response degree of the standard vector, filtering the standard vector according to the characteristic response degree, and using the standard vector retained after filtering as the image partition feature;

[0020] The correlation coefficient between the image partition features is calculated, and the feature similarity is obtained according to the correlation coefficient.

[0021] In one embodiment, the steps of respectively determining feature labels of the candidate features, classifying the feature labels to obtain classification results, and determining the features to be fused according to the classification results include:

[0022] Determine pixel values ​​corresponding to the candidate features respectively, and obtain feature parameters according to the pixel values;

[0023] Classify according to the characteristic parameters and determine the classification result;

[0024] Calculating a contribution value for the classification result, and determining a fusion weight according to the contribution value;

[0025] The features to be fused are determined according to the fusion weights.

[0026] In one embodiment, the step of performing feature fusion on the features to be fused based on the image recognition model to obtain target image features, and obtaining image recognition results according to the target image features includes:

[0027] Transmitting the features to be fused as input data to an image recognition model, so that the image recognition model performs convolution processing on the features to be fused to obtain convolution fusion features;

[0028] Weighting the convolution fusion features according to the weight parameters in the image recognition model to obtain weighted fusion features;

[0029] Performing feature fusion on the weighted fusion features to obtain target image features;

[0030] An image recognition result is obtained according to the target image features.

[0031] In one embodiment, the step of fusing the weighted fusion features to obtain target image features includes:

[0032] Determining a feature order of the weighted fusion features;

[0033] splicing the weighted fusion features according to the feature order to obtain fusion features;

[0034] Performing dimension reduction on the fusion feature to obtain a reduced-dimensional fusion feature;

[0035] Verifying the dimension reduction fusion features to obtain a verification result;

[0036] When the verification result meets the preset conditions, the dimension reduction fusion feature is used as the target image feature.

[0037] In one embodiment, after the step of verifying the dimension reduction fusion feature and obtaining the verification result, the step further includes:

[0038] When the verification result does not meet the preset condition, determining the offset value of the dimension reduction fusion feature;

[0039] The weight parameters of the image recognition model are updated according to the offset value, and the step of weighting the convolution fusion features according to the weight parameters in the image recognition model to obtain weighted fusion features is returned.

[0040] In addition, to achieve the above purpose, the present application also proposes an image recognition device with adaptive multi-scale feature fusion, the image recognition device with adaptive multi-scale feature fusion comprising:

[0041] An image partitioning module performs edge detection on the image to be identified to obtain image partitions of the image to be identified, and generates corresponding feature recognition image blocks based on the image partitions;

[0042] A feature recognition module, used to perform feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block;

[0043] A feature fusion module is used to respectively determine the feature labels of the candidate features, classify the feature labels to obtain classification results, and determine the features to be fused according to the classification results;

[0044] The image recognition module is used to perform feature fusion on the features to be fused based on the image recognition model to obtain target image features, and obtain image recognition results according to the target image features.

[0045] In addition, to achieve the above-mentioned objectives, the present application also proposes an image recognition device with adaptive multi-scale feature fusion, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the image recognition method with adaptive multi-scale feature fusion as described above.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image recognition method of adaptive multi-scale feature fusion as described above are implemented.

[0047] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the image recognition method of adaptive multi-scale feature fusion as described above are implemented.

[0048] One or more technical solutions proposed in the present application have at least the following technical effects: edge detection is performed on the image to be identified to obtain image partitions of the image to be identified, and corresponding feature identification image blocks are generated based on the image partitions; feature identification is performed on the feature identification image blocks to obtain candidate features of the feature identification image blocks; feature labels of the candidate features are determined respectively, the feature labels are classified to obtain classification results, and features to be fused are determined based on the classification results; feature fusion is performed on the features to be fused based on an image recognition model to obtain target image features, and image recognition results are obtained based on the target image features, which can identify and obtain key feature information in the image and improve image recognition quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0051] Figure 1 A flowchart diagram of the first embodiment of the image recognition method for adaptive multi-scale feature fusion of the present application;

[0052] Figure 2 A schematic diagram of the architecture of an image recognition model provided for an embodiment of an image recognition method for adaptive multi-scale feature fusion of the present application;

[0053] Figure 3 A convolutional neural network configuration provided for an embodiment of an image recognition method for adaptive multi-scale feature fusion of the present application;

[0054] Figure 4 This is a schematic diagram of the module structure of an image recognition device for adaptive multi-scale feature fusion according to an embodiment of the present application;

[0055] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the image recognition method for adaptive multi-scale feature fusion in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solutions of the embodiments of the present application are: performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature identification image blocks based on the image partitions; performing feature recognition on the feature identification image blocks to obtain candidate features of the feature identification image blocks; determining feature labels of the candidate features, classifying the feature labels to obtain classification results, and determining features to be fused based on the classification results; performing feature fusion on the features to be fused based on the image recognition model to obtain target image features, and obtaining image recognition results based on the target image features.

[0060] In this embodiment, for ease of description, the following description is made by taking an image recognition device for identifying adaptive multi-scale feature fusion as the execution subject.

[0061] Since the existing technology usually uses a single-scale method for image processing, in the single-scale image recognition process, only a fixed scale in the image can be used for feature extraction. When single-scale feature extraction is performed on the image, the image depth information will be lost. In addition, when image recognition is performed based on a single-scale method, especially when detecting small targets, due to the small size of the target, low contrast or little difference from the background, the recognition result deviates from the expectation, and the key feature information in the image cannot be identified, thereby reducing the image recognition quality.

[0062] The present application provides a solution by acquiring multi-scale features in an image, adaptively selecting appropriate image features according to image quality, fusing the multi-scale features, performing image recognition on the fused image features, and obtaining high-quality image recognition results.

[0063] It can be seen from the above embodiments that the present application performs edge detection on the image to be identified to obtain image partitions of the image to be identified, and generates corresponding feature identification image blocks based on the image partitions; performs feature recognition on the feature identification image blocks to obtain candidate features of the feature identification image blocks; determines feature labels of the candidate features, classifies the feature labels to obtain classification results, and determines features to be fused based on the classification results; performs feature fusion on the features to be fused based on an image recognition model to obtain target image features, and obtains image recognition results based on the target image features, which can identify and obtain key feature information in the image and improve image recognition quality.

[0064] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an image recognition device with adaptive multi-scale feature fusion, etc. The following takes the image recognition device with adaptive multi-scale feature fusion as an example to illustrate this embodiment and the following embodiments.

[0065] Based on this, the present application embodiment provides an image recognition method for adaptive multi-scale feature fusion, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the image recognition method of adaptive multi-scale feature fusion in the present application.

[0066] In this embodiment, the image recognition method of adaptive multi-scale feature fusion includes steps S10 to S40:

[0067] Step S10: performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature identification image blocks based on the image partitions.

[0068] It should be noted that the image to be identified refers to the image obtained through the input port when performing image recognition. The image to be identified is the original image transmitted by the user or other programs. The image partition refers to the imaging content in the image to be identified, and the image is divided into regions according to the imaging content. The feature recognition image block is the image block corresponding to the image partition for feature recognition, and the feature recognition image block is consistent with the outline and size of the image partition.

[0069] In a specific implementation, before edge detection is performed on an image to be identified, a preprocessing operation can be performed on the image to be identified, and the image can be smoothed to reduce noise interference on image recognition. On this basis, edge detection is performed on the image to be identified. For example, the Sobel operator is used in combination with the differential derivation of Gaussian smoothing, or detection is performed through the Canny edge detection algorithm. The edge information of the imaging content in the image can be obtained, and the image can be divided into multiple partition blocks according to the edge information to obtain image partitions. An understandable and intuitive example is to separate key elements such as portraits in an image from the background of the image to obtain key element image partitions and background image partitions. Of course, this is just a simple image partition. In a specific implementation, the image can be partitioned more finely. After the image partitions are obtained, corresponding feature recognition image blocks can be generated based on each image partition.

[0070] In a feasible implementation manner, the steps of performing edge detection on the image to be identified, obtaining image partitions of the image to be identified, and generating corresponding feature recognition image blocks based on the image partitions include:

[0071] The image to be identified is subjected to denoising and gray-scale processing through a filter to obtain a preprocessed image;

[0072] Calculating the gradient magnitude and gradient direction of the preprocessed image;

[0073] Determine edge information based on the gradient amplitude and the gradient direction, determine a main body area according to the edge information; and obtain a plurality of image partitions according to the main body area;

[0074] Performing feature recognition on the plurality of image partitions to obtain image partition features, and calculating feature similarity for the image partition features;

[0075] The image partitions are reorganized based on the feature similarity to obtain feature recognition image blocks.

[0076] It should be noted that the preprocessed image refers to the image after the preprocessing operation is performed on the image to be identified, the gradient amplitude represents the intensity of the brightness change of a point in the image, the gradient direction represents the direction in which the brightness of a point in the image changes fastest, the image partition refers to the imaging content in the image to be identified, and the area into which the image is divided according to the imaging content, the image partition feature refers to the image feature used for image partition, and is the key feature used to divide the partition, and the feature similarity is used to describe the similarity between features in the image.

[0077] In a specific implementation, when a filter is used to perform denoising and grayscale processing on an image to be identified to obtain a preprocessed image, a Gaussian filter can be used to perform denoising and grayscale processing on the image to be identified. The specific process is: the image to be identified is processed by a Gaussian filter to eliminate the noise information of the image to be identified, and the denoised image to be identified is grayscaled, and a fixed pattern is used to convert the RGB value into a grayscale value. The formula is: , or use deep learning to grayscale the image. The gradient amplitude and gradient direction of the preprocessed image are calculated based on the preprocessed image. Specifically, it can be expressed as: for a point in the image , gradient amplitude Can be:

[0078]

[0079] in, and They are of Direction and Directional gradient component.

[0080] When calculating the gradient direction, the inverse tangent of the gradient component can be used to calculate the gradient direction. The calculation formula is:

[0081]

[0082] if is a positive number, the gradient direction is the inverse tangent value; if It is a negative number and needs to be added Get the right direction if and , then the direction is ,if and , then the direction is .

[0083] By suppressing the non-maximum value of the gradient amplitude of each pixel, the edge is ensured to be the local maximum value along the gradient direction, and the final edge pixel is determined by the double threshold method, the high threshold is used to determine the strong edge, and the low threshold is used to determine the weak edge. Only when the gradient amplitude exceeds the high threshold, the pixel is determined as an edge pixel; if the gradient amplitude is lower than the high threshold but higher than the low threshold, and is adjacent to the strong edge pixel, it is also regarded as an edge pixel. Based on the edge pixels, the image can be divided into multiple regions, and the obtained regions are image partitions. And feature recognition is performed on each image partition, and high-level abstract features can be automatically extracted based on the convolutional neural network to obtain image partition features, and feature similarity is calculated for the features of the image partitions. The similarity of image partition features can be determined by calculating the Euclidean distance of the feature vector or according to the correlation coefficient, and then the image partition features with similarity can be reorganized to determine new image partition features, and then the feature recognition image module can be determined, which can reduce the number of image blocks recognized by the model and reduce the resource consumption of image recognition while ensuring the recognition results.

[0084] In a feasible implementation manner, the step of performing feature recognition on the plurality of image partitions to obtain image partition features, and calculating feature similarity for the image partition features comprises:

[0085] Performing feature recognition on the image partition to determine a plurality of feature points of the image partition;

[0086] Extracting description vectors from the plurality of feature points, and converting the description vectors into standard vectors;

[0087] Determining the characteristic response degree of the standard vector, filtering the standard vector according to the characteristic response degree, and using the standard vector retained after filtering as the image partition feature;

[0088] The correlation coefficient between the image partition features is calculated, and the feature similarity is obtained according to the correlation coefficient.

[0089] It should be noted that feature points refer to sets of pixels detected by feature detection of image partitions, which are used to reflect the characteristics of image partitions. Description vectors are used to describe the trend, position information, etc. of feature points, and standard vectors are obtained after standardization. Feature response degree refers to the importance of feature vectors in the current image partition. Image partition features refer to image features used for image partitions, which are key features for dividing partitions. The correlation coefficient is used to measure the strength and direction of the linear relationship between two vectors. Feature similarity is used to describe the similarity between features in images.

[0090] In a specific implementation, when performing feature recognition on an image partition, multiple feature points of the image partition can be determined by Harris corner point detection. Specifically, corner point detection can be performed on the image partition. is the window of the image partition, and the window can be Move small displacement , then the grayscale of the original image window changes to , which can be expressed as:

[0091]

[0092] Where: is the window existing in the image, , They are the grayscale values ​​of the pixel before and after it moves in the window.

[0093] set up is the autocorrelation matrix corresponding to the pixel, and the expression is:

[0094]

[0095] In the formula, and Represent the grayscale gradient of the image in the horizontal and vertical directions respectively, is a Gaussian function, so it can be determined .

[0096] So in the matrix Building the Harris response function Implement image corner detection and obtain feature points. The expression is:

[0097]

[0098] In the formula, is any small positive number.

[0099] After obtaining the feature points, the description vector can be extracted through the Brier operator. The representative size is And the smoothed pixel blocks are used Represents the midpoint of a pixel block The corresponding grayscale value is obtained to obtain the description vector. After normalizing the description vector, the vector can be converted to a standard vector. For each standard vector, its feature response degree needs to be evaluated. Random forest or ANOVA can be used to determine the feature response degree and filter it to remove those vectors whose feature response degree exceeds the threshold. The remaining standard vectors are used as image partition features and according to the formula:

[0100]

[0101] Where: and are the values ​​of two eigenvectors in specific dimensions, and is the average value.

[0102] Step S20: performing feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block.

[0103] It should be noted that candidate features refer to the set of features to be evaluated and selected in the feature selection process.

[0104] In a specific implementation, feature detection is performed on each feature recognition image block to identify key points, which are used to represent significant features in the image block. A descriptor is generated for each detected key point. The descriptor is a statistical description of the neighborhood around the key point and can capture the characteristic information of the key point. The descriptor of each key point is converted into a feature vector to form a set of candidate feature vectors. The most informative feature vector is selected from the set of candidate feature vectors, and the selected candidate feature vectors are normalized to eliminate the dimensional influence between different features to ensure that they are on the same scale, thereby obtaining the candidate features of the feature recognition image block.

[0105] Step S30, respectively determining feature labels of the candidate features, classifying the feature labels to obtain classification results, and determining features to be fused according to the classification results.

[0106] It should be noted that the feature label refers to the label information of the candidate feature, which is used to describe the attribute information of the candidate feature. The classification result is used to describe the set of feature parameters obtained after classifying the feature parameters. Classification is performed according to the type of feature parameters to obtain the classification result. The feature to be fused refers to the feature information used for feature fusion.

[0107] In a specific implementation, when extracting candidate features from an image, one or more feature labels can be assigned to each candidate feature. The label can indicate the category or attribute to which it belongs. The labeling process can be completed by manual or automated labeling tools, and classified according to the feature labels to facilitate subsequent image processing. For example, the feature labels can be classified into animals, plants, man-made objects, etc. Of course, the feature labels exemplified in this embodiment are exemplary, and the specific classification is determined according to the actual situation. After classifying the feature labels, the features in the same classification result can be determined as features to be fused.

[0108] In a feasible implementation manner, the steps of respectively determining feature labels of the candidate features, classifying the feature labels to obtain classification results, and determining the features to be fused according to the classification results include:

[0109] Determine pixel values ​​corresponding to the candidate features respectively, and obtain feature parameters according to the pixel values;

[0110] Classify according to the characteristic parameters and determine the classification result;

[0111] Calculating a contribution value for the classification result, and determining a fusion weight according to the contribution value;

[0112] The features to be fused are determined according to the fusion weights.

[0113] It should be noted that the feature parameters include the color information, light intensity information, texture information, shape information, etc. of the image. The classification result is used to describe the set of feature parameters obtained after classifying the feature parameters. Classification is performed according to the type of feature parameters to obtain the classification result. The fusion weight refers to the proportion distribution of each feature when performing feature fusion.

[0114] In a specific implementation, for candidate features in an image, the corresponding pixel values ​​are determined by the gradient direction and amplitude in SIFT, and the feature parameters in the current pixel value can be obtained from the pixel value, such as the color information, light intensity information, texture information and other feature parameters corresponding to the pixel value, wherein each feature parameter can correspond to a unique classification category number, and each feature parameter is classified according to the classification result, and the feature parameters are normalized, and the contribution value of each classification result in the corresponding pixel value information in the calculated classification result can be determined, and the fusion weight can be determined according to the proportion of the contribution value. The larger the contribution value, the larger the corresponding fusion weight, and the features to be fused in the classification result are determined based on the fusion weight.

[0115] Step S40, performing feature fusion on the features to be fused based on the image recognition model to obtain target image features, and obtaining image recognition results according to the target image features.

[0116] It should be noted that the image recognition model is a CNN network model, which includes convolutional layers, pooling layers and fully connected layers connected to each other. The target image features refer to the image features after the features are fused through the image recognition model, and the image recognition results are the results after the image recognition model recognizes the image features.

[0117] In a specific implementation, the features to be fused can be transmitted as input data to the image recognition model, so that the image recognition model performs convolution processing on the features to be fused to obtain convolution fusion features; weights the convolution fusion features according to the weight parameters in the image recognition model to obtain weighted fusion features; performs feature fusion on the weighted fusion features to obtain target image features; and obtains image recognition results according to the target image features. Figure 2 , Figure 2The schematic diagram of the architecture of the image recognition model. The specific process can be: first, the features to be fused are input as input data into the input layer, which is used to transfer the features to be fused to the convolution layer. The convolution layer can perform local mobile scanning on the two-dimensional data, and use the result of the convolution operation as the output of the neural node in the corresponding area. Then, the output is made nonlinear through an activation function such as the Sigmoid function, and the result of the convolution layer output is downsampled through the pooling layer to reduce the number of parameters of the model, and then through the fully connected layer to determine the probability of the category described in the image in the nonlinear result. In this application, the following method can be used: Figure 3 The convolutional neural network configuration is used for image processing. It consists of 5 convolutional modules (a total of 13 convolutional layers), three fully connected layers and softmax. In the figure, Conv represents the convolutional layer, which extracts various abstract high-level features by sliding the convolution kernel on the wear image and performing convolution calculations. 3×3 represents the size of the convolution kernel, and the last number represents the number of convolution kernels. It can be seen that from convolution modules 1 to 5, as the number of convolution kernels increases, the number of features extracted by the convolution operation also increases rapidly; MaxPooling represents the maximum pooling layer, which is responsible for further extracting image features and reducing the network size at the same time. The pooling size used is 2×2, that is, the maximum value of the image pixel value within the range of 2×2 is selected as the value of the area after pooling. After the feature extraction output feature map of the five convolutional modules, the output is processed using three fully connected layers FC1, FC2 and FC3, and finally the Sofmax function is used to complete the classification. As a feature extraction network, the VGG16 network does not need to be classified, so only the convolution module part is retained, and the fully connected layer and Sofimax are deleted, and only the image feature map is output.

[0118] In the convolutional layer, we can use the formula: in, represents the kth feature to be fused belonging to the lth layer, is the activation function, is the output of the nth neuron node in the l-1th layer, and the connection range of the neuron node is (k,m). is the weight parameter corresponding to the convolution operation, is the bias parameter. In actual tasks, the convolution kernel cannot calculate the pixels at the edge of the image, so some edge features on the feature map may be ignored. In order to solve this problem, 0 pixels are usually filled around the edges of the input feature map to keep the size of the output feature map consistent with the input feature map. Therefore, the size of the output feature map can be expressed as:

[0119]

[0120] In the formula, k is the width of the surrounding padded pixels, l is the step size of each sliding of the convolution kernel, f represents the size of the convolution kernel, m represents the size of the input feature map, and c represents the size of the output feature map.

[0121] The pooling layer will perform a pooling operation on the output feature map of the convolution layer. In order to obtain the most significant features in the feature map, the local features obtained by the convolution layer will be screened more carefully. The pooling layer can be divided into an average pooling layer and a maximum pooling layer. After pooling, not only can the feature dimension be effectively reduced, but the number of model parameters will also be reduced, which greatly reduces the difficulty of training some complex models and avoids overfitting of the model. The pooling operation is different from the continuous scanning on the feature map like the convolution operation. The pooling layer will first divide the feature map into image blocks, and then directly downsample from different image blocks.

[0122] The function of the fully connected layer is to summarize the feature information obtained from the convolution layer and the pooling layer, and to create a correspondence between the features and different labels. In addition, the fully connected layer also has the function of changing the dimension. It can convert multi-dimensional input features into one-dimensional output features. For the task of target detection, it can output the target category probability and the four coordinates of the positioning box at the same time. The fully connected layer is usually placed at the very end of the CNN network to learn the correspondence between feature information and categories, and obtain a set of probability values ​​through nonlinear modules. The probability values ​​in this set of data represent the probability that the network predicts the category to which the input image belongs. After the input image passes through multiple convolutional layers and pooling layers in CNN, the feature information of the image has been extracted more fully. When it reaches the end of CNN, the extracted feature information will be summarized on the fully connected layer. Generally speaking, the fully connected layer is composed of many neuron nodes connected in pairs between nodes in adjacent layers. Its essence is to establish a mapping model between feature information and category labels.

[0123] In a feasible implementation manner, the step of fusing the weighted fusion features to obtain target image features includes:

[0124] Determining a feature order of the weighted fusion features;

[0125] splicing the weighted fusion features according to the feature order to obtain fusion features;

[0126] Performing dimension reduction on the fusion feature to obtain a reduced-dimensional fusion feature;

[0127] Verifying the dimension reduction fusion features to obtain a verification result;

[0128] When the verification result meets the preset conditions, the dimension reduction fusion feature is used as the target image feature.

[0129] It should be noted that the feature order refers to the arrangement order of the features that need to be fused, the fusion feature refers to the image feature after weighted fusion, and the dimension reduction fusion feature refers to the fusion feature after redundant removal of the fusion feature.

[0130] In the specific implementation, the weight value of the weighted fusion feature is determined, the arrangement data of the feature is determined in the order of the weight, and the weighted features are spliced ​​according to the feature order to form a new feature vector to obtain the fusion feature. Since the fused features may be redundant, the fusion data can be reduced in dimension through dimensionality reduction technology, such as PCA dimensionality reduction, to reduce the dimension of the feature space and improve the computational efficiency. Use the test set or cross-validation method to verify the reduced-dimensional features and evaluate the performance of the model. The performance indicators of interest may include accuracy, recall rate, F1 score, etc. Based on the verification results, the performance of the reduced-dimensional fusion feature is evaluated. If the performance meets the preset conditions (such as reaching a certain accuracy threshold), the reduced-dimensional fusion feature can be used as the target image feature; otherwise, it may be necessary to return and adjust the feature fusion or dimensionality reduction process.

[0131] In a feasible implementation manner, after the step of verifying the dimension reduction fusion feature and obtaining the verification result, the step further includes:

[0132] When the verification result does not meet the preset condition, determining the offset value of the dimension reduction fusion feature;

[0133] The weight parameters of the image recognition model are updated according to the offset value, and the step of weighting the convolution fusion features according to the weight parameters in the image recognition model to obtain weighted fusion features is returned.

[0134] It should be noted that the offset value refers to the difference between the dimension reduction fusion feature and the fusion feature.

[0135] In a specific implementation, when the verification result does not meet the preset conditions, the fusion feature can be subtracted from the dimension reduction fusion feature to obtain an offset value of the dimension reduction fusion feature, and the weight parameter of the image recognition model is updated according to the offset value, specifically: according to the formula:

[0136]

[0137] in, is the sample size, It is Fusion features, It is Dimensionality reduction fusion features.

[0138] Update the model's weight parameters according to the gradient and learning rate. The parameter update formula is:

[0139]

[0140] in, are model parameters, is the learning rate, is the gradient of the loss function with respect to the parameter. And returns the step of weighting the convolution fusion feature according to the weight parameter in the image recognition model to obtain the weighted fusion feature.

[0141] The present embodiment provides an image recognition method for adaptive multi-scale feature fusion, which performs edge detection on an image to be recognized to obtain image partitions of the image to be recognized, and generates corresponding feature recognition image blocks based on the image partitions; performs feature recognition on the feature recognition image blocks to obtain candidate features of the feature recognition image blocks; determines feature labels of the candidate features, classifies the feature labels to obtain classification results, and determines features to be fused based on the classification results; performs feature fusion on the features to be fused based on an image recognition model to obtain target image features, and obtains image recognition results based on the target image features, which can identify and obtain key feature information in the image and improve image recognition quality.

[0142] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image recognition method of adaptive multi-scale feature fusion of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0143] This application also provides an image recognition device with adaptive multi-scale feature fusion, please refer to Figure 4 , the image recognition device for adaptive multi-scale feature fusion comprises:

[0144] An image partitioning module 10 performs edge detection on the image to be identified to obtain image partitions of the image to be identified, and generates corresponding feature recognition image blocks based on the image partitions;

[0145] A feature recognition module 20 is used to perform feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block;

[0146] A feature fusion module 30 is used to respectively determine feature labels of the candidate features, classify the feature labels to obtain classification results, and determine features to be fused according to the classification results;

[0147] The image recognition module 40 is used to perform feature fusion on the features to be fused based on an image recognition model to obtain target image features, and obtain image recognition results according to the target image features.

[0148] The image recognition device with adaptive multi-scale feature fusion provided by the present application adopts the image recognition method with adaptive multi-scale feature fusion in the above-mentioned embodiment, which can solve the technical problem that the single-scale feature cannot obtain all the features of the image in the prior art, thereby reducing the quality of image recognition. Compared with the prior art, the beneficial effects of the image recognition device with adaptive multi-scale feature fusion provided by the present application are the same as the beneficial effects of the image recognition method with adaptive multi-scale feature fusion provided by the above-mentioned embodiment, and the other technical features of the image recognition device with adaptive multi-scale feature fusion are the same as the features disclosed in the above-mentioned embodiment method, which will not be described in detail here.

[0149] The present application provides an image recognition device with adaptive multi-scale feature fusion, and the image recognition device with adaptive multi-scale feature fusion includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image recognition method with adaptive multi-scale feature fusion in the above-mentioned embodiment 1.

[0150] Reference below Figure 5 , which shows a schematic diagram of the structure of an image recognition device suitable for implementing the adaptive multi-scale feature fusion of the embodiment of the present application. The image recognition device with adaptive multi-scale feature fusion in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The image recognition device with adaptive multi-scale feature fusion shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0151] like Figure 5As shown, the image recognition device with adaptive multi-scale feature fusion may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image recognition device with adaptive multi-scale feature fusion are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the image recognition device with adaptive multi-scale feature fusion to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an image recognition device with adaptive multi-scale feature fusion of various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0152] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0153] The image recognition device with adaptive multi-scale feature fusion provided by the present application adopts the image recognition method with adaptive multi-scale feature fusion in the above embodiment, which can solve the technical problem that the single-scale feature cannot obtain all the features of the image in the prior art, thereby reducing the quality of image recognition. Compared with the prior art, the beneficial effects of the image recognition device with adaptive multi-scale feature fusion provided by the present application are the same as the beneficial effects of the image recognition method with adaptive multi-scale feature fusion provided by the above embodiment, and the other technical features of the image recognition device with adaptive multi-scale feature fusion are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0154] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0155] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0156] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the image recognition method for adaptive multi-scale feature fusion in the above-mentioned embodiment.

[0157] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0158] The computer-readable storage medium may be included in the adaptive multi-scale feature fusion image recognition device; or may exist independently without being assembled into the adaptive multi-scale feature fusion image recognition device.

[0159] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image recognition device for adaptive multi-scale feature fusion, the image recognition device for adaptive multi-scale feature fusion:

[0160] Performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature recognition image blocks based on the image partitions;

[0161] Performing feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block;

[0162] Determine the feature labels of the candidate features respectively, classify the feature labels to obtain classification results, and determine the features to be fused according to the classification results;

[0163] The features to be fused are fused based on the image recognition model to obtain target image features, and image recognition results are obtained according to the target image features.

[0164] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0167] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image recognition method of adaptive multi-scale feature fusion, and can solve the technical problem that all features of an image cannot be obtained by using single-scale features in the prior art, thereby reducing the quality of image recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the image recognition method of adaptive multi-scale feature fusion provided in the above-mentioned embodiment, and are not described in detail here.

[0168] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned image recognition method of adaptive multi-scale feature fusion.

[0169] The computer program product provided by the present application can solve the technical problem in the prior art that a single-scale feature cannot obtain all features of an image, thereby reducing the quality of image recognition. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the image recognition method of adaptive multi-scale feature fusion provided by the above-mentioned embodiment, and will not be elaborated here.

[0170] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image recognition method based on adaptive multi-scale feature fusion, characterized in that: The image recognition method of adaptive multi-scale feature fusion includes: Performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature recognition image blocks based on the image partitions; Performing feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block; Determine the feature labels of the candidate features respectively, classify the feature labels to obtain classification results, and determine the features to be fused according to the classification results; Performing feature fusion on the features to be fused based on the image recognition model to obtain target image features, and obtaining image recognition results according to the target image features; The step of performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature identification image blocks based on the image partitions comprises: The image to be identified is subjected to denoising and gray-scale processing through a filter to obtain a preprocessed image; Calculating the gradient magnitude and gradient direction of the preprocessed image; Determine edge information based on the gradient amplitude and the gradient direction, determine a main body area according to the edge information; and obtain a plurality of image partitions according to the main body area; Performing feature recognition on the plurality of image partitions to obtain image partition features, and calculating feature similarity for the image partition features; Reorganize the image partitions based on the feature similarity to obtain feature recognition image blocks; The step of performing feature recognition on the plurality of image partitions to obtain image partition features and calculating feature similarity for the image partition features comprises: Performing feature recognition on the image partition to determine a plurality of feature points of the image partition; Extracting description vectors from the plurality of feature points, and converting the description vectors into standard vectors; Determining the characteristic response degree of the standard vector, filtering the standard vector according to the characteristic response degree, and using the standard vector retained after filtering as the image partition feature; The correlation coefficient between the image partition features is calculated, and the feature similarity is obtained according to the correlation coefficient.

2. The method according to claim 1, characterized in that The steps of respectively determining the feature labels of the candidate features, classifying the feature labels to obtain classification results, and determining the features to be fused according to the classification results include: Determine pixel values ​​corresponding to the candidate features respectively, and obtain feature parameters according to the pixel values; Classify according to the characteristic parameters and determine the classification result; Calculating a contribution value for the classification result, and determining a fusion weight according to the contribution value; The features to be fused are determined according to the fusion weights.

3. The method according to claim 1, characterized in that The step of fusing the features to be fused based on the image recognition model to obtain target image features and obtaining image recognition results according to the target image features comprises: Transmitting the features to be fused as input data to an image recognition model, so that the image recognition model performs convolution processing on the features to be fused to obtain convolution fusion features; Weighting the convolution fusion features according to the weight parameters in the image recognition model to obtain weighted fusion features; Performing feature fusion on the weighted fusion features to obtain target image features; An image recognition result is obtained according to the target image features.

4. The method according to claim 3, characterized in that The step of fusing the weighted fusion features to obtain target image features comprises: Determining a feature order of the weighted fusion features; splicing the weighted fusion features according to the feature order to obtain fusion features; Performing dimension reduction on the fusion feature to obtain a reduced-dimensional fusion feature; Verifying the dimension reduction fusion features to obtain a verification result; When the verification result meets the preset conditions, the dimension reduction fusion feature is used as the target image feature.

5. The method according to claim 4, characterized in that After the step of verifying the dimension reduction fusion feature and obtaining the verification result, the method further includes: When the verification result does not meet the preset condition, determining the offset value of the dimension reduction fusion feature; The weight parameters of the image recognition model are updated according to the offset value, and the step of weighting the convolution fusion features according to the weight parameters in the image recognition model to obtain weighted fusion features is returned.

6. An image recognition device with adaptive multi-scale feature fusion, characterized in that: The device comprises: An image partitioning module performs edge detection on the image to be identified to obtain image partitions of the image to be identified, and generates corresponding feature recognition image blocks based on the image partitions; A feature recognition module, used to perform feature recognition on the feature recognition image block to obtain candidate features of the feature recognition image block; A feature fusion module is used to respectively determine the feature labels of the candidate features, classify the feature labels to obtain classification results, and determine the features to be fused according to the classification results; An image recognition module is used to perform feature fusion on the features to be fused based on an image recognition model to obtain target image features, and obtain image recognition results according to the target image features; The performing edge detection on the image to be identified to obtain image partitions of the image to be identified, and generating corresponding feature identification image blocks based on the image partitions respectively comprises: The image to be identified is subjected to denoising and gray-scale processing through a filter to obtain a preprocessed image; Calculating the gradient magnitude and gradient direction of the preprocessed image; Determine edge information based on the gradient amplitude and the gradient direction, determine a main body area according to the edge information; and obtain a plurality of image partitions according to the main body area; Performing feature recognition on the plurality of image partitions to obtain image partition features, and calculating feature similarity for the image partition features; Reorganize the image partitions based on the feature similarity to obtain feature recognition image blocks; The performing feature recognition on the plurality of image partitions to obtain image partition features, and calculating feature similarity of the image partition features comprises: Performing feature recognition on the image partition to determine a plurality of feature points of the image partition; Extracting description vectors from the plurality of feature points, and converting the description vectors into standard vectors; Determining the characteristic response degree of the standard vector, filtering the standard vector according to the characteristic response degree, and using the standard vector retained after filtering as the image partition feature; The correlation coefficient between the image partition features is calculated, and the feature similarity is obtained according to the correlation coefficient.

7. An image recognition device with adaptive multi-scale feature fusion, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image recognition method for adaptive multi-scale feature fusion according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image recognition method of adaptive multi-scale feature fusion according to any one of claims 1 to 5 are implemented.

Images