Image detection method, system and device based on multi-feature fusion

By combining the deep convolutional neural network ResNet and the Self-Attention mechanism and improving the loss function design, the problem of feature expression limitations and poor adaptability of image retrieval methods in existing technologies is solved, and efficient and accurate image retrieval effects are achieved.

CN120673106APending Publication Date: 2025-09-19SHENZHEN LIUFENG TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510039412.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-01-10
Publication Date
2025-09-19

AI Technical Summary

Technical Problem

Existing image retrieval methods based on convolutional neural networks have the following problems: feature expression is limited to a fixed receptive field, lacks global semantic information, has poor adaptability to data noise, occlusion and deformation, and has limited retrieval speed and model generalization ability.

Method used

The deep convolutional neural network ResNet and the Self-Attention mechanism are used, combined with improved loss function algorithms, including N-Pair Loss and Angular Loss, to optimize feature vectors, improve the ability to capture global and local features of image features, and enhance the model's ability to distinguish the feature distances between similar and dissimilar images.

Benefits of technology

It achieves high-precision image retrieval, improves the adaptability to complex scenes and multi-target tasks, enhances the generalization ability and retrieval efficiency on large-scale image datasets, optimizes the distance measurement of sample distribution, and improves the accuracy and speed of retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120673106A_ABST
    Figure CN120673106A_ABST
Patent Text Reader

Abstract

The invention discloses an image detection method, system and device based on multi-feature fusion, and the method comprises the following steps: S1, obtaining to-be-detected image data, and carrying out the preprocessing of the image data; s2, on the basis of a deep convolutional neural network ResNet and a Self-Attention mechanism, feature extraction is carried out on the image data preprocessed in the step S1; s3, optimizing the feature vectors extracted in the step S2 based on an improved loss function algorithm; s4, performing similarity calculation on the feature vectors optimized in the step S3; s5, reordering the retrieved images according to a similarity calculation result in the step S4, and outputting a final retrieval result; compared with an existing image feature extraction method, based on ResNet and Self-Attention mechanisms, the capturing capacity of the model for global information and local features of the image is improved, the model can intelligently pay attention to key areas in the image, meanwhile, noise interference is reduced, the image features can be efficiently extracted, and accurate image retrieval is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image detection method, system, device and computer storable medium based on multi-feature fusion. Background Art

[0002] Traditional image retrieval methods based on text tags require manual, detailed annotation and description of images. This approach is time-consuming, labor-intensive, and error-prone, and it lacks the ability to capture details in large datasets. For example, when users search by keyword, they often cannot accurately match target images containing specific details. To address these issues, content-based image retrieval (CBIR) technology has emerged. It extracts feature representations of images by analyzing their color, texture, shape, and local features, enabling accurate and efficient image matching.

[0003] Among the most recent technologies, CBIR based on convolutional neural networks (CNNs) is a common image detection method. CNNs extract image features, represent them using feature embedding, and employ vectorized indexing techniques for rapid retrieval. For example, after using ResNet to extract image features, similarity between images is measured using vector similarity (such as Euclidean distance or cosine similarity). This method uses deep convolutional networks to obtain multi-layered image representations and can capture both local and global features to a certain extent.

[0004] However, this method also has the following technical problems:

[0005] 1) Feature expression is limited to a fixed receptive field and lacks global semantic information; 2) It has poor adaptability to data noise, occlusion, and deformation; 3) The retrieval speed and model generalization ability are limited by the optimization capability of the loss function.

[0006] This also leads to significant limitations in its application in image retrieval, especially in terms of computational cost, local feature extraction and retrieval speed.

[0007] Therefore, the current relevant technologies and application research on image detection based on multi-feature fusion still need to be further improved. Summary of the Invention

[0008] In view of this, the present invention proposes an image detection method, system, device and computer-storable medium based on multi-feature fusion, which solves the technical problems in the existing technology of limited retrieval speed and model generalization ability and poor adaptability to data noise, occlusion and deformation.

[0009] The technical solution of the present invention is achieved as follows:

[0010] In one aspect, the present invention provides an image detection method based on multi-feature fusion, comprising the following steps:

[0011] S1. Obtain image data to be tested and pre-process the image data;

[0012] S2, based on the deep convolutional neural network ResNet and Self-Attention mechanism, feature extraction is performed on the image data preprocessed in step S1;

[0013] S3, optimizing the feature vector extracted in step S2 based on the improved loss function algorithm;

[0014] S4, performing similarity calculation on the feature vector optimized in step S3;

[0015] S5. Reorder the retrieved images according to the similarity calculation results in step S4, and output the final retrieval results.

[0016] On the basis of this technical solution, it is further preferred that the preprocessing in step S1 includes performing convolution operations, batch normalization and activation function processing on the image data.

[0017] On the basis of this technical solution, it is further preferred that the deep convolutional neural network ResNet in step S2 is multi-layer, each layer includes several convolution blocks, and the convolution blocks include convolution operation (Convolution), batch normalization (BatchNormalization), and activation function (Softmax / ReLU).

[0018] On the basis of this technical solution, it is further preferred that the deep convolutional neural network ResNet in step S2 has four layers, wherein:

[0019] The first layer is used to extract low-level features, including image edges, colors, and shapes;

[0020] The second layer is used to extract mid-level features, including texture and regional patterns;

[0021] The third layer is used to extract complex features, including the local shape of the object;

[0022] The fourth layer is used to extract high-level features, which include object structure and category information.

[0023] On the basis of this technical solution, it is further preferred that the Self-Attention mechanism described in step S2 is used to calculate the correlation between features and adjust feature weights.

[0024] On the basis of this technical solution, it is further preferred that the improved loss function algorithm in step S3 is N-Pair Loss, Angular Loss and cross entropy loss.

[0025] In a second aspect, the present invention further provides an image detection system based on multi-feature fusion, comprising:

[0026] A preprocessing module is used to obtain the image data to be tested and preprocess the image data;

[0027] The feature extraction module extracts features from the image data preprocessed by the preprocessing module based on the deep convolutional neural network ResNet and the Self-Attention mechanism;

[0028] The optimization module optimizes the feature vectors extracted by the feature extraction module based on the improved loss function algorithm;

[0029] A similarity calculation module is used to perform similarity calculation on the feature vectors optimized by the optimization module;

[0030] The result evaluation module reorders the retrieved images according to the calculation results of the similarity calculation module and outputs the final retrieval results.

[0031] On the basis of this technical solution, it is further preferred that the preprocessing module includes a convolution operation unit, a batch normalization unit and an activation function unit.

[0032] In a third aspect, the present invention further provides an electronic device comprising a processor and a memory for storing processor executable instructions, wherein the processor executes the instructions to implement the image detection method based on multi-feature fusion as described in any one of the first aspects.

[0033] Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image detection method based on multi-feature fusion as described in any one of the first aspects.

[0034] The image detection method, system, device, and computer-storable medium based on multi-feature fusion described in the present invention have the following advantages over the prior art:

[0035] This paper introduces a self-attention mechanism based on the ResNet network to capture global contextual information in images, improving its adaptability to complex scenarios and multi-target tasks. Furthermore, by improving the loss function design and introducing a joint optimization method of N-Pair Loss and Angular Loss, the model's ability to distinguish feature distances between similar and dissimilar images is further enhanced, thereby achieving high-precision image retrieval.

[0036] The image detection method and system based on multi-feature fusion provided by the present invention can more effectively capture global and local features, improve the comprehensiveness and fine-grained expressiveness of feature extraction; in addition, it enhances the generalization ability and retrieval efficiency on large-scale image datasets; and through the innovative loss function design, it optimizes the distance measurement of sample distribution, thereby improving the accuracy and speed of retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flow chart of the image detection method based on multi-feature fusion described in Example 1 of the present invention;

[0039] Figure 2 This is a schematic diagram of the image detection method based on multi-feature fusion according to Example 2 of the present invention;

[0040] Figure 3 This is a framework diagram of the image detection system based on multi-feature fusion described in Example 3 of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] The present invention provides an image detection method, system, device, and computer-storable medium based on multi-feature fusion. This method primarily utilizes deep learning methods, combined with a ResNet convolutional neural network and a self-attention mechanism, to achieve efficient feature extraction from images. This method is used to capture global contextual information within images and enhance adaptability to complex scenarios and multi-target tasks. Furthermore, the present invention improves the loss function design and introduces a joint optimization method using N-Pair Loss and Angular Loss, further enhancing the model's ability to distinguish feature distances between similar and dissimilar images, thereby enabling high-precision image retrieval.

[0043] Example 1

[0044] like Figure 1 As shown, an image detection method based on multi-feature fusion includes the following steps:

[0045] S1. Obtain image data to be tested and pre-process the image data;

[0046] S2, based on the deep convolutional neural network ResNet and Self-Attention mechanism, feature extraction is performed on the image data preprocessed in step S1;

[0047] S3, optimizing the feature vector extracted in step S2 based on the improved loss function algorithm;

[0048] S4, performing similarity calculation on the feature vector optimized in step S3;

[0049] S5. Reorder the retrieved images according to the similarity calculation results in step S4, and output the final retrieval results.

[0050] In this embodiment, specifically, the preprocessing in step S1 includes performing convolution operations, batch normalization, and activation function processing on the image data.

[0051] In this embodiment, specifically, the deep convolutional neural network ResNet in step S2 is multi-layer, each layer includes several convolution blocks, and the convolution blocks include convolution operations, batch normalization and activation functions.

[0052] In this embodiment, specifically, the deep convolutional neural network ResNet in step S2 is a four-layer network, wherein:

[0053] The first layer is used to extract low-level features, including image edges, colors, and shapes;

[0054] The second layer is used to extract mid-level features, including texture and regional patterns;

[0055] The third layer is used to extract complex features, including the local shape of the object;

[0056] The fourth layer is used to extract high-level features, which include object structure and category information.

[0057] In this embodiment, specifically, the Self-Attention mechanism in step S2 is used to calculate the correlation between features and adjust feature weights.

[0058] The benefit of this is that it can improve the model's ability to capture global information and local features of the image. Self-Attention dynamically assigns attention weights, enabling the model to intelligently focus on key areas in the image while reducing noise interference.

[0059] In this embodiment, specifically, the improved loss function algorithm described in step S3 is N-Pair Loss, Angular Loss, and Cross Entropy Loss. This design can significantly improve the model's feature differentiation ability, making similar images closer and dissimilar images farther apart, thereby improving retrieval accuracy.

[0060] Example 2

[0061] like Figure 2 As shown, this embodiment will provide a detailed description of the method according to the specific operating steps of the method.

[0062] An image detection method based on multi-feature fusion includes the following steps:

[0063] S1. Obtain image data to be tested and pre-process the image data;

[0064] Specifically, the input directly obtains the image data to be measured, and the image data is represented as a matrix Where H is the image height, W is the image width, and C is the number of channels. The image data is then preprocessed, including convolution, batch normalization, and activation function (CBS).

[0065] Specifically, the calculation formula of the convolution operation is:

[0066] Z=W*X+b

[0067] Among them, W is the convolution kernel; b is the bias term, Here, H' and W' respectively represent the height and width of the output image after convolution processing in CBS processing, and K represents the number of convolution kernels (i.e., the number of channels).

[0068] Specifically, the calculation formula for batch normalization is:

[0069]

[0070] Among them, μ and ρ 2 are the mean and variance in the current mini-batch, ζ and β are trainable parameters used to rescale the distribution.

[0071] Specifically, the ReLU activation function is:

[0072]

[0073] The main function of CBS processing is to extract the preliminary local features of the image to be tested, including edges and textures, accelerate training and stabilize feature distribution through batch normalization, and then use the ReLU activation function to introduce nonlinear capabilities, so that complex features can be learned.

[0074] S2, based on the deep convolutional neural network ResNet and Self-Attention mechanism, feature extraction is performed on the image data preprocessed in step S1;

[0075] Specifically, the deep convolutional neural network ResNet in step S2 is multi-layer, each layer includes several convolution blocks, and the convolution blocks include convolution operations, batch normalization and activation functions.

[0076] Specifically, the deep convolutional neural network ResNet in step S2 is a four-layer network, wherein:

[0077] The first layer is used to extract low-level features, including image edges, colors, and shapes;

[0078] The second layer is used to extract mid-level features, including texture and regional patterns;

[0079] The third layer is used to extract complex features, including the local shape of the object;

[0080] The fourth layer is used to extract high-level features, which include object structure and category information.

[0081] Specifically, the deep convolutional neural network ResNet includes a fully connected layer, which convolutionally transforms the feature A 4 (4 represents the output of the fourth layer) flattened into a one-dimensional vector And perform a linear transformation:

[0082] v FC =W FC *Flatten(A (4) )+b FC

[0083] Flatten represents a flattening operation.

[0084] The fully connected layer is mainly used to flatten the features from a two-dimensional matrix to a one-dimensional vector, perform linear transformation on the features, and prepare input for the subsequent Self-Attention mechanism.

[0085] Specifically, the Self-Attention mechanism is used to calculate the correlation between each position in the input feature and dynamically adjust the feature weights.

[0086] The computation process includes linear transformation to generate query, key, and value: given input features The query, key, and value are obtained through linear transformation. The calculation formula is:

[0087] Q=W Q V FC , K=W k v FC , V=W v v FC

[0088] in

[0089] Calculate the weights, calculate the relevance of the query to the key by dot product, and regularize by softmax:

[0090]

[0091] Output features: The output features of the self-attention mechanism are:

[0092] v SA =Attention(Q, K, V)

[0093] The role of the Self-Attention mechanism is to perform similarity calculation or classification tasks and provide a high-dimensional feature vector of the image to facilitate the next step of processing.

[0094] S3, optimizing the feature vector extracted in step S2 based on the improved loss function algorithm;

[0095] Specifically, the improved loss function algorithms in step S3 are N-Pair Loss, Angular Loss, and cross entropy loss.

[0096] Specifically, N-Pair Loss is optimized by introducing multiple negative samples to ensure that the distance between each sample and a large number of non-similar samples remains in a reasonable range. Angular Loss further enhances the feature similarity between similar samples through angle constraints and reduces the ambiguity between categories.

[0097] The improved algorithm combining N-Pair Loss, Angular Loss and cross entropy loss helps to achieve comprehensive optimization of feature vectors.

[0098] S4, performing similarity calculation on the feature vector optimized in step S3;

[0099] Specifically, the similarity between feature vectors is calculated using Euclidean distance or cosine similarity, and the results closest to the image to be tested are quickly screened out, and the retrieval process is accelerated with the help of efficient indexing technology (such as KD tree or hash index).

[0100] S5. Reorder the retrieved images according to the similarity calculation results in step S4, and output the final retrieval results.

[0101] Specifically, based on the results of similarity calculation, the retrieved images to be tested are reordered and the result closest to the image to be tested is output; an image matching list that meets user needs is provided to complete the final retrieval result.

[0102] The performance of this method was evaluated using the Average Mean Precision (MAP) indicator. The experimental results showed that the MAP reached 90.6%, indicating that this method can improve the performance of the retrieval system on large-scale datasets.

[0103] Example 3

[0104] like Figure 3 As shown, an image detection system based on multi-feature fusion includes:

[0105] A preprocessing module is used to obtain the image data to be tested and preprocess the image data;

[0106] In this embodiment, specifically, the preprocessing module includes a convolution operation unit, a batch normalization unit and an activation function unit.

[0107] This module serves as the starting point of the entire feature extraction and retrieval process, providing basic data for subsequent feature processing.

[0108] Perform various enhancements on the input image, including rotation, scaling, cropping, flipping, and adding noise, to increase the diversity of training data and simulate changes in different scenarios; improve the generalization ability of the model, making it robust in the face of different conditions (such as noise, deformation, and lighting changes).

[0109] The feature extraction module extracts features from the image data preprocessed by the preprocessing module based on the deep convolutional neural network ResNet and the Self-Attention mechanism;

[0110] Specifically, ResNet in this module extracts local and global image features through deep convolution; Self-Attention is mainly responsible for capturing the global dependencies between features, dynamically adjusting the attention distribution, enhancing the expression of key features, and suppressing background noise.

[0111] Furthermore, the features extracted by the ResNet-SelfAttention module are integrated (such as pooling operations or specific aggregation methods); in this way, multi-layer features (such as channel features or spatial features) are compressed into a feature vector of fixed length; data redundancy is reduced, providing efficient input for subsequent feature similarity comparison.

[0112] The optimization module optimizes the feature vectors extracted by the feature extraction module based on the improved loss function algorithm;

[0113] Perform dimension regularization and normalization on the aggregated feature vectors to unify the feature vectors into a fixed dimension to ensure consistent input format; perform normalization on the features to facilitate subsequent similarity calculations and avoid the influence of feature scale on the results.

[0114] A similarity calculation module is used to perform similarity calculation on the feature vectors optimized by the optimization module;

[0115] By calculating the similarity between the input features and the features in the database, we can determine which images in the database are closest to the input image. We can also compare the features using methods such as Euclidean distance and cosine similarity. This provides a basis for subsequent sorting and result output.

[0116] The result evaluation module reorders the retrieved images according to the calculation results of the similarity calculation module and outputs the final retrieval results.

[0117] In a preferred embodiment, the present invention also provides an electronic device, which includes a processor and a memory for storing processor executable instructions, and the processor executes the instructions to implement the image detection method based on multi-feature fusion as described in any one of the first aspects.

[0118] In a preferred embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image detection method based on multi-feature fusion as described in any one of the first aspects.

[0119] In summary, the present invention provides an image detection method, system, device and computer-storable medium based on multi-feature fusion, which solves the problem of poor adaptability of existing technologies to complex scenes (such as background noise, occlusion and deformation). Compared with traditional solutions based on CBIR and Triplet Loss optimization, the method of the present invention can more effectively capture global and local features, improve the comprehensiveness and fine-grained expression of feature extraction; secondly, it enhances the generalization ability and retrieval efficiency on large-scale image datasets; thirdly, through the innovative loss function design, the distance metric of the sample distribution is optimized, so that the accuracy and speed of retrieval are improved, and the accuracy and robustness of image feature expression are significantly improved.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image detection method based on multi-feature fusion, characterized in that: The steps include: S1. Obtain image data to be tested and pre-process the image data; S2, based on the deep convolutional neural network ResNet and Self-Attention mechanism, feature extraction is performed on the image data preprocessed in step S1; S3, optimizing the feature vector extracted in step S2 based on the improved loss function algorithm; S4, performing similarity calculation on the feature vector optimized in step S3; S5. Reorder the retrieved images according to the similarity calculation results in step S4, and output the final retrieval results.

2. The image detection method based on multi-feature fusion according to claim 1, characterized in that: The preprocessing in step S1 includes performing convolution operations, batch normalization, and activation function processing on the image data.

3. The image detection method based on multi-feature fusion according to claim 2, characterized in that: The deep convolutional neural network ResNet in step S2 is multi-layer, and each layer includes several convolution blocks, and the convolution blocks include convolution operations, batch normalization and activation functions.

4. The image detection method based on multi-feature fusion according to claim 3, characterized in that: The deep convolutional neural network ResNet in step S2 is a four-layer network, where: The first layer is used to extract low-level features, including image edges, colors, and shapes; The second layer is used to extract mid-level features, including texture and regional patterns; The third layer is used to extract complex features, including the local shape of the object; The fourth layer is used to extract high-level features, which include object structure and category information.

5. The image detection method based on multi-feature fusion according to claim 1, wherein: The Self-Attention mechanism described in step S2 is used to calculate the correlation between features and adjust feature weights.

6. The image detection method based on multi-feature fusion according to claim 1, characterized in that: The improved loss function algorithms described in step S3 are N-Pair Loss, Angular Loss, and cross entropy loss.

7. An image detection system based on multi-feature fusion, characterized in that: include: A preprocessing module is used to obtain the image data to be tested and preprocess the image data; The feature extraction module extracts features from the image data preprocessed by the preprocessing module based on the deep convolutional neural network ResNet and the Self-Attention mechanism; The optimization module optimizes the feature vectors extracted by the feature extraction module based on the improved loss function algorithm; A similarity calculation module is used to perform similarity calculation on the feature vectors optimized by the optimization module; The result evaluation module reorders the retrieved images according to the calculation results of the similarity calculation module and outputs the final retrieval results.

8. The image detection system based on multi-feature fusion according to claim 7, characterized in that: The preprocessing module includes a convolution operation unit, a batch normalization unit and an activation function unit.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory for storing processor-executable instructions, and the processor executes the instructions to implement the image detection method based on multi-feature fusion as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, which, when executed by a processor, implements the image detection method based on multi-feature fusion as described in any one of claims 1 to 6.