A multi-instance image classification method based on pre-examination feature fusion

Abstract

This invention relates to the field of deep learning multi-instance image classification technology, specifically to a multi-instance image classification method based on pre-detection feature fusion. The method includes: acquiring multi-instance images of the target; performing region pre-detection on each instance image using a pre-detection module; extracting pre-detection features from the pre-detection region using a pre-detection feature encoder; weighting and fusing the features of each instance using an instance fusion module to obtain an instance fusion view; mapping each instance to the same spatial plane and compressing spatial features using a spatial fusion module to obtain a spatial fusion view; and performing joint prediction of multi-instance classification using the instance fusion view and the spatial fusion view. This technical solution can reduce image size and computational load in multi-instance scenarios, improve the effectiveness of multi-instance image information, and solve the information sparsity and high computational requirements problems in multi-instance image learning tasks, thereby improving the accuracy and speed of multi-instance image classification.

Description

Technical Field

[0001] This invention relates to the field of deep learning multi-instance image classification technology, and in particular to a multi-instance image classification method based on pre-detection feature fusion. Background Technology

[0002] With the large-scale application of deep learning technology in computer vision, deep learning-based image multi-instance learning algorithms have become mainstream in multi-instance learning tasks. However, existing multi-instance image classification methods often suffer from information sparsity due to the large number of images involved in multi-instance tasks. Furthermore, multi-instance tasks increase computational complexity, posing significant challenges to improving information effectiveness and reducing computational burden, resulting in low accuracy in multi-instance image classification. This is particularly true in scenarios such as remote sensing and medical imaging where the target area is small or blurry. The large image size and small target size exacerbate the information sparsity problem in multi-instance images, causing many useless background images to participate in the computation.

[0003] Therefore, traditional multi-instance image classification methods perform poorly in scenarios with a large number of instances, small target size, and insufficient computing resources, and are difficult to achieve lightweight computation and real-time image analysis. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-instance image classification method based on pre-detection feature fusion, which solves the problems of poor detection performance of traditional multi-instance image classification methods in scenarios with a large number of instances, small target size, and insufficient computing resources, and the difficulty in achieving lightweight computing and real-time image analysis.

[0005] To achieve the above objectives, the present invention provides a multi-instance image classification method based on pre-detection feature fusion, the multi-instance image classification method based on pre-detection feature fusion comprising the following steps:

[0006] Multi-instance imaging is obtained using a multi-instance image acquisition tool;

[0007] The multi-instance images are input into the pre-detection module for region pre-detection to obtain the pre-detection region for each instance;

[0008] The pre-detection region is used to perform single-instance feature extraction to obtain pre-detection features;

[0009] The pre-detection features are weighted and fused using the instance fusion module to obtain an instance fusion view;

[0010] The pre-detection features are compressed using a spatial fusion module to obtain a spatial fusion view.

[0011] The target category is obtained by jointly predicting the multi-instance classification using the instance fusion view and the spatial fusion view.

[0012] The specific content of the step "obtaining multi-instance imaging through a multi-instance image acquisition tool" is as follows:

[0013] For the target to be identified, multiple instances of images are acquired using multiple imaging and multi-angle imaging methods in both time and space dimensions;

[0014] Representative images are selected from the multiple instance images by selecting images with different imaging times and different imaging angles, and similar images with adjacent imaging times and the same imaging angle are deleted to improve data representativeness.

[0015] The specific content of the step "inputting the multi-instance images into the pre-detection module for region pre-detection" is as follows:

[0016] The aspect ratio, width and height dimensions, and center offset distance of the prior bounding box are set for the multi-instance imaging. The center offset distance is calculated by forming a matrix containing only two elements from the difference between the horizontal and vertical coordinates of the geometric center point of the image and the geometric center point of the prior bounding box. The difference between the horizontal coordinates in the matrix is ​​divided by the image width, and the difference between the vertical coordinates in the matrix is ​​divided by the image height. Then, the Frobenius norm of the matrix is ​​calculated to obtain the center offset distance.

[0017] The pre-detection module uses the multi-stage detection model HTC to perform region detection, and selects the predicted region with the largest difference between confidence and center offset distance as the pre-detection region.

[0018] In the step of “extracting pre-detection features from the pre-detection region using a pre-detection feature encoder”, the pre-detection feature encoder includes a single-instance feature extractor and a single-instance feature mixer.

[0019] The single-instance feature extractor includes a feature dimensionality reduction module and a feature aggregation module. The feature dimensionality reduction module consists of a convolutional layer with a stride of 2, a random pooling layer, and a hybrid pooling layer in parallel to achieve feature dimensionality reduction. The feature aggregation module consists of a linear mapping layer, a state-space model, and a dot product attention layer in serial order to achieve feature aggregation.

[0020] The single-instance feature mixer is used to concatenate dimensionality reduction features and aggregated features along the channel dimension, and uses two-group convolutions and one-group pointwise convolutions to achieve channel compression and channel information mixing for the dimensionality reduction features and aggregated features.

[0021] The specific content of the step "using the instance fusion module to perform multi-instance feature weighting and fusion of the pre-detection features to obtain an instance fusion view" is as follows:

[0022] The multi-instance features are arranged according to the instance order, and the sequence number is converted into a binary number to obtain the sequence code. The sequence code is accumulated with the corresponding instance features and mapped to a one-dimensional sequence to obtain the multi-instance sequence features.

[0023] The multi-instance sequence features are input into a 2D state-space model and a multilayer perceptron to predict the weights w1 to w2 for each instance. n Associate each instance with its corresponding weight w i Multiply and pass through a one-dimensional convolutional layer to obtain a fused view of instances.

[0024] The 2D state space model employs a bidirectional scanning mechanism in the channel dimension to achieve feature mixing across instances, and in the spatial dimension to achieve feature mixing across spatial locations.

[0025] The specific content of the step "compressing the pre-detection features using the spatial fusion module to obtain a spatial fusion view" is as follows:

[0026] Each instance of the pre-detection feature is mapped to the same spatial plane and horizontally stitched together to obtain a spatial stitched feature map with a height of h and a width of n×w, thereby realizing the spatial dimension of instance rearrangement;

[0027] The spatial stitching feature map is used to obtain a spatial fusion view using horizontal and vertical asymmetric convolutional blocks.

[0028] The specific content of the horizontal and vertical asymmetric convolution block is as follows: first, perform ordinary convolution on the features, and then perform asymmetric horizontal convolution of length n×d and asymmetric vertical convolution of length d, where n is the number of instances and d is the convolution kernel length parameter.

[0029] The specific content of the step "jointly predicting multi-instance classification by combining the instance fusion view and the spatial fusion view" is as follows:

[0030] Global average pooling and multilayer perceptron prediction are performed on the instance fusion view and spatial fusion view in single-view and dual-view modes to obtain the instance fusion view prediction value, spatial fusion view prediction value and dual-view prediction value.

[0031] The instance fusion view prediction value, spatial fusion view prediction value, and dual view prediction value are concatenated into a one-dimensional vector, and a multilayer perceptron is used to reduce the feature dimensionality of the one-dimensional vector to achieve multi-view joint prediction.

[0032] This invention discloses a multi-instance image classification method based on pre-detection feature fusion, comprising: acquiring multi-instance images of a target; performing region pre-detection on each instance image using a pre-detection module; extracting pre-detection features from the pre-detection regions using a pre-detection feature encoder; weighting and fusing the features of each instance using an instance fusion module to obtain an instance fusion view; mapping each instance to the same spatial plane and compressing spatial features using a spatial fusion module to obtain a spatial fusion view; and performing joint prediction of multi-instance classification using the instance fusion view and the spatial fusion view. By achieving multi-instance classification learning of images through pre-detection feature fusion, the image size and computational load in multi-instance scenarios can be reduced, and the effectiveness of multi-instance image information can be improved. This addresses the information sparsity problem and high computational demand problem in multi-instance image learning tasks, thereby improving the accuracy and speed of multi-instance image classification. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the steps of the multi-instance image classification method based on pre-detection feature fusion provided by the present invention.

[0035] Figure 2 This is a structural diagram of the multi-instance pre-detection feature fusion network provided by the present invention.

[0036] Figure 3 This is a structural diagram of the pre-detection feature encoder provided by the present invention.

[0037] Figure 4 This is a schematic diagram of the scanning direction of the 2D state space model provided by the present invention. Detailed Implementation

[0038] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0039] Please see Figures 1 to 4 This invention provides a multi-instance image classification method based on pre-detection feature fusion, which includes the following steps:

[0040] Step 101: Obtain multiple instance images of the target.

[0041] Computer equipment acquires multiple instances of images of the target in both time and space using multiple imaging and multi-angle imaging methods. Representative images are then selected from these multiple instances by choosing images with different imaging times and angles, while similar images with adjacent imaging times or the same imaging angle are deleted to improve data representativeness.

[0042] Step 102: Use the pre-detection module to perform region pre-detection on each instance image.

[0043] The computer equipment sets the aspect ratio, width and height dimensions, and center offset distance of the prior bounding box for multi-instance imaging. The center offset distance is calculated by forming a matrix containing only two elements from the differences in the horizontal and vertical coordinates of the geometric center point of the image and the geometric center point of the prior bounding box. The difference in the horizontal coordinates of the matrix is ​​divided by the image width, and the difference in the vertical coordinates of the matrix is ​​divided by the image height. Then, the Frobenius norm of the matrix is ​​calculated to obtain the center offset distance. The pre-detection module uses the multi-stage detection model HTC to perform region detection and selects the predicted region with the largest difference between the confidence level and the center offset distance as the pre-detection region.

[0044] Step 103: Use the pre-detection feature encoder to extract pre-detection features from the pre-detection region using single-instance features.

[0045] The computer equipment utilizes a single-instance feature extractor and a single-instance feature mixer to perform pre-detection feature extraction and mixing. The single-instance feature extractor includes a feature dimensionality reduction module and a feature aggregation module. The feature dimensionality reduction module consists of a convolutional layer with a stride of 2, a random pooling layer, and a hybrid pooling layer in parallel to achieve feature dimensionality reduction. The feature aggregation module consists of a linear mapping layer, a state-space model, and a dot product attention layer in serial order to achieve feature aggregation. The single-instance feature mixer concatenates the dimensionality-reduced features and aggregated features along the channel dimension, and uses a grouped convolution with 2 groups and a pointwise convolution with 1 group to achieve channel compression and channel information mixing of the dimensionality-reduced features and aggregated features.

[0046] Step 104: Use the instance fusion module to perform feature weighting on each instance and fuse them to obtain the instance fusion view.

[0047] The computer device arranges the multi-instance features according to the instance order, converts the sequence number into a binary number to obtain the sequence code, accumulates the sequence code with the corresponding instance features and maps it to a one-dimensional sequence to obtain the multi-instance sequence features; inputs the multi-instance sequence features into a 2D state space model and a multilayer perceptron to predict the weight corresponding to each instance, multiplies each instance with the corresponding weight and passes it through a one-dimensional convolutional layer to obtain the instance fusion view.

[0048] Step 105: Use the spatial fusion module to map each instance onto the same spatial plane and perform spatial feature compression to obtain a spatial fusion view.

[0049] Computer equipment can map each instance of the pre-detection features onto the same spatial plane and perform horizontal stitching to obtain a spatial stitched feature map with a height of h and a width of n×w, thereby realizing the spatial rearrangement of instances; the spatial stitched feature map is then used to obtain a spatial fusion view through horizontal and vertical asymmetric convolutional blocks.

[0050] Step 106: Perform joint prediction of multi-instance classification using the instance fusion view and the spatial fusion view.

[0051] The computer device performs global average pooling and multilayer perceptron prediction on the instance fusion view and spatial fusion view in single-view and dual-view modes to obtain the instance fusion view prediction value, spatial fusion view prediction value and dual-view prediction value. The three prediction values ​​are concatenated into a one-dimensional vector, and the multilayer perceptron is used to reduce the feature dimensionality of the one-dimensional vector to achieve multi-view joint prediction.

[0052] In one embodiment, such as Figure 2 As shown, a specific network structure for a multi-instance image classification method based on pre-detection feature fusion is provided. The multi-instance pre-detection feature fusion network can acquire multi-instance images of the target; the pre-detection module performs region pre-detection on each instance image; the pre-detection feature encoder extracts single-instance features from the pre-detection region to obtain pre-detection features; the instance fusion module performs feature weighting and fusion on each instance to obtain an instance fusion view; the spatial fusion module maps each instance to the same spatial plane and performs spatial feature compression to obtain a spatial fusion view; and the instance fusion view and the spatial fusion view are used for joint prediction of multi-instance classification.

[0053] In one embodiment, a multi-instance image classification method based on pre-detection feature fusion may further include obtaining multi-instance images through a multi-instance image acquisition tool, including: acquiring multi-instance images of the target in the time and space dimensions using multiple imaging and multi-angle imaging methods; performing representative image screening on the multi-instance images, selecting images with different imaging times and different imaging angles, and deleting similar images with adjacent imaging times and the same imaging angle to improve representativeness.

[0054] In one embodiment, a multi-instance image classification method based on pre-detection feature fusion may further include setting the aspect ratio, width and height dimensions, and center offset distance of the prior bounding boxes for multi-instance imaging. The center offset distance is calculated by forming a matrix containing only two elements from the differences in the horizontal and vertical coordinates of the image geometric center point and the prior bounding box geometric center point. The difference in the horizontal coordinates in the matrix is ​​divided by the image width, and the difference in the vertical coordinates in the matrix is ​​divided by the image height. Then, the Frobenius norm of the matrix is ​​calculated to obtain the center offset distance. The pre-detection module uses a multi-stage detection model HTC to perform region detection, and the predicted region with the largest difference between the confidence level and the center offset distance is selected as the pre-detection region.

[0055] In one embodiment, such as Figure 3 As shown, the multi-instance image classification method based on pre-detection feature fusion can further include a pre-detection feature encoder, which includes a single-instance feature extractor and a single-instance feature mixer. The single-instance feature extractor includes a feature dimensionality reduction module and a feature aggregation module. The feature dimensionality reduction module consists of a convolutional layer with a stride of 2, a random pooling layer, and a hybrid pooling layer in parallel to achieve feature dimensionality reduction. The feature aggregation module consists of a linear mapping layer, a state-space model, and a dot product attention layer in serial order to achieve feature aggregation. The single-instance feature mixer concatenates the dimensionality-reduced features and aggregated features along the channel dimension, and uses a grouped convolution with 2 groups and a pointwise convolution with 1 group to achieve channel compression and channel information mixing of the dimensionality-reduced features and aggregated features.

[0056] In one embodiment, a multi-instance image classification method based on pre-detection feature fusion may further include arranging the multi-instance features according to the instance order, converting the sequence number into a binary number to obtain a sequence code, accumulating the sequence code with the corresponding instance features and mapping it to a one-dimensional sequence to obtain multi-instance sequence features; and inputting the multi-instance sequence features into a 2D state-space model and a multilayer perceptron to predict the weights w1 to w2 corresponding to each instance. n Associate each instance with its corresponding weight w i Multiply and pass through a one-dimensional convolutional layer to obtain a fused view of instances.

[0057] In one embodiment, a multi-instance image classification method based on pre-detection feature fusion may further include mapping each instance of the pre-detection features to the same spatial plane and performing horizontal stitching to obtain a spatial stitched feature map with a height of h and a width of n×w, thereby realizing the spatial dimension of instance rearrangement; and using horizontal and vertical asymmetric convolutional blocks on the spatial stitched feature map to obtain a spatial fusion view.

[0058] In one embodiment, a multi-instance image classification method based on pre-detection feature fusion may further include performing global average pooling and multilayer perceptron prediction on instance fusion view and spatial fusion view in single-view and dual-view modes to obtain instance fusion view prediction value, spatial fusion view prediction value and dual-view prediction value; concatenating instance fusion view prediction value, spatial fusion view prediction value and dual-view prediction value into a one-dimensional vector, and using multilayer perceptron to achieve feature dimensionality reduction of the one-dimensional vector to achieve multi-view joint prediction.

[0059] In one embodiment, such as Figure 4 As shown, the multi-instance image classification method based on pre-detection feature fusion can also include a 2D state space model. The 2D state space model uses a bidirectional scanning mechanism in the channel dimension to achieve cross-instance feature mixing and in the spatial dimension to achieve cross-spatial location feature mixing.

[0060] In one embodiment, a multi-instance image classification method based on pre-detection feature fusion may further include horizontal and vertical asymmetric convolution blocks. The horizontal and vertical asymmetric convolution blocks first perform ordinary convolution on the features, and then perform asymmetric horizontal convolution of length n×d and asymmetric vertical convolution of length d, where n is the number of instances and d is the convolution kernel length parameter.

[0061] In summary, the multi-instance image classification method based on pre-detection feature fusion provided by this technical solution can reduce the image size and computational load in multi-instance scenarios, improve the effectiveness of multi-instance image information, solve the information sparsity problem and high computational demand problem in multi-instance image learning tasks, and improve the accuracy and speed of multi-instance image classification.

[0062] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A multi-instance image classification method based on pre-detection feature fusion, characterized in that, Includes the following steps: Multi-instance imaging is obtained using a multi-instance image acquisition tool; The multi-instance images are input into the pre-detection module for region pre-detection to obtain the pre-detection region for each instance; The pre-detection region is used to perform single-instance feature extraction to obtain pre-detection features; The pre-detection features are weighted and fused using the instance fusion module to obtain an instance fusion view; The pre-detection features are compressed using a spatial fusion module to obtain a spatial fusion view. The target category is obtained by jointly predicting the multi-instance classification using the instance fusion view and the spatial fusion view; The pre-detection feature encoder includes a single-instance feature extractor and a single-instance feature mixer. The single-instance feature extractor includes a feature dimensionality reduction module and a feature aggregation module. The feature dimensionality reduction module is composed of a convolutional layer with a stride of 2, a random pooling layer, and a hybrid pooling layer in parallel to achieve feature dimensionality reduction. The feature aggregation module is composed of a linear mapping layer, a state space model, and a dot product attention layer in serial order to achieve feature aggregation. The single-instance feature mixer is used to concatenate dimensionality reduction features and aggregated features along the channel dimension, and uses a grouped convolution with 2 groups and a pointwise convolution with 1 group to achieve channel compression and channel information mixing for the dimensionality reduction features and aggregated features. The specific content of obtaining the instance fusion view is as follows: the multi-instance features are arranged according to the instance order, the sequence number is converted into a binary number to obtain the sequence code, the sequence code is accumulated with the corresponding instance features and mapped to a one-dimensional sequence to obtain the multi-instance sequence features; The multi-instance sequence features are input into a 2D state-space model and a multilayer perceptron to predict the weights corresponding to each instance. to Assign each instance to its corresponding weight. Multiply and pass through a one-dimensional convolutional layer to obtain a fused view of instances.

2. The multi-instance image classification method based on pre-detection feature fusion as described in claim 1, characterized in that, The specific details of the step "Obtaining multi-instance imaging through a multi-instance image acquisition tool" are as follows: For the target to be identified, multiple instances of images are acquired using multiple imaging and multi-angle imaging methods in both time and space dimensions; Representative images are selected from the multiple instance images by selecting images with different imaging times and different imaging angles, and similar images with adjacent imaging times and the same imaging angle are deleted to improve data representativeness.

3. The multi-instance image classification method based on pre-detection feature fusion as described in claim 1, characterized in that, The specific content of the step "inputting the multi-instance images into the pre-detection module for region pre-detection" is as follows: The aspect ratio, width and height dimensions, and center offset distance of the prior bounding box are set for the multi-instance imaging. The center offset distance is calculated by forming a matrix containing only two elements from the difference between the horizontal and vertical coordinates of the geometric center point of the image and the geometric center point of the prior bounding box. The difference between the horizontal coordinates in the matrix is ​​divided by the image width, and the difference between the vertical coordinates in the matrix is ​​divided by the image height. Then, the Frobenius norm of the matrix is ​​calculated to obtain the center offset distance. The pre-detection module uses the multi-stage detection model HTC to perform region detection, and selects the predicted region with the largest difference between confidence and center offset distance as the pre-detection region.

4. The multi-instance image classification method based on pre-detection feature fusion as described in claim 3, characterized in that, The 2D state space model employs a bidirectional scanning mechanism in the channel dimension to achieve feature mixing across instances, and in the spatial dimension to achieve feature mixing across spatial locations.

5. The multi-instance image classification method based on pre-detection feature fusion as described in claim 1, characterized in that, The specific content of the step "compressing the pre-detection features using the spatial fusion module to obtain a spatial fusion view" is as follows: Each instance of the pre-detection feature is mapped to the same spatial plane and horizontally stitched together to obtain a spatial stitched feature map with a height of h and a width of n×w, thereby realizing the spatial dimension of instance rearrangement; The spatial stitching feature map is used to obtain a spatial fusion view using horizontal and vertical asymmetric convolutional blocks.

6. The multi-instance image classification method based on pre-detection feature fusion as described in claim 5, characterized in that, The specific content of the horizontal and vertical asymmetric convolution block is as follows: first, perform ordinary convolution on the features, then perform asymmetric horizontal convolution of length n×d and asymmetric vertical convolution of length d, where n is the number of instances and d is the convolution kernel length parameter.

7. The multi-instance image classification method based on pre-detection feature fusion as described in claim 1, characterized in that, The specific content of the step "jointly predicting multi-instance classification by combining the instance fusion view and the spatial fusion view" is as follows: Global average pooling and multilayer perceptron prediction are performed on the instance fusion view and spatial fusion view in single-view and dual-view modes to obtain the instance fusion view prediction value, spatial fusion view prediction value and dual-view prediction value. The instance fusion view prediction value, spatial fusion view prediction value, and dual view prediction value are concatenated into a one-dimensional vector, and a multilayer perceptron is used to reduce the feature dimensionality of the one-dimensional vector to achieve multi-view joint prediction.

Images