An image fusion classification method and device based on segmentation guided data enhancement

Abstract

This invention provides an image fusion and classification method and apparatus based on segmentation-guided data augmentation, relating to the field of computer vision technology. The method includes: segmenting each image in the training dataset using a FastSAM network; calculating the cosine similarity between the image and the segmentation map, selecting a balanced segmentation map, and constructing a segmentation dataset; constructing a distance norm metric and information gain to determine the number of segmentation map augmentations to obtain an augmented dataset, and training a classification model. This invention combines image segmentation technology, accurately extracting segmentation features from images to identify key semantic regions and structural features, thereby better maintaining the semantic consistency and structural integrity of images during data augmentation. A dynamic interpolation strategy is introduced to adaptively adjust the parameters and methods of the augmentation operation based on the image's segmentation features and contextual information, achieving diversity and rationality of the augmented data while minimizing changes to the original data distribution.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image fusion classification method and apparatus based on segmentation-guided data augmentation. Background Technology

[0002] With the development of deep learning technology, image classification has been widely applied and occupies a core position in the field of computer vision, finding broad application in various practical scenarios such as autonomous driving, medical image analysis, and facial recognition. Deep neural networks, with their powerful feature extraction capabilities and end-to-end learning mechanisms, have significantly improved the accuracy and efficiency of image classification. However, the performance improvement of deep learning models largely depends on high-quality and diverse training data, enabling them to maintain high classification accuracy even when faced with unseen data. In practical applications, obtaining large amounts of high-quality labeled data often presents numerous challenges. On the one hand, data collection and labeling processes are typically time-consuming and labor-intensive, especially in fields requiring specialized knowledge, where the cost of obtaining high-quality labeled data is even higher. On the other hand, the distribution of training data is often uneven, with data for certain categories potentially being scarce, leading to significant underperformance of the model in these categories. Furthermore, with the increasing popularity of resource-constrained platforms such as mobile and embedded devices, the demand for lightweight models (such as MobileNet and SqueezeNet) is growing. However, due to limitations in the number of parameters and computational cost, their learning ability is relatively weak, and they are easily affected by the scarcity and uneven distribution of training data, resulting in insufficient generalization ability of the model and affecting the actual application effect.

[0003] Data augmentation techniques, as an important means to improve model performance, have received widespread attention. Traditional data augmentation methods mainly include image flipping, cropping, rotation, scaling, and color transformation. By performing simple geometric and color transformations on the original image, they expand the training dataset, thereby alleviating the data scarcity problem to some extent. However, traditional data augmentation methods often ignore the semantic information and structural features of images, which may lead to deviations in semantic consistency and structural integrity of the augmented image. Generative data augmentation methods that have emerged in recent years, such as automatic data augmentation (AutoAugment) and adversarial example generation (Adversarial Example Generation), significantly improve data diversity and model robustness by learning data distribution or generating new samples. However, these generative methods are usually accompanied by significant computational overhead, especially during the training phase, requiring additional computing resources to generate and process augmented data. At the same time, generative methods may introduce data distribution bias in some cases, meaning that the generated data may deviate from the distribution of real data, resulting in the model's performance in real-world environments not necessarily improving, and may even deteriorate.

[0004] Current lightweight image classification algorithms have achieved good performance, but they perform poorly when dealing with small and unevenly distributed data samples. Therefore, the question is how to improve the classification accuracy and generalization ability of lightweight models or models with limited computing resources by addressing the issues of small data sample size and uneven distribution.

[0005] With the application of data augmentation technology in the field of computer vision, data augmentation methods have transformed from traditional methods to various advanced generative methods, and have made some progress in the field of image classification.

[0006] First, there are issues with data scarcity and uneven distribution. Generative data augmentation methods sometimes generate samples that are inconsistent with the real data distribution, leading to a decline in the predictive performance of the model in practical applications. Furthermore, fixed augmentation strategies cannot be adjusted according to dynamic changes in the data, limiting the model's adaptability to different data distributions, thus affecting the model's generalization ability and robustness.

[0007] Secondly, the semantic consistency of the augmented data is insufficient. Traditional data augmentation methods mainly focus on transforming the geometric and color attributes of images, neglecting the semantic information and structural features of the images. Excessive geometric or color transformations may lead to the loss or distortion of key features, destroying the overall semantic consistency and structural integrity of the image, and potentially generating invalid or noisy data, thereby affecting the model's correct understanding of image content and classification performance.

[0008] Finally, there is the computational cost and overfitting risk of the model. While advanced data augmentation methods can significantly increase data diversity, their complex computational processes and high computational resource requirements limit their widespread application in real-time applications and resource-constrained devices, making it difficult to balance the increase in the amount of augmented data with the improvement in model performance. Summary of the Invention

[0009] To address the technical challenge of improving image classification accuracy in lightweight models with limited data and resources by utilizing data augmentation, image segmentation, and fusion techniques in existing technologies, this invention provides an image fusion classification method and apparatus based on segmentation-guided data augmentation. The technical solution is as follows:

[0010] On the one hand, an image fusion classification method based on segmentation-guided data augmentation is provided. This method is implemented by an image fusion classification device and includes:

[0011] S1. Obtain the training dataset; the training dataset includes multiple images from multiple categories.

[0012] S2. For each image in the training dataset, use the FastSAM network to segment it, and obtain the segmentation map corresponding to each image; the segmentation map contains the key semantic regions and structural features of the image.

[0013] S3. Calculate the cosine similarity between each image and its corresponding segmentation map, select a balanced segmentation map based on the cosine similarity, and construct a segmentation dataset based on the balanced segmentation map.

[0014] S4. Construct the distance norm metric and information gain. Determine the number of segmentation map enhancements based on the distance norm metric and information gain. Select enhancement images from the segmentation dataset based on the number of segmentation map enhancements. Fuse the enhancement images with the training dataset to obtain the enhancement dataset.

[0015] S5. Using ResNet50 as the classification model, the classification model is trained based on the augmented dataset to obtain a trained classification model.

[0016] S6. Obtain the image to be classified, input the image to be classified into the trained classification model, and obtain the image classification result.

[0017] Optionally, in S3, the cosine similarity between each image and its corresponding segmentation map is calculated, a balanced segmentation map is selected based on the cosine similarity, and a segmentation dataset is constructed based on the balanced segmentation map, including:

[0018] S31. Calculate the cosine similarity between each image and the corresponding segmentation image using the following formula (1):

[0019] (1)

[0020] In the formula, Represents cosine similarity. Represents the first in the training dataset One image, Indicates the first The segmentation map corresponding to each image.

[0021] S32. Based on the preset cosine similarity threshold, the segmentation map is divided into undersegmented map, balanced map, and oversegmented map. The balanced map is selected to construct the segmentation dataset.

[0022] Optionally, S4 constructs a distance norm metric and information gain, and determines the amount of segmentation map augmentation based on the distance norm metric and information gain, including:

[0023] S41. Construct a distance norm metric.

[0024] S42. Construct information gain.

[0025] S43. Construct a loss metric based on the distance norm metric and information gain.

[0026] S44. Minimize the loss metric to determine the number of augmentations in the segmentation map.

[0027] Alternatively, the distance norm metric in S41 is shown in equation (2) below:

[0028] (2)

[0029] In the formula, Represents the distance norm metric. Indicates the number of enhanced images. This represents the average feature vector of the augmented dataset. This represents the average feature vector of the training dataset. This represents the standard deviation vector of the augmented dataset. This represents the standard deviation vector of the training dataset.

[0030] Optionally, the information gain in S42 is as shown in equation (3) below:

[0031] (3)

[0032] In the formula, Represents the information gain function. Indicates the number of enhanced images. This represents the first hyperparameter used for adjustment. Indicates the size of the training dataset. This represents the second hyperparameter used for adjustment.

[0033] Alternatively, the loss metric in S43 is as shown in equation (4) below:

[0034] (4)

[0035] In the formula, Represents the loss metric. Indicates the number of enhanced images. Indicates the third hyperparameter. Represents the distance norm metric. This represents the fourth hyperparameter. This represents the information gain function.

[0036] Optionally, in S4, augmented images are selected from the segmentation dataset based on the number of segmentation map augmentations, and these augmented images are fused with the training dataset to obtain an augmented dataset, including:

[0037] The balanced segmentation images in the segmentation dataset are sorted in descending order according to their cosine similarity. Enhanced images are selected from the sorted images based on the number of enhancements required. The enhanced images are then fused with the corresponding images in the training dataset to obtain the enhanced dataset.

[0038] On the other hand, an image fusion classification apparatus based on segmentation-guided data augmentation is provided. This apparatus is applied to an image fusion classification method based on segmentation-guided data augmentation. The apparatus includes:

[0039] The acquisition module is used to acquire the training dataset, which includes multiple images from multiple categories.

[0040] The segmentation module is used to segment each image in the training dataset using the FastSAM network, resulting in a segmentation map for each image. The segmentation map contains the key semantic regions and structural features of the image.

[0041] The segmentation dataset construction module is used to calculate the cosine similarity between each image and its corresponding segmentation map, select a balanced segmentation map based on the cosine similarity, and construct a segmentation dataset based on the balanced segmentation map.

[0042] The augmented dataset construction module is used to construct the distance norm metric and information gain, determine the number of segmentation map augmentations based on the distance norm metric and information gain, select augmented images from the segmentation dataset based on the number of segmentation map augmentations, and fuse the augmented images with the training dataset to obtain the augmented dataset.

[0043] The training module is used to train the classification model using ResNet50 as the classification model based on the augmented dataset, and obtain the trained classification model.

[0044] The classification module is used to acquire the image to be classified, input the image to be classified into the trained classification model, and obtain the image classification result.

[0045] Optionally, the dataset segmentation building module is further used for:

[0046] S31. Calculate the cosine similarity between each image and the corresponding segmentation image using the following formula (1):

[0047] (1)

[0048] In the formula, Represents cosine similarity. Represents the first in the training dataset One image, Indicates the first The segmentation map corresponding to each image.

[0049] S32. Based on the preset cosine similarity threshold, the segmentation map is divided into under-segmentation map, balanced segmentation map, and over-segmentation map. The balanced segmentation map is selected to construct the segmentation dataset.

[0050] Optionally, the dataset building module can be enhanced for further use in:

[0051] S41. Construct a distance norm metric.

[0052] S42. Construct information gain.

[0053] S43. Construct a loss metric based on the distance norm metric and information gain.

[0054] S44. Minimize the loss metric to determine the number of augmentations in the segmentation map.

[0055] Alternatively, the distance norm metric is as shown in equation (2):

[0056] (2)

[0057] In the formula, Represents the distance norm metric. Indicates the number of enhanced images. This represents the average feature vector of the augmented dataset. This represents the average feature vector of the training dataset. This represents the standard deviation vector of the augmented dataset. This represents the standard deviation vector of the training dataset.

[0058] Alternatively, the information gain is as shown in equation (3):

[0059] (3)

[0060] In the formula, Represents the information gain function. Indicates the number of enhanced images. This represents the first hyperparameter used for adjustment. Indicates the size of the training dataset. This represents the second hyperparameter used for adjustment.

[0061] Alternatively, the loss metric is as shown in equation (4):

[0062] (4)

[0063] In the formula, Represents the loss metric. Indicates the number of enhanced images. Indicates the third hyperparameter. Represents the distance norm metric. This represents the fourth hyperparameter. This represents the information gain function.

[0064] Optionally, the dataset building module can be enhanced for further use in:

[0065] The balanced segmentation images in the segmentation dataset are sorted in descending order according to their cosine similarity. Enhanced images are selected from the sorted images based on the number of enhancements required. The enhanced images are then fused with the corresponding images in the training dataset to obtain the enhanced dataset.

[0066] On the other hand, an image fusion classification device is provided, the image fusion classification device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the above-described image fusion classification methods based on segmentation-guided data augmentation is implemented.

[0067] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described image fusion classification methods based on segmentation-guided data augmentation.

[0068] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0069] In this invention, image classification performance largely depends on the quality and diversity of the training data. When data is scarce, the core issue is how to improve the model's generalization ability and avoid overfitting without disrupting the original data distribution. Currently, traditional data augmentation methods, such as flipping, random cropping, and color contrast transformation, are mainly used to address the problem of limited data. These methods can achieve good results when data is plentiful and can increase data diversity to some extent. However, they still have limitations in classification problems with limited datasets. This invention proposes that segmentation maps generated by segmentation networks can further extract structural and semantic features of images, generating diverse augmented data. This helps the model learn richer features, reduces ineffective augmentation that may result from traditional augmentation methods, effectively expands the training dataset, and alleviates the problem of insufficient data.

[0070] Since data generated through segmentation networks also carries the risk of introducing invalid augmentation data and noise, this invention aims to further improve the quality of augmented data by using a distance norm metric to measure the consistency between the augmented data and the original data distribution. This ensures that the augmented data does not deviate from the semantic distribution of the original data, further reducing the risk of overfitting. Simultaneously, an information gain optimization strategy is employed to evaluate the value of the augmented data. By dynamically adjusting the amount of augmented data, it ensures that the generated augmented data has high information content. Only the segmentation maps most helpful in improving model performance are added to the training set, avoiding the introduction of redundant data and thus optimizing model performance under conditions of data scarcity.

[0071] This invention optimizes cosine similarity and information gain to intelligently select high-quality augmented data, reducing the workload of manual screening.

[0072] This invention significantly improves the performance of several classic image classification models, increasing the Top-1 accuracy of ResNet-50 from 77.02% to 81.72% and the Top-1 accuracy of RepViT from 85.61% to 87.03%. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart of an image fusion and classification method based on segmentation-guided data enhancement provided in an embodiment of the present invention;

[0075] Figure 2 This is a diagram illustrating the overall structure of an image fusion and classification method based on segmentation-guided data enhancement, as provided in an embodiment of the present invention.

[0076] Figure 3 This is a schematic diagram of an image fusion classification method based on segmentation-guided data enhancement provided in an embodiment of the present invention;

[0077] Figure 4 This is a flowchart of the FastSAM segmentation process provided in an embodiment of the present invention;

[0078] Figure 5 This is a flowchart of cosine similarity filtering provided in an embodiment of the present invention;

[0079] Figure 6 This is a flowchart of the optimal data selection strategy provided in the embodiments of the present invention;

[0080] Figure 7 This is a block diagram of an image fusion and classification device based on segmentation-guided data enhancement provided in an embodiment of the present invention;

[0081] Figure 8 This is a schematic diagram of the structure of an image fusion classification device provided in an embodiment of the present invention. Detailed Implementation

[0082] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0083] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0084] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0085] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0086] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0087] This invention provides an image fusion classification method based on segmentation-guided data augmentation. This method can be implemented by an image fusion classification device, which can be a terminal or a server. Figure 1 The flowchart shown is for an image fusion classification method based on segmentation-guided data augmentation. The processing flow of this method may include the following steps:

[0088] S1. Obtain the training dataset; the training dataset includes multiple images from multiple categories.

[0089] In one feasible implementation, the image classification dataset used in this invention is the Mini-ImageNet dataset, which has 100 classes, 48 ​​images per class, and a total of 48,000 images with an image size of 224×224. This dataset is widely used for evaluating image classification tasks.

[0090] This invention aims to provide a segmentation-based image fusion classification data augmentation strategy, such as SegAug. Figure 2 As shown, this addresses the issues of data sparsity and inconsistent distribution.

[0091] S2. For each image in the training dataset, use the FastSAM network to segment it, and obtain the segmentation map corresponding to each image; the segmentation map contains the key semantic regions and structural features of the image.

[0092] In one feasible implementation, Figure 3 This is a schematic diagram of the method flow according to an embodiment of the present invention. The entire process can be divided into four steps: image segmentation, optimal data selection, model training, and model inference.

[0093] Specifically, the FastSAM model is used for the input image. Perform segmentation and generate segmentation map As shown in Equation (1), this process ensures high quality and structural consistency of the image data. The segmented dataset is obtained, providing the basic data for the subsequent enhancement process.

[0094] (1)

[0095] FastSAM (Fast Segment Anything Model) is a lightweight segmentation network that effectively extracts structural features from key regions of an image, providing semantic guidance for subsequent enhancement, avoiding the destruction of semantic structure by traditional enhancement methods, and distinguishing between foreground and background. Compared with other traditional SAM (Segment Anything Model) frameworks, FastSAM is more efficient, significantly reducing data preprocessing time. The FastSAM segmentation process is as follows: Figure 4 As shown, generating high-quality segmentation feature maps helps the model identify key structures in the image, ensuring that structural changes during data augmentation are reasonably preserved and that the semantic consistency of the original data is maintained.

[0096] In this embodiment of the invention, during the data preprocessing stage, a segmentation model is used for feature extraction to extract key semantic regions and structural features from the image. This segmentation information helps identify important objects and background regions in the image, providing guidance for subsequent data augmentation operations. FastSAM is used instead of the traditional segmentation model to reduce preprocessing computational overhead and adapt to resource-constrained devices. Augmentation samples are preferentially generated in sparse data regions to reduce redundancy in dense regions and improve data augmentation efficiency.

[0097] S3. Calculate the cosine similarity between each image and its corresponding segmentation map, select a balanced segmentation map based on the cosine similarity, and construct a segmentation dataset based on the balanced segmentation map.

[0098] In one feasible implementation, segmentation maps are filtered based on cosine similarity, and adaptive data augmentation is performed using the Information Gain Optimization (IGO) method. Balanced segmentation maps are selected based on similarity values, eliminating over-segmented and under-segmented images.

[0099] Optionally, step S3 above may include the following steps S31-S32:

[0100] S31. Determine the similarity between the image and the segmentation map by calculating the cosine similarity.

[0101] Specifically, the cosine similarity between each image and its corresponding segmentation map is calculated using the following formula (2):

[0102] (2)

[0103] In the formula, Represents cosine similarity. Represents the first in the training dataset One image, Indicates the first The segmentation map corresponding to each image.

[0104] S32. Based on the preset cosine similarity threshold, the segmentation map is divided into under-segmentation map, balanced segmentation map, and over-segmentation map. The balanced segmentation map is selected to construct the segmentation dataset.

[0105] In one feasible implementation, based on the calculation results, balanced segmentation maps are selected, i.e., those segmentation maps that are highly consistent with the structure of the original image, ensuring that the structural changes in the augmented data conform to the semantic features of the original data. Images with high cosine similarity are marked as under-segmented and removed; images with low similarity are marked as over-segmented and also removed. The cosine similarity filtering process is as follows: Figure 5 As shown.

[0106] By calculating cosine similarity, the effectiveness of adaptive data augmentation is guaranteed. This ensures that the augmented data increases sample diversity while maintaining consistency with the original data, avoiding the negative impact of excessive or insufficient augmented data on the training process. Augmentation parameters are dynamically adjusted based on the contextual information of the segmentation map and the original image, achieving scene-adaptive augmentation.

[0107] In this embodiment of the invention, a dynamic interpolation strategy is proposed, which adjusts the parameters of the data augmentation operation according to the segmentation features and uses cosine similarity screening to ensure that the augmented data achieves a balance between diversity and distribution consistency.

[0108] S4. Construct the distance norm metric and information gain. Determine the number of segmentation map enhancements based on the distance norm metric and information gain. Select enhancement images from the segmentation dataset based on the number of segmentation map enhancements. Fuse the enhancement images with the training dataset to obtain the enhancement dataset.

[0109] In one feasible implementation, the number of interpolated samples is dynamically adjusted using an information gain optimization strategy. This prioritizes generating augmented data in sparse regions of the data, ensuring uniform data distribution and avoiding excessive redundant data in dense regions. The processed dataset is then used for subsequent steps.

[0110] Optionally, constructing the distance norm metric and information gain in S4, and determining the amount of segmentation map enhancement based on the distance norm metric and information gain, may include the following steps S41-S44:

[0111] S41. Construct a distance norm metric.

[0112] In one feasible implementation, a distribution consistency metric is first defined to measure the distribution consistency between the original dataset and the augmented dataset. This metric combines the mean and variance between the input image and the segmented image, as shown in Equation (3):

[0113] (3)

[0114] In the formula, Represents the distance norm metric. Indicates the number of enhanced images. This represents the average feature vector of the augmented dataset. This represents the average feature vector of the training dataset. This represents the standard deviation vector of the augmented dataset. This represents the standard deviation vector of the training dataset. Distance specification evaluates the semantic similarity between datasets by taking into account the difference between the mean and standard deviation. The smaller the value, the higher the consistency between the augmented dataset and the original dataset, which is crucial for maintaining model performance.

[0115] S42. Construct information gain.

[0116] In one feasible implementation, information gain optimization is used to adjust the amount of augmented data, ensuring that the added data effectively improves model performance. The information gain function is as follows:

[0117] (4)

[0118] In the formula, Represents the information gain function. Indicates the number of enhanced images. This represents the first hyperparameter used for adjustment. Indicates the size of the training dataset. This represents the second hyperparameter used for adjustment. Information gain optimization ensures the optimal amount of data augmentation, avoids excessive noise introduced by too many samples, and further optimizes the training process by balancing performance gains and the risk of overfitting.

[0119] S43. Construct a loss metric based on the distance norm metric and information gain.

[0120] Optionally, the distance norm is used as a metric. and information gain Combined into a new loss metric:

[0121] (5)

[0122] In the formula, Represents the loss metric. Indicates the number of enhanced images. Indicates the third hyperparameter. This represents the fourth hyperparameter, which controls the weights of distribution consistency and information gain in the final loss. Represents the distance norm metric. This represents the information gain function.

[0123] S44. Minimize the loss metric to determine the number of augmentations in the segmentation map.

[0124] In one feasible implementation, by minimizing this comprehensive loss, it is ensured that the augmented data not only maintains the distributional consistency of the original data, but also maximizes its contribution to model performance.

[0125] This invention designs a distance normalization metric to quantify the distributional differences between augmented and original data from the dimensions of mean and variance, thus constraining the rationality of the generated data. An information gain optimization function is proposed to dynamically balance the number of augmented samples with model performance gain, avoiding the risk of overfitting caused by redundant data. By jointly optimizing distribution consistency and information gain through a comprehensive loss function, the optimal selection of augmented samples is achieved.

[0126] Optionally, in S4, augmented images are selected from the segmentation dataset based on the number of segmentation map augmentations, and these augmented images are fused with the training dataset to obtain an augmented dataset, including:

[0127] The balanced segmentation maps in the segmentation dataset are sorted in descending order according to their cosine similarity. With a step size of 50, the segmentation maps are traversed sequentially and merged with the original map. The optimal number of segmentation map samples to augment is determined based on the loss metric, resulting in the augmented dataset.

[0128] In one feasible implementation, the optimal data augmentation samples are selected and fused with the original image data. This ensures that the augmented dataset is both diverse and consistent with the distribution of the original data, thereby improving the model's performance and efficiency. The optimal data selection process is as follows: Figure 6 As shown.

[0129] In this embodiment of the invention, an optimal data selection strategy is proposed. First, a distribution consistency evaluation index is used to maintain the consistency of data distribution. Second, an information gain optimization index is designed to dynamically adjust the number of augmented samples. Finally, a comprehensive loss function is used to select the optimal number of augmented samples while maintaining the consistency of data distribution, thus avoiding overfitting caused by excessive augmentation.

[0130] S5. Using ResNet50 as the classification model, the classification model is trained based on the augmented dataset to obtain a trained classification model.

[0131] In one feasible implementation, the processed data from steps S2, S3, and S4 are input into the classification model, and cross-validation and hyperparameter tuning are used to ensure that the model performance reaches its optimal level. The optimal model parameters are saved during training for subsequent inference.

[0132] Specifically, experiments were conducted using classic and state-of-the-art classification models: ResNet50, RepViT, MobileViT, EfficientFormerV2, and FastViT, respectively. In this invention, ResNet50 was used as the base classification model. This model consists of a 50-layer residual network, enabling effective deep-level image feature extraction. During training, model performance was evaluated through cross-validation, and gradient descent optimization was performed using the Adam optimizer.

[0133] Furthermore, the dataset is first divided into a training set and a validation set; then, the training set data is processed through steps S2, S3, and S4 to obtain the augmented dataset, which serves as the input to the classification model; finally, hyperparameter tuning is performed, adjusting hyperparameters such as the learning rate, batch size, and number of training epochs using a random search method to ensure that the model achieves optimal performance on the training data. The model is then trained using the training set data and saved.

[0134] Furthermore, the best-performing model obtained through training is used for inference. The trained model is loaded from a specified path, ensuring that the model parameters are correct. Preprocessed test data is input into the trained model for classification. By comparing the predicted results with the true labels, classification accuracy and other evaluation metrics are calculated to assess the model's inference performance.

[0135] Regarding model parameter details, when using the ResNet-50 model, the batch size was set to 160, the epochs to 100, and the learning rate to 0.1; when using the RepViT model, the batch size was set to 256, the epochs to 300, and the learning rate to 0.001. All other parameters were fixed and remained consistent throughout the experiments.

[0136] This invention uses the PyTorch framework, and the specific experiments were conducted on an NVIDIA GeForce RTX 3090 for model training and evaluation.

[0137] S6. Obtain the image to be classified, input the image to be classified into the trained classification model, and obtain the image classification result.

[0138] This invention aims to improve the sample size and diversity of image classification data by utilizing image segmentation methods and information gain optimization strategies. The cosine similarity between the segmented image and the original image serves as a constraint on the augmentation operation, ensuring semantic consistency between the generated data and the original image. Furthermore, distribution consistency and information gain optimization are employed to dynamically adjust the augmented data, thereby enhancing the model's generalization ability and prediction accuracy.

[0139] Existing patent (CN116740482A) discloses an underwater small-sample augmentation method based on image segmentation and fusion. This method extracts single-sample information from the image of the sample to be fused, converts the background color hue to approximate the hue of the underwater background to be fused, and uses Poisson fusion to generate an image with added samples, achieving the effects of enriching the target recognition information and balancing the training data. The main differences between this invention and that patent are: 1) It uses a FastSAM network to segment the image and extract key semantic regions and structural features; 2) It proposes a dynamic augmentation sample and fusion strategy, adjusting the parameters of the data augmentation operation according to the segmentation features, and using cosine similarity screening to ensure a balance between the diversity and distribution consistency of the augmented data; 3) It significantly improves the accuracy on classification models such as RepViT.

[0140] In this invention, image classification performance largely depends on the quality and diversity of the training data. When data is scarce, the core issue is how to improve the model's generalization ability and avoid overfitting without disrupting the original data distribution. Currently, traditional data augmentation methods, such as flipping, random cropping, and color contrast transformation, are mainly used to address the problem of limited data. These methods can achieve good results when data is plentiful and can increase data diversity to some extent. However, they still have limitations in classification problems with limited datasets. This invention proposes that segmentation maps generated by segmentation networks can further extract structural and semantic features of images, generating diverse augmented data. This helps the model learn richer features, reduces ineffective augmentation that may result from traditional augmentation methods, effectively expands the training dataset, and alleviates the problem of insufficient data.

[0141] Since data generated through segmentation networks also carries the risk of introducing invalid augmentation data and noise, this invention aims to further improve the quality of augmented data by using a distance norm metric to measure the consistency between the augmented data and the original data distribution. This ensures that the augmented data does not deviate from the semantic distribution of the original data, further reducing the risk of overfitting. Simultaneously, an information gain optimization strategy is employed to evaluate the value of the augmented data. By dynamically adjusting the amount of augmented data, it ensures that the generated augmented data has high information content. Only the segmentation maps most helpful in improving model performance are added to the training set, avoiding the introduction of redundant data and thus optimizing model performance under conditions of data scarcity.

[0142] This invention optimizes cosine similarity and information gain to intelligently select high-quality augmented data, reducing the workload of manual screening.

[0143] This invention significantly improves the performance of several classic image classification models, increasing the Top-1 accuracy of ResNet-50 from 77.02% to 81.72% and the Top-1 accuracy of RepViT from 85.61% to 87.03%.

[0144] Figure 7 This is a block diagram of an image fusion classification apparatus based on segmentation-guided data augmentation, according to an exemplary embodiment. The apparatus is used in an image fusion classification method based on segmentation-guided data augmentation. (Refer to...) Figure 7 The device includes an acquisition module 310, a segmentation module 320, a segmentation dataset construction module 330, an augmentation dataset construction module 340, a training module 350, and a classification module 360. Wherein:

[0145] The acquisition module 310 is used to acquire the training dataset; wherein, the training dataset includes multiple images of multiple categories.

[0146] The segmentation module 320 is used to segment each image in the training dataset using the FastSAM network to obtain a segmentation map for each image; the segmentation map contains the key semantic regions and structural features of the image.

[0147] The segmentation dataset construction module 330 is used to calculate the cosine similarity between each image and the corresponding segmentation map, select a balanced segmentation map based on the cosine similarity, and construct a segmentation dataset based on the balanced segmentation map.

[0148] The augmented dataset construction module 340 is used to construct the distance norm metric and information gain, determine the number of segmentation map augmentations based on the distance norm metric and information gain, select augmented images from the segmentation dataset based on the number of segmentation map augmentations, and fuse the augmented images with the training dataset to obtain the augmented dataset.

[0149] Training module 350 is used to train the classification model using ResNet50 as the classification model based on the augmented dataset, and obtain the trained classification model.

[0150] The classification module 360 ​​is used to acquire the image to be classified, input the image to be classified into the trained classification model, and obtain the image classification result.

[0151] In this invention, image classification performance largely depends on the quality and diversity of the training data. When data is scarce, the core issue is how to improve the model's generalization ability and avoid overfitting without disrupting the original data distribution. Currently, traditional data augmentation methods, such as flipping, random cropping, and color contrast transformation, are mainly used to address the problem of limited data. These methods can achieve good results when data is plentiful and can increase data diversity to some extent. However, they still have limitations in classification problems with limited datasets. This invention proposes that segmentation maps generated by segmentation networks can further extract structural and semantic features of images, generating diverse augmented data. This helps the model learn richer features, reduces ineffective augmentation that may result from traditional augmentation methods, effectively expands the training dataset, and alleviates the problem of insufficient data.

[0152] Since data generated through segmentation networks also carries the risk of introducing invalid augmentation data and noise, this invention aims to further improve the quality of augmented data by using a distance norm metric to measure the consistency between the augmented data and the original data distribution. This ensures that the augmented data does not deviate from the semantic distribution of the original data, further reducing the risk of overfitting. Simultaneously, an information gain optimization strategy is employed to evaluate the value of the augmented data. By dynamically adjusting the amount of augmented data, it ensures that the generated augmented data has high information content. Only the segmentation maps most helpful in improving model performance are added to the training set, avoiding the introduction of redundant data and thus optimizing model performance under conditions of data scarcity.

[0153] This invention optimizes cosine similarity and information gain to intelligently select high-quality augmented data, reducing the workload of manual screening.

[0154] This invention significantly improves the performance of several classic image classification models, increasing the Top-1 accuracy of ResNet-50 from 77.02% to 81.72% and the Top-1 accuracy of RepViT from 85.61% to 87.03%.

[0155] Figure 8 This is a schematic diagram of the structure of an image fusion classification device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the image fusion classification device may include the above-mentioned Figure 7 The image fusion classification apparatus shown is based on segmentation-guided data augmentation. Optionally, the image fusion classification apparatus 410 may include a first processor 2001.

[0156] Optionally, the image fusion classification device 410 may also include a memory 2002 and a transceiver 2003.

[0157] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0158] The following is combined with Figure 8 A detailed description of each component of the image fusion classification device 410 is provided below:

[0159] The first processor 2001 is the control center of the image fusion classification device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0160] Optionally, the first processor 2001 can perform various functions of the image fusion classification device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0161] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 8 CPU0 and CPU1 are shown in the diagram.

[0162] In a specific implementation, as one example, the image fusion classification device 410 may also include multiple processors, for example... Figure 8 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0163] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0164] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the image fusion classification device 410. Figure 8 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0165] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0166] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 8 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0167] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the image fusion classification device 410. Figure 8 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0168] It should be noted that, Figure 8 The structure of the image fusion classification device 410 shown does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0169] Furthermore, the technical effect of the image fusion classification device 410 can be referred to the technical effect of the image fusion classification method based on segmentation-guided data enhancement described in the above method embodiments, and will not be repeated here.

[0170] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0171] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0172] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0173] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0174] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0175] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0178] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0181] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image fusion classification method based on segmentation-guided data augmentation, characterized in that, The method includes: S1. Obtain the training dataset; wherein, the training dataset includes multiple images of multiple categories; S2. For each image in the training dataset, use the FastSAM network to segment it, and obtain a segmentation map corresponding to each image; the segmentation map contains the key semantic regions and structural features of the image; S3. Calculate the cosine similarity between each image and its corresponding segmentation map, select a balanced segmentation map based on the cosine similarity, and construct a segmentation dataset based on the balanced segmentation map; S4. Construct a distance norm metric and information gain, determine the number of segmentation map enhancements based on the distance norm metric and information gain, select enhancement images from the segmentation dataset based on the number of segmentation map enhancements, and fuse the enhancement images with the training dataset to obtain an enhancement dataset; S5. Using ResNet50 as the classification model, train the classification model based on the augmented dataset to obtain a trained classification model; S6. Obtain the image to be classified, input the image to be classified into the trained classification model, and obtain the image classification result.

2. The image fusion classification method based on segmentation-guided data augmentation according to claim 1, characterized in that, S3 involves calculating the cosine similarity between each image and its corresponding segmentation map, selecting a balanced segmentation map based on the cosine similarity, and constructing a segmentation dataset based on the balanced segmentation map, including: S31. Calculate the cosine similarity between each image and the corresponding segmentation image using the following formula (1): （1） In the formula, Represents cosine similarity. Represents the first in the training dataset One image, Indicates the first Segmentation maps corresponding to each image; S32. Based on a preset cosine similarity threshold, the segmentation map is divided into under-segmentation maps, balanced segmentation maps, and over-segmentation maps. The balanced segmentation map is selected to construct the segmentation dataset.

3. The image fusion classification method based on segmentation-guided data augmentation according to claim 1, characterized in that, The construction of distance norm metric and information gain in S4, and the determination of the segmentation map enhancement amount based on the distance norm metric and information gain, includes: S41. Construct a distance norm metric; S42. Construct information gain; S43. Construct a loss metric based on the distance norm metric and information gain; S44. Minimize the loss metric to determine the number of segmentation map augmentations.

4. The image fusion classification method based on segmentation-guided data augmentation according to claim 3, characterized in that, The distance norm metric in S41 is shown in equation (2) below: （2） In the formula, Represents the distance norm metric. Indicates the number of enhanced images. This represents the average feature vector of the augmented dataset. This represents the average feature vector of the training dataset. This represents the standard deviation vector of the augmented dataset. This represents the standard deviation vector of the training dataset.

5. The image fusion classification method based on segmentation-guided data augmentation according to claim 3, characterized in that, The information gain in S42 is shown in equation (3) below: （3） In the formula, Represents the information gain function. Indicates the number of enhanced images. This represents the first hyperparameter used for adjustment. Indicates the size of the training dataset. This represents the second hyperparameter used for adjustment.

6. The image fusion classification method based on segmentation-guided data augmentation according to claim 3, characterized in that, The loss metric in S43 is shown in equation (4) below: （4） In the formula, Represents the loss metric. Indicates the number of enhanced images. Indicates the third hyperparameter. Represents the distance norm metric. This represents the fourth hyperparameter. This represents the information gain function.

7. The image fusion classification method based on segmentation-guided data augmentation according to claim 1, characterized in that, In step S4, an augmented image is selected from the segmentation dataset based on the number of augmentations in the segmentation image, and the augmented image is fused with the training dataset to obtain an augmented dataset, including: The balanced segmentation images in the segmentation dataset are sorted in descending order according to their cosine similarity. An enhanced image is selected from the sorted images based on the number of enhancements required for each segmentation image. The enhanced image is then fused with the corresponding image in the training dataset to obtain the enhanced dataset.

8. An image fusion classification apparatus based on segmentation-guided data augmentation, wherein the image fusion classification apparatus based on segmentation-guided data augmentation is used to implement the image fusion classification method based on segmentation-guided data augmentation as described in any one of claims 1-7, characterized in that, The device includes: An acquisition module is used to acquire a training dataset; wherein the training dataset includes multiple images of multiple categories; The segmentation module is used to segment each image in the training dataset using the FastSAM network to obtain a segmentation map for each image; the segmentation map contains the key semantic regions and structural features of the image; The segmentation dataset construction module is used to calculate the cosine similarity between each image and the corresponding segmentation map, select a balanced segmentation map based on the cosine similarity, and construct a segmentation dataset based on the balanced segmentation map. An augmented dataset construction module is used to construct a distance norm metric and information gain, determine the number of segmentation map augmentations based on the distance norm metric and information gain, select augmented images from the segmentation dataset based on the number of segmentation map augmentations, and fuse the augmented images with the training dataset to obtain an augmented dataset; The training module is used to train the classification model using ResNet50 as the classification model based on the augmented dataset to obtain the trained classification model. The classification module is used to acquire the image to be classified, input the image to be classified into the trained classification model, and obtain the image classification result.

9. An image fusion and classification device, characterized in that, The image fusion classification device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images