A melanoma image recognition method and device for unbalanced samples

Abstract

This invention discloses a method and apparatus for melanoma image recognition based on imbalanced samples, belonging to the technical field. The method includes: inputting the image to be recognized into a core perceptual region extraction module of a melanoma recognition model to obtain a preliminary feature map; obtaining weight coefficients of the preliminary feature map using a weighted aggregation unit; obtaining a heatmap based on the preliminary feature map, weight coefficients, and class activation mapping; obtaining a core perceptual image based on the heatmap and a segmentation extraction unit; inputting clinical information into a clinical data feature extraction module to obtain clinical feature data; inputting the image to be recognized and the core perceptual image into a dual-path feature extraction unit to obtain image feature information; and inputting the image feature information and clinical feature data into a gating fusion unit to obtain the recognition result of the image to be recognized. Using this invention, the problem of misjudgment caused by high intra-class variability and low inter-class variability in small-sample melanoma images can be solved, reducing computational complexity and improving recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and apparatus for recognizing melanoma images of imbalanced samples. Background Technology

[0002] Artificial intelligence-assisted tumor identification technology is moving from the laboratory to clinical practice, assisting doctors in achieving accurate diagnosis and improving the efficiency of early screening. Image-based melanoma identification refers to using image data of skin lesions and automatically analyzing image features through computer vision and deep learning algorithms to distinguish melanoma from benign lesions (such as pigmented nevi and seborrheic keratosis). In melanoma diagnostic research, dermoscopic images of patients are typically input, and melanoma identification algorithms diagnose the category of the patient's dermoscopic image. The patient's image category includes melanoma and non-melanoma (benign) lesions. Intelligent melanoma identification is a specific application of computer-aided medical diagnosis, and it is more challenging than identifying cancers with high incidence rates and abundant publicly available datasets.

[0003] Current melanoma classification and identification algorithms typically employ data augmentation methods and transfer learning techniques. These methods utilize generative adversarial networks to synthesize high-resolution samples with complex and detailed features, increasing the number of samples from low-level classes in the dataset. They leverage pre-trained models on large-scale image datasets as feature extractors for melanoma classification datasets, fine-tuning the pre-trained model based on melanoma features, and using loss functions and optimizers to optimize model parameters, thereby improving classification performance. However, existing technologies suffer from the following problems: First, melanoma identification methods based on transfer learning typically involve sharing the low-level features of a model. This involves using a model pre-trained on large-scale natural images or publicly available medical datasets (source domain) and transferring the data to melanoma data from a specific hospital / institution (target domain). However, there are distributional differences between the two. If the feature representations of the source and target domains are inconsistent, sharing the low-level features cannot capture the key information of the target domain. Secondly, when doctors make diagnoses, in addition to observing lesion images, they also consider metadata information such as the patient's age, gender, and lesion location. Deep learning algorithms typically require a large amount of data, but melanoma image datasets are relatively small and lack sufficient data; relying solely on image information may not achieve ideal results. Melanoma data is imbalanced, with significant differences in appearance among samples of the same class, leading to a scattered distribution of similar samples in the feature space. Conversely, samples of different classes have high appearance similarity, causing their features to overlap or become close in space. Basic cross-entropy loss only focuses on the classification accuracy of a single sample, lacking explicit constraints on the distribution of samples in the feature space, making it difficult to effectively distinguish these special samples. Summary of the Invention

[0004] To address the problem of uneven sample distribution in existing technologies, this invention provides a method and apparatus for melanoma image recognition based on unevenly distributed samples. The technical solution is as follows: On the one hand, a method for melanoma image recognition based on imbalanced samples is provided. This method is implemented by a melanoma image recognition device for imbalanced samples, and includes: S1. Obtain images of melanoma to be identified, clinical information, and a trained melanoma identification model; The trained melanoma recognition model includes a core perception region extraction module, a dual-path feature fusion module, and a clinical data feature extraction module. The core perception region extraction module includes convolutional units, weighted aggregation units, class activation mapping, and segmentation extraction units. The dual-path feature fusion module includes dual-path feature extraction units and gated fusion units. S2. Input the melanoma image to be identified into the core perception region extraction module. Obtain a preliminary feature map based on the convolution unit. Obtain the weight coefficient of each preliminary feature map based on the preliminary feature map and the weighted aggregation unit. Obtain a heat map based on the preliminary feature map, the weight coefficient and the class activation mapping. Obtain the core perception image based on the heat map and the segmentation extraction unit. S3. Input clinical information into the clinical data feature extraction module to obtain clinical feature data; S4. Input the melanoma image and the core perception image into the dual-path feature extraction unit to obtain image feature information. Input the image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of the melanoma image.

[0005] On the other hand, a melanoma image recognition device for imbalanced samples is provided. This device is applied to a melanoma image recognition method for imbalanced samples, and the device includes: The acquisition unit is used to acquire images of melanoma to be identified, clinical information, and a trained melanoma recognition model. The trained melanoma recognition model includes a core perception region extraction module, a dual-path feature fusion module, and a clinical data feature extraction module. The core perception region extraction module includes convolutional units, weighted aggregation units, class activation mapping, and segmentation extraction units. The dual-path feature fusion module includes dual-path feature extraction units and gated fusion units. The core perception image extraction unit is used to input the melanoma image to be identified into the core perception region extraction module, obtain a preliminary feature map based on the convolution unit, obtain the weight coefficient of each preliminary feature map based on the preliminary feature map and the weighted aggregation unit, obtain a heat map based on the preliminary feature map, the weight coefficient and the class activation mapping, and obtain the core perception image based on the heat map and the segmentation extraction unit. The clinical data feature extraction unit is used to input clinical information into the clinical data feature extraction module to obtain clinical feature data. The feature fusion and recognition unit is used to input melanoma images and core perception images into the dual-path feature extraction unit to obtain image feature information, and input the image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of melanoma images.

[0006] On the other hand, a melanoma image recognition device for imbalanced samples is provided, the melanoma image recognition device for imbalanced samples includes: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for melanoma image recognition for imbalanced samples.

[0007] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for melanoma image recognition of imbalanced samples.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) This invention addresses the misclassification problem in existing methods for small-sample melanoma images caused by high intra-class variability and low inter-class variability: The cost-sensitive learning function based on sample weighting in this embodiment can solve this problem. Based on the sample weighting strategy, different weights are assigned to samples of different categories. By reducing the weight of easily classified samples, the learning ability for samples with weaker representativeness is improved. By utilizing the weight adjustment mechanism and integrating the advantages of multiple functions, the model can learn more discriminative feature representations. Compared with the traditional cross-entropy loss, the proposed cost-sensitive learning function is more suitable for melanoma image diagnosis.

[0009] (2) The core perception feature extraction module and the heterogeneous data fusion module can more accurately and quickly focus on the high-perception regions that the model focuses on in the image: the model learns the importance weight of each region based on the feature information of the image and clinical data, and automatically adjusts the attention intensity to different regions during the feature extraction process according to the unique features of the sample itself, and mines the model's own potential attention to extract core perception features. This helps to reduce computational complexity, while extracting more features, and extracting more discriminative features through fusion, thereby improving accuracy and generalization performance. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a melanoma image recognition method for imbalanced samples provided by an embodiment of the present invention; Figure 2 This is a flowchart of a melanoma image recognition method for imbalanced samples provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the specific structure of an EMA-EfficientNet network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a gated fusion unit provided in an embodiment of the present invention; Figure 5 This is a block diagram of a melanoma image recognition device for imbalanced samples provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a melanoma image recognition device for imbalanced samples provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0013] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0014] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0015] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0016] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0017] This invention provides a method for melanoma image recognition based on imbalanced samples. This method can be implemented using a melanoma image recognition device for imbalanced samples, which can be a terminal or a server. Figure 1 The flowchart shown is for a melanoma image recognition method for imbalanced samples. Figure 2 The flowchart shown is for a melanoma image recognition method for imbalanced samples. The processing flow of this method may include the following steps: S1. Obtain images of melanoma to be identified, clinical information, and a trained melanoma identification model.

[0018] The trained melanoma recognition model includes a core perception region extraction module, a dual-path feature fusion module, and a clinical data feature extraction module. The core perception region extraction module includes convolutional units, weighted aggregation units, class activation mapping, and segmentation extraction units. The dual-path feature fusion module includes dual-path feature extraction units and gated fusion units.

[0019] Class activation mapping is a visualization technique used to generate heatmaps that show the degree of attention a model pays to different regions of an input image when making decisions. Brighter areas in the heatmap represent regions that the model considers more important for its decisions.

[0020] Optionally, the trained melanoma recognition model uses the cost-sensitive learning-based ACLoss loss function for parameter optimization during training. The ACLoss loss function integrates the face recognition loss function ArcFace Loss and the center loss, and the calculation formula is as follows (1): (1) in, It is a hyperparameter that balances the face recognition loss and the centering loss. After a large number of experiments, it was set to 0.1 to balance the contributions of face recognition loss and centering loss in optimizing the model. The ArcFaceLoss function for face recognition is expressed in the following formula (2): (2) in, N n' represents the total number of samples in the current training batch, and n' represents the total number of categories in the classification task. This represents the feature vector of the i-th input sample and the true class of input sample i. The angle between the corresponding weight center vectors, m' represents the additive angular margin, which directly adds a penalty term in the rotation angle space to compress intra-class distance and expand inter-class distance, and s represents the feature scaling factor. This represents the angle between the input sample feature and the center vector of the j-th non-true class.

[0021] The center loss is expressed by the following formula (3): (3) in, Indicates input sample Features Indicates input sample The true category, It is a category The center vector is the average value of the samples in the corresponding category.

[0022] In one feasible implementation, benign and malignant melanoma sample images have low morphological distinguishability and high visual similarity. This special type of sample leads to the distance between similar features in the feature space being greater than the distance between different classes, and the image feature distances of different classes being similar, resulting in misjudgment. To solve the problems of class imbalance, large intra-class distance, and small inter-class distance in melanoma images, an ACLoss function based on cost-sensitive learning is proposed. This function combines the advantages of Arcface Loss and Center Loss, and the face recognition loss function (Arcface Loss) is shown in equation (2) above. The face recognition loss function can effectively strengthen the discriminative boundary between classes, effectively alleviate the problem of small inter-class distance caused by similar lesion features, and force the model to extract the most discriminative fine-grained features in complex visual backgrounds by geometrically constraining the feature distribution.

[0023] Center loss can make features within the same category more consistent and separate features between different categories, thereby enhancing the discriminative power of extracted features. This can improve the model's ability to distinguish between different categories.

[0024] By integrating the center loss and the face recognition loss, a cost-sensitive learning ACLoss function is designed, as shown in Equation (1) above, to optimize the two challenges of class imbalance and large intra-class distance and small inter-class distance in melanoma diagnosis, thereby improving classification performance.

[0025] S2. Input the melanoma image to be identified into the core perception region extraction module. Obtain preliminary feature maps based on the convolution unit. Obtain weight coefficients for each preliminary feature map based on the preliminary feature maps and the weighted aggregation unit. Obtain heat maps based on the preliminary feature maps, weight coefficients, and class activation maps. Obtain core perception images based on the heat maps and the segmentation extraction unit.

[0026] In one feasible implementation, the melanoma image is first input into a module consisting of multiple "CONV" (convolutional layers). The convolutional layers perform convolution operations by sliding convolution kernels across the image, extracting basic features and generating corresponding preliminary feature maps through a forward propagation process. The gradient values ​​of each element within each feature map are calculated using a weighted summation method to obtain the weight coefficients corresponding to each feature map. A heatmap is constructed based on Grad-CAM++, and a binary mask is introduced to visualize the heatmap. This method emphasizes regions corresponding to high intensity values; the higher the clarity and saliency of a region in the heatmap, the more focused the model is on that region. After completing the above steps, the key highlighted regions in the heatmap are segmented, and the mask region is extracted and cropped to obtain the image region containing the key features.

[0027] Optionally, weighted summation or other operations can be performed on the feature map T. Different weights W assign different importance to different parts of the feature map, highlighting key features for calculating class scores. The specific calculation method is as follows: S21. Calculate the output score for class a. Regarding the kth preliminary feature map at position ( i , j Characteristic response value at ) First-order partial derivatives Second-order partial derivatives and third-order partial derivatives .

[0028] Calculate the position weight coefficients based on the second-order partial derivatives, third-order partial derivatives, and characteristic response values: (1) Where (i,j) is used to mark the position of the differentiation point, and (y,z) is used to mark the summation point during traversal, i.e. It is the feature response value calculated for position (i,j). It is the characteristic response value of traversing the summation points. It is an aggregation of global information of the entire activation graph. The table is used to measure the contribution of this position in the weight calculation.

[0029] The calculation method is as follows: (2) The positional weight coefficients are weighted and summed with the first-order partial derivatives processed by the ReLU activation function to obtain the comprehensive weight of the k-th preliminary feature map for category a: (3).

[0030] in, This represents the overall weight of the k-th feature map for category a, which is obtained by weighted summation of gradient information at all spatial locations of the feature map. This represents a nonlinear activation function used to preserve gradient responses that contribute positively to the target class.

[0031] Optionally, S2 obtains a heatmap based on the preliminary feature map, weight coefficients, and class activation mapping, including: S22. Weighted summation of all preliminary feature maps is performed to obtain the core perceptual features. The calculation formula is as follows (4): (4) in, The response value of the core perceptual feature map of category a at position (i,j) represents the importance of position (i,j) for the classification of the target category.

[0032] ReLU activation is applied to the core perceptual features to retain regions that positively contribute to the prediction results. After normalization and upsampling, the final heatmap is generated. In the heatmap, the brighter the region, the higher the model’s attention to that region.

[0033] Optionally, S2 obtains the core perception image based on the heatmap and the segmentation extraction unit, including: S23. Binarize the heat map to generate a binary mask matrix with the same size as the heat map. In the mask matrix, the area with a value of 1 corresponds to the area in the heat map that is higher than the preset threshold, and the area with a value of 0 corresponds to the area that is lower than the preset threshold.

[0034] Locate the maximum activation region in the binary mask matrix, and calculate the center coordinates of the maximum activation region according to the following formula (5): (5) Where U represents the column coordinate of the top-left corner of the ROI rectangle, V represents the row coordinate of the top-left corner of the ROI rectangle, L represents the width of the ROI rectangle, and H represents the height of the ROI rectangle.

[0035] The final set of pixel coordinates of the cropped region is represented by the following formula (6): (6) in, This represents the pixel coordinates of the final cropped area, where m represents the x-coordinate of the center point of the maximum activation region, and n represents the y-coordinate of the center point of the maximum activation region. The height can be set. =1000, The width can be set. =1000. The dimension of Z is H×L.

[0036] In one feasible implementation, the core perceived feature regions are cropped, and a binary mask is applied to the heatmap. The binary mask is a two-dimensional matrix of the same size as the heatmap, with elements containing only 0 and 1 values. When the binary mask is applied to the heatmap, areas with a value of 1 retain the original color information of the heatmap, while areas with a value of 0 are masked and displayed as black. In a melanoma heatmap, the binary mask makes the areas that the model considers to be lesions clearer. The specific location and coverage of the target region are quantified using a region of interest (ROI) covering rectangle. The core parameters of this rectangle include the horizontal coordinates (U) and vertical coordinates (V) of the top-left vertex, as well as the rectangle's horizontal span (L) and vertical span (H). Based on this, the center coordinates of the ROI covering rectangle are further calculated, and these coordinates are consistent with the coordinates of the largest activation point within the region.

[0037] To address the challenge of insufficient feature representation in melanoma images, this invention proposes using class activation mapping. By visualizing high-response regions that the model focuses on during decision-making through heatmaps, mask regions are segmented and used to extract core perceptual region images. Core perceptual features are extracted by mining the model's own potential attention.

[0038] S3. Input clinical information into the clinical data feature extraction module to obtain clinical feature data.

[0039] In one feasible implementation, the clinical data used includes information such as the patient's age, gender, and lesion distribution corresponding to each image.

[0040] The patient's clinical information is input into the Clinical Net module for feature extraction. After being converted into dense vectors through entity embedding, the clinical data is first processed by entity embedding and normalization to obtain a standardized vector representation.

[0041] In this embodiment of the invention, features of clinical data are extracted using a heterogeneous information extraction method during feature extraction. After cleaning and normalization, the clinical data is transformed into dense vectors via entity embedding, serving as the basic input for subsequent deep feature fusion. The extracted clinical data features are used to control the importance of image feature channels. By leveraging patient clinical data to provide more information for the imbalanced minority class, computational complexity is reduced, thereby improving the model's accuracy and generalization performance.

[0042] S4. Input the melanoma image and the core perception image into the dual-path feature extraction unit to obtain image feature information. Input the image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of the melanoma image.

[0043] Optionally, the dual-path feature extraction unit includes two parallel and weight-shared EMA-EfficientNet networks. Each EMA-EfficientNet network contains multiple EMA-MBConv modules, which integrate the inverse residual structure MBConv with the efficient multi-scale attention mechanism EMA.

[0044] The processing method of the dual-path feature extraction unit is as follows: The original melanoma image is input into the first EMA-EfficientNet network to extract global visual features.

[0045] The core perception image is input into the second EMA-EfficientNet network to extract local detail features.

[0046] Global visual features and local detail features are fused to obtain image feature information.

[0047] In one feasible implementation, the original image and the cropped high-sensory image are input into an improved EMA-EfficientNet dual-branch network. This improved EMA-EfficientNet dual-branch network consists of two structurally identical EMA-EfficientNet networks sharing weight parameters. Its core feature extraction relies on the EMA-EfficientNet network model, mining and extracting effective feature information from the original input image and the core high-sensory feature regions, respectively. Each EMA-EfficientNet network is a lightweight and efficient convolutional neural network. The input image enters a 3×3 convolutional layer with a stride of 2 for initial feature extraction and downsampling, followed by a series of core EMA-MBConv modules. These modules integrate the inverted residual structure (MBConv, Mobile Inverted Bottleneck Convolution) with efficient multi-scale attention (EMA). Through alternating feature transformation, dimensionality reduction, and channel attention weighting using different convolutional kernels and strides, multi-level features are gradually extracted. The network finally performs feature fusion through a 1×1 convolutional layer. The overall design enhances the model's representational ability while maintaining a low parameter count. Figure 3 The specific structure of the EMA-EfficientNet network is explained below: The network consists of a 3×3 convolutional input layer, 7 main EMA-MBConv modules (Block 1~Block 7), and 1×1 channel fusion convolutions. Each of the 7 modules contains multiple sub-layers, as shown in the following structure: (1) Block1: EMA-MBConv1, k3×3, stride1×1 Module type: EMA-MBConv1 (expansion ratio = 1, i.e., no dimensionality increase) Convolution kernel: 3×3 depthwise separable convolution Step size: 1×1 (preserving feature map size) Internal structure: Batch Normalization Swish activation function Depthwise Conv Batch Norm Swish activation function EMA Attention Module Conv 1×1 (dimensionality reduction) Batch Norm Residual connection Output dimensions: 112×112×16 (2) Block2: EMA-MBConv6, k3×3, stride2×2 Module type: EMA-MBConv6 (expansion ratio = 6, first increase dimension then decrease dimension) Convolution kernel: 3×3 depthwise separable convolution Step size: 2×2 (downsampling, size halved) Output dimensions: 56×56×24 (3) Block3: EMA-MBConv6, k5×5, stride2×2 Module type: EMA-MBConv6 Convolution kernel: 5×5 depthwise separable convolution (larger receptive field) Step size: 2×2 (downsampling) Output dimensions: 28×28×40 (4) Block4: EMA-MBConv6, k3×3, stride2×2 Module type: EMA-MBConv6 Convolution kernel: 3×3 depthwise separable convolution Step size: 2×2 (downsampling) Output dimensions: 14×14×80 (5) Block5: EMA-MBConv6, k5×5, stride1×1 Module type: EMA-MBConv6 Convolution kernel: 5×5 depthwise separable convolution Step size: 1×1 (maintain size) Output dimensions: 14×14×112 (6) Block6: EMA-MBConv6, k5×5, stride2×2 Module type: EMA-MBConv6 Convolution kernel: 5×5 depthwise separable convolution Step size: 2×2 (downsampling) Output dimensions: 7×7×192 (7) Block7: EMA-MBConv6, k3×3, stride1×1 Module type: EMA-MBConv6 Convolution kernel: 3×3 depthwise separable convolution Step size: 1×1 (maintain size) Output dimensions: 7×7×320 To address the challenge of limited learning, this invention proposes an improved EMA-EfficientNet network based on transfer learning strategies. Since Efficient Multi-Scale Attention (EMA) offers advantages such as cross-dimensional interaction, enhanced context capture, and efficient computation, the EMA module is combined with the MBConv module of EfficientNet to form the EMA-MBConv module. This allows for the extraction of finer-grained features, enabling the model to more fully utilize key information and thus improve classification performance for melanoma images with similar morphology and limited sample size. This invention constructs a weight-sharing dual-input network structure. This network employs a dual-path design, with one input being the original image and the other being the core receptive region image, extracting visual features separately. By fusing features from the original image and the core receptive region, the model can combine global context and local details to obtain a richer representation of melanoma features. This fusion mechanism not only enhances the model's perception of key regions but also improves feature discriminability. This allows the model to achieve high convergence accuracy in fewer training epochs and exhibits better generalization performance.

[0048] Optionally, S4 inputs image feature information and clinical feature data into a gating fusion unit to obtain the recognition result of the melanoma image, including: Clinical feature data are sequentially transformed through a fully connected layer and a GELU activation function, and then channel weights are generated through a convolutional layer and a Sigmoid activation function.

[0049] By multiplying image feature information with channel weights, the features of different channels can be enhanced or suppressed.

[0050] By using gating coefficients, the image feature information after spatial transformation by the tanh activation function is weighted and summed with the clinical feature data to generate a final feature map that integrates clinical data information. This map is then input into the classification layer to obtain the recognition result of the melanoma image.

[0051] In one feasible implementation, a common data fusion method directly connects the feature vectors of clinical data with the visual feature vectors extracted from the last layer of a convolutional neural network, followed by one or more fully connected layers. This connection does not directly consider the potential impact of clinical data on the visual feature extraction process. This invention proposes a data fusion method based on gating units, enabling direct interaction between clinical data and visual features. Figure 4As shown, the encoded clinical data vector is sequentially passed through a Dense layer and a GELU activation function for feature transformation, and then through a convolutional layer and a Sigmoid activation function to generate channel weights. The output visual features and these channel weights are input into a gating unit to fuse the features of the two modalities. A linear transformation layer and a Sigmoid activation function are used to generate gating coefficients, which represent the model's trust weights for visual and clinical information. The tanh activation function is used to spatially transform the two original modal features to enhance their discriminative power. The transformed modal features are then weighted and summed according to the gating coefficients to obtain a feature map that fuses clinical data information. This embodiment of the invention proposes a data fusion method based on a gating unit, enabling direct interaction between clinical data and visual features. It adaptively adjusts the importance of each channel feature in the last convolutional layer using clinical data, allowing the network to focus on key channel features based on supplementary information from the clinical data.

[0052] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) This invention addresses the misclassification problem in existing methods for small-sample melanoma images caused by high intra-class variability and low inter-class variability: The cost-sensitive learning function based on sample weighting in this embodiment can solve this problem. Based on the sample weighting strategy, different weights are assigned to samples of different categories. By reducing the weight of easily classified samples, the learning ability for samples with weaker representativeness is improved. By utilizing the weight adjustment mechanism and integrating the advantages of multiple functions, the model can learn more discriminative feature representations. Compared with the traditional cross-entropy loss, the proposed cost-sensitive learning function is more suitable for melanoma image diagnosis.

[0053] (2) The core perception feature extraction module and the heterogeneous data fusion module can more accurately and quickly focus on the high-perception regions that the model focuses on in the image: the model learns the importance weight of each region based on the feature information of the image and clinical data, and automatically adjusts the attention intensity to different regions during the feature extraction process according to the unique features of the sample itself, and mines the model's own potential attention to extract core perception features. This helps to reduce computational complexity, while extracting more features, and extracting more discriminative features through fusion, thereby improving accuracy and generalization performance.

[0054] Figure 5 This is a block diagram of a melanoma image recognition device for imbalanced samples provided in an embodiment of the present invention. This device is used in a melanoma image recognition method for imbalanced samples. (Refer to...) Figure 5 The device includes an acquisition unit 510, a core perception image extraction unit 520, a clinical data feature extraction unit 530, and a feature fusion and recognition unit 540. Among them: The acquisition unit 510 is used to acquire the image of the melanoma to be identified, clinical information, and the trained melanoma recognition model. The trained melanoma recognition model includes a core perception region extraction module, a dual-path feature fusion module, and a clinical data feature extraction module. The core perception region extraction module includes convolutional units, weighted aggregation units, class activation mapping, and segmentation extraction units. The dual-path feature fusion module includes dual-path feature extraction units and gated fusion units. The core perception image extraction unit 520 is used to input the melanoma image to be identified into the core perception region extraction module, obtain a preliminary feature map according to the convolution unit, obtain the weight coefficient of each preliminary feature map according to the preliminary feature map and the weighted aggregation unit, obtain a heat map according to the preliminary feature map, the weight coefficient and the class activation mapping, and obtain the core perception image according to the heat map and the segmentation extraction unit. The clinical data feature extraction unit 530 is used to input clinical information into the clinical data feature extraction module to obtain clinical feature data; The feature fusion and recognition unit 540 is used to input the melanoma image and the core perception image into the dual-path feature extraction unit to obtain image feature information, and input the image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of the melanoma image.

[0055] Figure 6 This is a schematic diagram of the structure of a melanoma image recognition device for imbalanced samples provided in an embodiment of the present invention, as shown below. Figure 6 As shown, a melanoma image recognition device for imbalanced samples may include the above-mentioned Figure 5 The illustrated melanoma image recognition device for imbalanced samples. Optionally, the melanoma image recognition device 610 for imbalanced samples may include a first processor 2001.

[0056] Optionally, the melanoma image recognition device 610 for imbalanced samples may also include a memory 2002 and a transceiver 2003.

[0057] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0058] The following is combined with Figure 6 The following is a detailed description of the various components of the melanoma image recognition device 610 for imbalanced samples: The first processor 2001 is the control center of the melanoma image recognition device 610 for imbalanced samples. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0059] Optionally, the first processor 2001 can perform various functions of the melanoma image recognition device 610 for imbalanced samples by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0060] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 are shown in the diagram.

[0061] In a specific implementation, as one example, the melanoma image recognition device 610 for imbalanced samples may also include multiple processors, for example... Figure 6 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0062] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0063] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the melanoma image recognition device 610 for imbalanced samples (…). Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0064] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0065] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 6 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0066] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the melanoma image recognition device 610 for imbalanced samples. Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0067] It should be noted that, Figure 6 The structure of the imbalanced sample-oriented melanoma image recognition device 610 shown does not constitute a limitation on this router. Actual imbalanced sample-oriented melanoma image recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0068] Furthermore, the technical effect of the melanoma image recognition device 610 for imbalanced samples can be referred to the technical effect of the melanoma image recognition method for imbalanced samples described in the above method embodiments, and will not be repeated here.

[0069] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0070] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0071] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0072] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0073] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0074] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0075] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0077] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for melanoma image recognition based on imbalanced samples, characterized in that, The method includes: S1. Obtain images of melanoma to be identified, clinical information, and a trained melanoma identification model; The trained melanoma recognition model includes a core perception region extraction module, a dual-path feature fusion module, and a clinical data feature extraction module. The core perception region extraction module includes convolutional units, weighted aggregation units, class activation mapping, and segmentation extraction units. The dual-path feature fusion module includes dual-path feature extraction units and gated fusion units. S2. Input the melanoma image to be identified into the core perception region extraction module. Obtain a preliminary feature map based on the convolution unit. Obtain the weight coefficient of each preliminary feature map based on the preliminary feature map and the weighted aggregation unit. Obtain a heat map based on the preliminary feature map, the weight coefficient and the class activation mapping. Obtain the core perception image based on the heat map and the segmentation extraction unit. S3. Input clinical information into the clinical data feature extraction module to obtain clinical feature data; S4. Input the melanoma image and the core perception image into the dual-path feature extraction unit to obtain image feature information. Input the image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of the melanoma image.

2. The melanoma image recognition method for imbalanced samples according to claim 1, characterized in that, The weight coefficients for each preliminary feature map obtained by S2 based on the preliminary feature map and the weighted aggregation unit include: Calculate the output score for class a. Regarding the kth preliminary feature map at position ( i , j Characteristic response value at ) First-order partial derivatives Second-order partial derivatives and third-order partial derivatives ; Calculate the position weight coefficients based on the second-order partial derivatives, third-order partial derivatives, and characteristic response values: （1） Where (i,j) is used to mark the position of the differentiation point, and (y,z) is used to mark the summation point during traversal, i.e. It is the feature response value calculated for position (i,j). It is the characteristic response value of traversing the summation points. It is an aggregation of global information across the entire activation graph; The calculation method is as follows: （2） The positional weight coefficients are weighted and summed with the first-order partial derivatives processed by the ReLU activation function to obtain the comprehensive weight of the k-th preliminary feature map for category a: （3）。 3. The melanoma image recognition method for imbalanced samples according to claim 1, characterized in that, The process of S2, which obtains a heatmap based on the preliminary feature map, weight coefficients, and class activation mapping, includes: The core perceptual features are obtained by weighted summation of all preliminary feature maps, and the calculation formula is as follows (4): （4） in, The response value of the core perceptual feature map of category a at position (i,j) is used to characterize the importance of position (i,j) for the target category discrimination. ReLU activation is applied to the core perceptual features to retain regions that positively contribute to the prediction results. After normalization and upsampling, the final heatmap is generated. In the heatmap, the brighter the region, the higher the model’s attention to that region.

4. The melanoma image recognition method for imbalanced samples according to claim 1, characterized in that, The core perception image obtained by S2 based on the heatmap and segmentation extraction unit includes: The heat map is binarized to generate a binary mask matrix with the same size as the heat map. In the mask matrix, the region with a value of 1 corresponds to the region in the heat map that is higher than a preset threshold, and the region with a value of 0 corresponds to the region that is lower than the preset threshold. Locate the maximum activation region in the binary mask matrix, and calculate the center coordinates of the maximum activation region according to the following formula (5): （5） Where U represents the column coordinate of the top-left corner of the ROI rectangle, V represents the row coordinate of the top-left corner of the ROI rectangle, L represents the width of the ROI rectangle, and H represents the height of the ROI rectangle. The final set of pixel coordinates of the cropped region is represented by the following formula (6): （6） in, This represents the pixel coordinates of the final cropped area, where m represents the x-coordinate of the center point of the maximum activation region, and n represents the y-coordinate of the center point of the maximum activation region. Indicates altitude, Indicates the width.

5. The melanoma image recognition method for imbalanced samples according to claim 1, characterized in that, The dual-path feature extraction unit includes two parallel and weight-shared EMA-EfficientNet networks. Each EMA-EfficientNet network contains multiple EMA-MBConv modules. The EMA-MBConv modules integrate the inverse residual structure MBConv with the efficient multi-scale attention mechanism EMA. The processing method of the dual-path feature extraction unit is as follows: The original melanoma image is input into the first EMA-EfficientNet network to extract global visual features; The core perception image is input into the second EMA-EfficientNet network to extract local detail features; The global visual features and local detail features are fused to obtain image feature information.

6. The melanoma image recognition method for imbalanced samples according to claim 1, characterized in that, The S4 inputs image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of the melanoma image, including: Clinical feature data are sequentially transformed through a fully connected layer and a GELU activation function, and then channel weights are generated through a convolutional layer and a Sigmoid activation function. The image feature information is multiplied by the channel weights to enhance or suppress features of different channels. By using gating coefficients, the image feature information after spatial transformation by the tanh activation function is weighted and summed with the clinical feature data to generate a final feature map that integrates clinical data information. This map is then input into the classification layer to obtain the recognition result of the melanoma image.

7. The melanoma image recognition method for imbalanced samples according to claim 1, characterized in that, The trained melanoma recognition model uses the cost-sensitive learning-based ACLoss loss function for parameter optimization during training. The ACLoss loss function integrates the face recognition loss function ArcFace Loss and the center loss, and the calculation formula is as follows (7): （7） in, It is a hyperparameter that balances the loss of face recognition and the loss of centering. The ArcFaceLoss function for face recognition is expressed in the following formula (8): （8） in, N n' represents the total number of samples in the current training batch, and n' represents the total number of categories in the classification task. This represents the feature vector of the i-th input sample and the true class of input sample i. The angle between the corresponding weight center vectors, m' represents the additive angular margin, which directly adds a penalty term in the rotation angle space to compress intra-class distance and expand inter-class distance, and s represents the feature scaling factor. This represents the angle between the input sample feature and the center vector of the j-th non-true class; The center loss is expressed by the following formula (9): （9） in, Indicates input sample Features Indicates input sample The true category, It is a category The center vector is the average value of the samples in the corresponding category.

8. A melanoma image recognition device for imbalanced samples, wherein the melanoma image recognition device for imbalanced samples is used to implement the melanoma image recognition method for imbalanced samples as described in any one of claims 1-7, characterized in that, The device includes: The acquisition unit is used to acquire images of melanoma to be identified, clinical information, and a trained melanoma recognition model. The trained melanoma recognition model includes a core perception region extraction module, a dual-path feature fusion module, and a clinical data feature extraction module. The core perception region extraction module includes convolutional units, weighted aggregation units, class activation mapping, and segmentation extraction units. The dual-path feature fusion module includes dual-path feature extraction units and gated fusion units. The core perception image extraction unit is used to input the melanoma image to be identified into the core perception region extraction module, obtain a preliminary feature map based on the convolution unit, obtain the weight coefficient of each preliminary feature map based on the preliminary feature map and the weighted aggregation unit, obtain a heat map based on the preliminary feature map, the weight coefficient and the class activation mapping, and obtain the core perception image based on the heat map and the segmentation extraction unit. The clinical data feature extraction unit is used to input clinical information into the clinical data feature extraction module to obtain clinical feature data. The feature fusion and recognition unit is used to input melanoma images and core perception images into the dual-path feature extraction unit to obtain image feature information, and input the image feature information and clinical feature data into the gating fusion unit to obtain the recognition result of melanoma images.

9. A melanoma image recognition device for imbalanced samples, characterized in that, The melanoma image recognition device for imbalanced samples includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images