Model training method and device, electronic equipment and storage medium

Abstract

The application provides a model training method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The method comprises the following steps: obtaining a first enhanced view and a second enhanced view; obtaining a first spatial feature map and a first patch label sequence corresponding to the first enhanced view, and a second spatial feature map and a second patch label sequence corresponding to the second enhanced view; obtaining a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; generating a first global-local energy contrast activation map based on the first frequency domain enhanced feature map, and generating a second global-local energy contrast activation map based on the second frequency domain enhanced feature map; obtaining an effective anchor point matching pair set; constructing an anchor point consistency loss based on the effective anchor point matching pair set, and updating parameters of the backbone feature extraction network and filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor point consistency loss to obtain a trained backbone feature extraction network.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] The reliability of intelligent analysis of medical images largely depends on the stability of feature representations. In practical applications, the annotation cost of medical images is high, and there are many privacy and ethical restrictions. Diseases and tissue states are difficult to fully cover in a single training phase, often requiring models to learn under conditions where a small number of labeled images coexist with a large number of unlabeled images. Compared to natural images, meaningful evidence in medical images is often more localized and subtle, such as slight changes in mucosal texture, minor differences in organ boundaries, localized variations in tissue density, and small structural anomalies in pathological sections.

[0003] Most common medical image models extract features in the spatial domain through convolution, visual Transformers, or attention mechanisms. These methods typically perform well when known category labels are sufficiently defined, but when encountering unknown categories or new categories with similar shapes, the model tends to rely on recurring texture cues from the labeled categories for judgment. In this situation, truly discriminative local lesion areas are often obscured by common interferences such as background tissue texture, organ surface reflections, image borders, instrument occlusion, or imaging artifacts.

[0004] To enhance structural and texture information, existing solutions have attempted to incorporate frequency domain processing techniques such as high-pass filtering, low-pass filtering, discrete cosine transform, wavelet transform, Fourier transform, or frequency domain attention into the medical image processing workflow. However, fixed high-pass filtering amplifies both boundary details and reflections and noise simultaneously; fixed low-pass filtering tends to smooth out fine-grained lesion boundaries; even after converting the image or features to the frequency domain and then fusing them with spatial domain features, the model still struggles to reliably select local regions that are truly semantically relevant to the lesion.

[0005] It is evident that in the current process of medical image feature extraction, local lesion-related features are easily affected by reflection, boundary artifacts, repetitive anatomical textures, and fixed frequency band filtering, resulting in insufficient stability of feature representation and difficulty in providing reliable local semantic support for subsequent tasks such as classification, segmentation, lesion localization, and discovery of unknown categories. Summary of the Invention

[0006] This invention provides a model training method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies in medical image feature extraction, where local lesion-related features are easily interfered with by reflection, boundary artifacts, repetitive anatomical textures, and fixed frequency band filtering, resulting in insufficient feature representation stability. This invention achieves stable and reliable feature representation of lesion-related local areas.

[0007] This invention provides a model training method, comprising the following steps: Acquire medical image samples, and generate a first enhanced view and a second enhanced view based on the medical image samples; The first enhanced view and the second enhanced view are respectively input into the backbone feature extraction network to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The first spatial feature map and the second spatial feature map are subjected to frequency domain transformation and channel-wise learnable frequency domain filtering respectively to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; A first global-local energy contrast activation map is generated based on the first frequency domain enhanced feature map, and a second global-local energy contrast activation map is generated based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; Based on the first global-local energy contrast activation map, a first set of semantic anchor points is selected from the first patch marker sequence; and based on the second global-local energy contrast activation map, a second set of semantic anchor points is selected from the second patch marker sequence. Perform nearest neighbor matching on the first set of semantic anchors and the second set of semantic anchors to obtain a set of valid anchor matching pairs; Anchor consistency loss is constructed based on the set of effective anchor matching pairs, and the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter are updated based on the anchor consistency loss to obtain the trained backbone feature extraction network.

[0008] According to a model training method provided by the present invention, the method further includes: During the inference phase of the trained backbone feature extraction network, a medical image to be processed is acquired; the medical image to be processed is input into the trained backbone feature extraction network to obtain image features corresponding to the medical image to be processed; based on the image features, feature extraction results are output or prediction results are output based on the task head; wherein, the processing of the medical image to be processed does not include the frequency domain transformation processing, the channel-by-channel learnable frequency domain filtering processing, the generation processing of the first global-local energy contrast activation map and the second global-local energy contrast activation map, the selection processing of the first semantic anchor set and the second semantic anchor set, and the nearest neighbor matching processing.

[0009] According to a model training method provided by the present invention, the step of performing frequency domain transformation on the first spatial feature map and the second spatial feature map respectively and performing channel-by-channel learnable frequency domain filtering to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map includes: Two-dimensional Fourier transform and spectral centering are performed on the first spatial feature map and the second spatial feature map respectively to obtain the first complex frequency domain feature and the second complex frequency domain feature respectively; For any complex frequency domain feature among the first complex frequency domain feature and the second complex frequency domain feature, a first channel-wise convolutional layer is used to filter the real part feature of the any complex frequency domain feature to obtain the filtered real part feature corresponding to the any complex frequency domain feature; and a second channel-wise convolutional layer is used to filter the imaginary part feature of the any complex frequency domain feature to obtain the filtered imaginary part feature corresponding to the any complex frequency domain feature. The real part feature of the filter and the imaginary part feature of the filter are combined to form the complex frequency domain feature of the filter corresponding to any complex frequency domain feature. The complex frequency domain feature of the filter is then subjected to inverse spectrum centering and two-dimensional inverse Fourier transform in sequence, and the real part is taken to obtain the frequency domain enhancement feature map corresponding to any complex frequency domain feature. The frequency domain enhancement feature map corresponding to the first complex frequency domain feature is used as the first frequency domain enhancement feature map, and the frequency domain enhancement feature map corresponding to the second complex frequency domain feature is used as the second frequency domain enhancement feature map; Wherein, the first channel-wise convolutional layer and the second channel-wise convolutional layer are both depthwise convolutional layers, and the number of groups in the depthwise convolutional layer is equal to the number of channels of the input feature; the convolutional kernel parameters of the first channel-wise convolutional layer and the second channel-wise convolutional layer are independent of each other.

[0010] According to a model training method provided by the present invention, in the initial stage of training, the convolution kernel parameters of the first channel-wise convolutional layer and the second channel-wise convolutional layer are initialized to an approximate identity mapping; wherein, the initialization method of the approximate identity mapping includes: initializing the weight at the center position of the convolution kernel to one, initializing the weights at other positions of the convolution kernel except the center position to zero, and initializing the bias to zero.

[0011] According to a model training method provided by the present invention, the step of generating a first global-local energy contrast activation map based on a first frequency domain enhanced feature map and generating a second global-local energy contrast activation map based on a second frequency domain enhanced feature map includes: For any one of the frequency domain enhancement feature maps in the first and second frequency domain enhancement feature maps, channel aggregation is performed on the any one frequency domain enhancement feature map to obtain a single-channel energy map corresponding to the any one frequency domain enhancement feature map; for each spatial position in the single-channel energy map, a preset local neighborhood centered on the spatial position is determined, and based on the non-negative difference between the mean square energy value and the square of the mean energy value within the preset local neighborhood, the local energy variance corresponding to the spatial position is obtained; a local energy variance map is composed of the local energy variances corresponding to all spatial positions in the single-channel energy map. Based on the local energy variance map, a global-local energy contrast activation map corresponding to any frequency domain enhanced feature map is generated. The global-local energy contrast activation map corresponding to the first frequency domain enhanced feature map is used as the first global-local energy contrast activation map, and the global-local energy contrast activation map corresponding to the second frequency domain enhanced feature map is used as the second global-local energy contrast activation map.

[0012] According to a model training method provided by the present invention, generating a global-local energy contrastive activation map corresponding to any frequency domain enhanced feature map based on the local energy variance map includes: Calculate the global mean and global dispersion statistic of the local energy variance map across the entire space; based on the local energy variance, the global mean, and the global dispersion statistic, obtain an energy contrast value; wherein the energy contrast value is directly proportional to the difference between the local energy variance and the global mean, inversely proportional to the sum of the global dispersion statistic and a preset constant, and the activation intensity is adjusted by a learnable scaling parameter; perform a nonlinear mapping on the energy contrast value to obtain a global-local energy contrast activation map corresponding to any frequency domain enhanced feature map.

[0013] According to a model training method provided by the present invention, after generating the first global-local energy contrast activation map and the second global-local energy contrast activation map, the method further includes: For any spatial feature map in the first spatial feature map and the second spatial feature map, the global-local energy contrast activation map corresponding to the first spatial feature map is multiplied positionally with the first spatial feature map, and then added positionally with the first spatial feature map to obtain the residual enhancement feature map corresponding to the first spatial feature map; wherein, the global-local energy contrast activation map corresponding to the first spatial feature map is the first global-local energy contrast activation map, and the global-local energy contrast activation map corresponding to the second spatial feature map is the second global-local energy contrast activation map; The residual enhancement feature map corresponding to the first spatial feature map is used as the first residual enhancement feature map, and the residual enhancement feature map corresponding to the second spatial feature map is used as the second residual enhancement feature map. The residual enhanced feature map corresponding to the first spatial feature map and the residual enhanced feature map corresponding to the second spatial feature map are used for obtaining patch features corresponding to subsequent semantic anchors.

[0014] According to a model training method provided by the present invention, the step of selecting a first set of semantic anchor points from a first patch label sequence based on a first global-local energy contrast activation map, and selecting a second set of semantic anchor points from a second patch label sequence based on a second global-local energy contrast activation map, includes: For any one of the first and second global-local energy contrast activation maps, the global-local energy contrast activation map is divided into multiple patch regions according to the spatial grid corresponding to the patch marker sequence. For each patch region, the activation values ​​within the patch region are aggregated to obtain the patch response score corresponding to the patch region. All the patch response scores are normalized to obtain the patch response distribution corresponding to the global-local energy contrast activation map. Based on the patch response distribution, a preset number of patch markers with the highest response values ​​are selected from the corresponding patch marker sequence as the semantic anchor set corresponding to the global-local energy contrast activation map. The set of semantic anchor points corresponding to the first global-local energy contrast activation map is used as the first semantic anchor point set, and the set of semantic anchor points corresponding to the second global-local energy contrast activation map is used as the second semantic anchor point set.

[0015] According to a model training method provided by the present invention, the step of performing nearest neighbor matching on the first semantic anchor set and the second semantic anchor set to obtain a set of effective anchor matching pairs includes: A similarity matrix is ​​obtained based on the similarity between each first semantic anchor in the first semantic anchor set and each second semantic anchor in the second semantic anchor set. For any first semantic anchor in the first semantic anchor set, the nearest neighbor semantic anchor in the second semantic anchor set is determined based on the similarity matrix. When the nearest neighbor of the nearest neighbor semantic anchor in the first semantic anchor set is the first semantic anchor, and the similarity between the first semantic anchor and the nearest neighbor semantic anchor is not lower than a preset similarity threshold, the first semantic anchor and the nearest neighbor semantic anchor are determined as a valid anchor matching pair. All determined valid anchor matching pairs constitute the set of valid anchor matching pairs.

[0016] According to a model training method provided by the present invention, the step of constructing an anchor consistency loss based on the set of effective anchor matching pairs includes: For each valid anchor pair in the set of valid anchor pair matches, a matching weight is determined based on the patch response scores corresponding to the two semantic anchors in the valid anchor pair. Based on the matching weight and the similarity between the two semantic anchors, a pairwise consistency loss is calculated for the valid anchor pair. All the pairwise consistency losses corresponding to the medical image sample are weighted and summed to obtain the anchor consistency loss corresponding to the medical image sample. The pairwise consistency loss is obtained based on the difference between the first and second semantic anchors and the cosine similarity between the two semantic anchors, or based on the squared distance between the two semantic anchors.

[0017] According to a model training method provided by the present invention, the method further includes: When the number of valid anchor matching pairs in the set of valid anchor matching pairs corresponding to the medical image sample is lower than the preset minimum number of matches, the medical image sample is not used in the loss calculation of the current training batch.

[0018] According to a model training method provided by the present invention, the step of updating the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor consistency loss to obtain the trained backbone feature extraction network includes: The anchor consistency loss and the basic training loss are combined by weight to obtain the total training loss. Based on the total training loss, the parameters of the backbone feature extraction network and the filtering parameters corresponding to the channel-by-channel learnable frequency domain filtering are updated to obtain the trained backbone feature extraction network. The basic training loss includes at least one of representation learning loss, classification loss, clustering loss, pseudo-label loss, and entropy regularization loss. The weights corresponding to the anchor consistency loss are gradually increased in a linear warm-up manner at the beginning of training and are maintained at the preset maximum weights after reaching the preset number of warm-up rounds.

[0019] According to a model training method provided by the present invention, the medical image samples include at least one of endoscopic images, CT images, MRI images, ultrasound images, fundus images, and pathological images; the output features of the trained backbone feature extraction network are used for at least one of the following tasks: classification, retrieval, clustering, segmentation, lesion localization, unknown category discovery, and auxiliary analysis.

[0020] The present invention also provides a model training device, comprising the following modules: An acquisition module is used to acquire medical image samples and generate a first enhanced view and a second enhanced view based on the medical image samples; The input module is used to input the first enhanced view and the second enhanced view into the backbone feature extraction network respectively to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The transformation module is used to perform frequency domain transformation on the first spatial feature map and the second spatial feature map respectively and perform channel-wise learnable frequency domain filtering to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; The generation module is used to generate a first global-local energy contrast activation map based on the first frequency domain enhanced feature map, and to generate a second global-local energy contrast activation map based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; The selection module is used to select a first set of semantic anchors from the first patch marker sequence based on the first global-local energy contrast activation map; and to select a second set of semantic anchors from the second patch marker sequence based on the second global-local energy contrast activation map, and to perform mutual nearest neighbor matching on the first set of semantic anchors and the second set of semantic anchors to obtain a set of effective anchor matching pairs; The construction module is used to construct an anchor consistency loss based on the set of effective anchor matching pairs, and update the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor consistency loss, so as to obtain the trained backbone feature extraction network.

[0021] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method as described above.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the model training method as described above.

[0023] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model training method as described above.

[0024] This invention provides a model training method, apparatus, electronic device, and storage medium. By performing frequency domain transformation on two enhanced views of the same medical image sample and applying independent filtering parameters to the real and imaginary parts of the frequency domain features, a channel-by-channel learnable filtering method is executed. This allows the backbone feature extraction network to adaptively learn frequency responses semantically related to lesions across the entire spectrum, avoiding the biased loss of boundary details or low-frequency structures caused by fixed-band filtering. Simultaneously, a global-local energy contrast activation map is generated based on the deviation of local energy from global energy statistics, reducing reflections, boundary artifacts, and repetitive anatomical textures. Once global common interference is effectively suppressed, the local regions related to potential lesions are relatively highlighted. Then, based on the global-local energy contrast activation map, semantic anchors are selected from the patch label sequence, and unreliable correspondences caused by disturbances, misalignments, or background similarity are screened out between the semantic anchor sets of the two enhanced views through mutual nearest neighbor matching. Consistency constraints are applied only to stable and effective anchor matching pairs with local semantic orientation, thereby promoting the backbone feature extraction network to form view-invariant and stable feature representations of lesion-related local regions, significantly improving the reliability of the medical image feature extraction model at the level of local lesions. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 A schematic flowchart of the model training method provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the relationship between the frequency domain semantic anchor consistency branch in the training phase and the backbone feature extraction path in the inference phase, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the full-spectrum adaptive frequency domain filtering module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of filtering the real and imaginary parts of complex frequency domain features channel by channel, as provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the global-local energy contrast activation map generation process provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the semantic anchor point selection and mutual nearest neighbor matching process provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the model training device provided by the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The model training method provided in this embodiment of the invention can be executed by an electronic device with data processing capabilities, including but not limited to training servers, graphics workstations, cloud computing nodes, embedded high-performance computing units, medical image analysis workstations, and processing devices deployed in hospital information systems or image archiving and communication systems.

[0029] In the embodiments described below, a training device is used as the execution subject for illustrative purposes. It should be understood that the exemplary selection of the execution subject does not constitute a limitation on the scope of protection of this invention.

[0030] Figure 1 This is a flowchart illustrating the model training method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the training method for the medical image feature extraction model provided in this embodiment of the invention includes the following steps.

[0031] Step S101: Obtain medical image samples and generate a first enhanced view and a second enhanced view based on the medical image samples.

[0032] The medical image samples in this embodiment of the invention refer to medical image data units used to train the backbone feature extraction network. They can be two-dimensional medical images, two-dimensional slices in three-dimensional volume data, or the three-dimensional volume data itself.

[0033] The acquisition of medical image samples can include various methods such as directly reading from a medical image database, obtaining from the interface of a hospital information system or image archiving and communication system, acquiring and transmitting in real time through medical image acquisition equipment, and batch extraction from labeled or unlabeled medical image datasets. This invention does not limit the acquisition method.

[0034] For each medical image sample, the training device performs two independent random perturbation processes on the medical image sample to obtain a first enhanced view and a second enhanced view. Both the first and second enhanced views correspond to the same medical image sample, but there are controllable differences between them at the pixel level, enabling the backbone feature extraction network to learn stable representations of local regions related to the same potential lesion under different image perturbation conditions.

[0035] The random perturbation processing used may include, but is not limited to, geometric transformation perturbations and pixel-level perturbations. Geometric transformation perturbations include random cropping, horizontal flipping, vertical flipping, rotation, and scale changes, while pixel-level perturbations include brightness changes, contrast adjustments, color channel perturbations, and gamma correction.

[0036] In one exemplary embodiment, the training device reads a medical image sample from a colonoscopy image dataset. x The medical image sample is then input into the enhancement module. The enhancement module then... x Two different random enhancement combinations were performed. The first random enhancement combination included random cropping and horizontal flipping to obtain the first enhanced view. The second random enhancement combination includes random cropping, brightness perturbation, and color channel perturbation to obtain the second enhanced view. .

[0037] Step S102: Input the first enhanced view and the second enhanced view into the backbone feature extraction network respectively to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view.

[0038] The backbone feature extraction network in this embodiment of the invention refers to a neural network structure that can map an input image into a high-dimensional feature representation carrying local location information. The backbone feature extraction network can employ a convolutional neural network structure, such as the ResNet series, DenseNet series, and EfficientNet series; it can also employ a visual Transformer structure, such as ViT, Swin Transformer, and PVT; or it can employ a hybrid structure of convolutional and visual Transformers.

[0039] Both the first spatial feature map and the second spatial feature map are multi-channel feature tensors output by the backbone feature extraction network in the spatial dimension, denoted as... ,in v ∈{1,2} is used to distinguish the features corresponding to two enhanced views. B Indicates batch size. C The number of channels in the spatial feature map. H and W These represent the height and width of the spatial feature map, respectively.

[0040] Patch marker sequence refers to a set of local features that are spatially aligned with the spatial feature map, denoted as ,in N Indicates the number of patches. D This represents the feature dimension of a single patch tag.

[0041] There are two ways to obtain patch marker sequences: one is to expand the convolutional feature map output by the backbone feature extraction network into a spatial grid when the backbone feature extraction network is a convolutional neural network or a similar structure, and the channel vector corresponding to each spatial position is used as a patch marker; the other is to directly use the markers corresponding to the image patches output by the backbone feature extraction network as patch markers when the backbone feature extraction network is a visual Transformer structure.

[0042] In one exemplary embodiment, the backbone feature extraction network employs a ResNet-18 network pre-trained on ImageNet, with fine-tuning performed on the final stage of the ResNet-18 network. When the size of the input medical image sample is 224×224, the spatial size of the convolutional feature map output by the final stage of the ResNet-18 network is 7×7, and the number of channels is 512. H = W =7、 C =512. The training device unfolds the convolutional feature map into a 7×7 spatial grid to obtain the number of patches. N =49, Single patch marking dimension D=512 patch marker sequences. It should be noted that the embodiments of the present invention do not limit the specific type of backbone feature extraction network, input image size, number of patches, and patch marker dimension.

[0043] Step S103: Perform frequency domain transformation on the first spatial feature map and the second spatial feature map respectively, and perform channel-wise learnable frequency domain filtering to obtain the first frequency domain enhanced feature map and the second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters.

[0044] The frequency domain transformation in this embodiment of the invention refers to the reversible transformation that maps a feature tensor in the spatial dimension to the frequency domain space. The frequency domain transformation can be a two-dimensional discrete Fourier transform, a two-dimensional fast Fourier transform, a two-dimensional discrete cosine transform, a two-dimensional wavelet transform, or other frequency domain analysis transformations. The output of the frequency domain transformation is usually a complex frequency domain feature carrying real and imaginary components.

[0045] Channel-by-channel learnable frequency domain filtering refers to the process of independently applying learnable filter weights to each feature channel in the frequency domain. A key limitation of this invention is that the real and imaginary parts of the complex frequency domain features do not share filter parameters; instead, they are processed separately using two independent sets of learnable filter parameters. This approach breaks the limitation of traditional frequency domain processing that bundles the real and imaginary parts for filtering. It allows different feature channels to independently learn and adapt to the frequency response of medical image samples in the frequency domain, and to possess different modulation capabilities in the real and imaginary dimensions, thereby adaptively preserving frequency components semantically related to lesions across the entire spectrum.

[0046] A frequency-domain enhanced feature map refers to a feature tensor with the same dimension as the spatial feature map, obtained by performing an inverse frequency-domain transformation on the filtered complex frequency-domain features and taking the real part. It is still in the spatial domain, but its internal information has been adaptively reweighted at the frequency domain level.

[0047] like Figure 3 As shown, the training device trains the first spatial feature map. The first frequency domain enhanced feature map is obtained by sequentially performing frequency domain transformation, real part filtering and imaginary part filtering, combination of filtered complex frequency domain features, and inverse frequency domain transformation and taking the real part. ; for the second spatial feature map Perform the same processing procedure to obtain the second frequency domain enhanced feature map. Step S104: Generate a first global-local energy contrast activation map based on the first frequency domain enhanced feature map, and generate a second global-local energy contrast activation map based on the second frequency domain enhanced feature map. Both the first and second global-local energy contrast activation maps are generated based on the degree of deviation of local energy from global energy statistics.

[0048] A global-local energy contrast activation map is a two-dimensional response map that reflects the degree to which the energy distribution at each spatial location in the frequency domain enhanced feature map deviates from the overall background statistics. Its core mechanism involves: first, deriving the local energy feature quantity at each spatial location from the frequency domain enhanced feature map; then, using the statistics of this local energy feature quantity across the entire feature map (such as the global mean and global dispersion) as a benchmark, calculating the degree of deviation of the local energy feature quantity at each spatial location relative to the global benchmark; and finally, transforming this deviation degree into an activation response value through a nonlinear mapping.

[0049] like Figure 5 As shown, the core function of the global-local energy contrast activation map is to highlight local regions that have significant deviations in the statistical sense of the whole feature map, thereby suppressing interference signals such as reflections, boundary artifacts, and repetitive anatomical textures that exist in a globally common manner.

[0050] While reflections, boundary artifacts, or repetitive anatomical textures may not necessarily have low energy, they are widely distributed across the entire map, and their local energy characteristics deviate little from the global baseline, thus obtaining a low response on the global-local energy contrast activation map. In contrast, true local lesion areas obtain a higher response because their local energy characteristics deviate significantly from the global baseline.

[0051] The training device is based on the first frequency domain enhanced feature map Second frequency domain enhanced feature map First global-local energy contrast activation maps are generated respectively. Second global-local energy contrast activation map .

[0052] Step S105: Select a first set of semantic anchor points from the first patch marker sequence according to the first global-local energy contrast activation map; and select a second set of semantic anchor points from the second patch marker sequence according to the second global-local energy contrast activation map.

[0053] Semantic anchors are patch markers considered most likely to correspond to local regions related to potential lesions. The first set of semantic anchors and the second set of semantic anchors are sets of patch markers with high responsiveness selected from the two enhanced views, respectively.

[0054] The training device is based on the first global-local energy contrast activation map. The spatial response distribution reflected from the first patch marker sequence The top-ranking patch markers based on response values ​​were selected as the first semantic anchor set. Similarly, based on the second global-local energy contrast activation map... From the second patch marker sequence Select the second semantic anchor set from the middle .

[0055] The number of elements in the semantic anchor set can be a preset value, or it can be adaptively determined based on factors such as the total number of patches, the type of medical image samples, or the sparsity of the activation map response distribution.

[0056] Perform nearest neighbor matching on the first semantic anchor set and the second semantic anchor set to obtain a set of valid anchor matching pairs.

[0057] Mutual nearest neighbor matching refers to the matching process of establishing a bidirectional nearest neighbor relationship between two sets of semantic anchor points. For example... Figure 6 As shown, for any first semantic anchor in the first semantic anchor set, there exists a nearest neighbor anchor in the second semantic anchor set; at the same time, for this nearest neighbor anchor, its nearest neighbor in the first semantic anchor set is also exactly that first semantic anchor; this relationship of being each other's nearest neighbors is called the mutual nearest neighbor condition.

[0058] The effective anchor pair set refers to the set of anchor pairs that satisfy the mutual nearest neighbor condition and pass the similarity screening. The core function of mutual nearest neighbor matching is to eliminate anchor pairs that are unidirectionally optimal but not semantically corresponding due to factors such as image perturbation, random cropping misalignment, and background similarity through bidirectional consistency constraints, thereby significantly reducing the proportion of noisy matches faced by subsequent consistency constraints.

[0059] The training device is based on the first set of semantic anchors. Second semantic anchor set The set of valid anchor point matching pairs is obtained by filtering through mutual nearest neighbor matching. .

[0060] Step S106: Based on the set of effective anchor matching pairs, construct the anchor consistency loss, and update the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor consistency loss, to obtain the trained backbone feature extraction network.

[0061] Anchor consistency loss is a loss function used to measure the consistency in feature representation between two semantic anchors that belong to a valid anchor matching pair in two augmented views of the same medical image sample. The value of anchor consistency loss is positively correlated with the degree of inconsistency between the feature vectors of the two semantic anchors in the matching pair.

[0062] For each valid anchor matching pair, the training device calculates the single-pair consistency loss based on the difference between the feature vectors of the two semantic anchors in the matching pair (e.g., measured by subtracting cosine similarity or by squared distance), combined with a certain matching weight; then, it summarizes the single-pair consistency losses of all valid anchor matching pairs corresponding to the current medical image sample to obtain the anchor consistency loss corresponding to the medical image sample.

[0063] After obtaining the anchor consistency loss, the training device performs gradient backpropagation and parameter updates on the parameters of the backbone feature extraction network based on this loss. This allows the backbone feature extraction network to gradually learn to form view-invariant representations of lesion-related local regions. The training process can be performed iteratively in batches, with each batch containing a certain number of medical image samples. The training process terminates when the number of training epochs reaches a preset maximum, the loss value stabilizes, or the metrics on the validation set meet a preset standard, thus obtaining the trained backbone feature extraction network. The trained backbone feature extraction network can be used to directly output medical image features or combined with a task head to output prediction results for a specific task.

[0064] like Figure 2 As shown, the frequency domain transformation, channel-by-channel learnable frequency domain filtering, global-local energy contrast activation map generation, semantic anchor selection, and neighbor matching involved in the aforementioned steps S103 to S106 are all auxiliary branches in the training phase to participate in the training of the backbone feature extraction network, and are not part of the backbone feature extraction network itself.

[0065] In this application, the backbone feature extraction network is able to focus its training attention on stable lesion-related regions with local semantic orientation under the synergistic effect of dual-view frequency domain enhancement mechanism and global-local energy contrast mechanism. Furthermore, through the mutual nearest neighbor consistency constraint, the feature tables of the same semantic regions in the two enhanced views are driven to converge to the same stable vector, which ultimately significantly improves the stability and anti-interference ability of the medical image feature extraction model in representing local lesion-related features.

[0066] Optionally, the method further includes: During the inference phase of the trained backbone feature extraction network, the medical image to be processed is acquired. The medical image to be processed is input into the trained backbone feature extraction network to obtain the image features corresponding to the medical image to be processed. Based on the image features, output the feature extraction results or output the prediction results based on the task header.

[0067] In this application, the medical image to be processed refers to the medical image input into the trained backbone feature extraction network during the inference phase to obtain image features or prediction results. Unlike the medical image samples during the training phase, the medical image to be processed usually corresponds to real images in actual clinical scenarios that require feature extraction, assisted diagnosis, or downstream analysis. It no longer needs to generate enhanced views and does not participate in any form of loss calculation or parameter update.

[0068] like Figure 2 As shown in the second half, during the inference phase, the training or deployment device performs the following operations: First, it acquires the medical image to be processed. This image can be acquired directly from medical image acquisition devices, such as endoscopic acquisition systems, computed tomography (CT) scanners, magnetic resonance imaging (MRI) devices, ultrasound imaging devices, fundus cameras, and pathology slide scanners; it can also be retrieved on demand from a medical image archiving system; or it can receive medical image data from other systems via a network interface. The medical image to be processed can undergo routine preprocessing such as size normalization and pixel value normalization before being input into the trained backbone feature extraction network.

[0069] Next, the medical image to be processed is directly input into the trained backbone feature extraction network, which then outputs the image features corresponding to the medical image. These image features can be the global feature vector output from the end of the backbone feature extraction network, a spatial feature map, or a patch label sequence; the specific form is selected according to the needs of the downstream task.

[0070] Finally, depending on the needs of downstream applications, image features can be directly output as feature extraction results for tasks such as retrieval, clustering, and visualization analysis; alternatively, image features can be further input into the task head, which will then output prediction results. The task head refers to a shallow neural network module designed according to a specific downstream task, including but not limited to fully connected layers for classification tasks, decoders for segmentation tasks, prediction heads for detection tasks, and regression heads for lesion localization tasks.

[0071] It is particularly important to note that the entire inference process for the medical image to be processed does not include the frequency domain transformation processing, channel-by-channel learnable frequency domain filtering processing, generation of the first and second global-local energy contrast activation maps, selection of the first and second semantic anchor sets, and neighbor-to-neighbor matching processing mentioned in the training phase. In other words, the frequency domain anchor branch used to assist training is completely disabled during the inference phase.

[0072] In an exemplary embodiment, the ResNet-18 network trained based on the training method provided in the above embodiments is deployed to a hospital image analysis workstation. Each time the workstation receives an endoscopic image as a medical image to be processed, it sequentially feeds it into the ResNet-18 network and the classification task head. The classification task head outputs the lesion category probability distribution as the prediction result. No Fourier transform, energy contrast activation map calculation, or anchor point matching operations are performed during the entire process.

[0073] In this application, by completely confining the frequency domain enhancement and semantic anchor consistency constraint mechanism introduced during the training phase within the training process, and keeping the original computation path of the backbone feature extraction network unchanged during the inference phase, the present invention improves the feature representation stability of the model after training without increasing the number of parameters, floating-point operations, inference latency, and deployment complexity during the inference phase. This is beneficial for the practical application of the present invention in clinical deployment environments.

[0074] Optionally, the step of performing frequency domain transformation on the first spatial feature map and the second spatial feature map respectively, and then performing channel-by-channel learnable frequency domain filtering to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map includes: Two-dimensional Fourier transform and spectral centering are performed on the first spatial feature map and the second spatial feature map respectively to obtain the first complex frequency domain feature and the second complex frequency domain feature respectively; For any complex frequency domain feature among the first complex frequency domain feature and the second complex frequency domain feature, a first channel-wise convolutional layer is used to filter the real part feature of the any complex frequency domain feature to obtain the filtered real part feature corresponding to the any complex frequency domain feature; and a second channel-wise convolutional layer is used to filter the imaginary part feature of the any complex frequency domain feature to obtain the filtered imaginary part feature corresponding to the any complex frequency domain feature. The real part feature of the filter and the imaginary part feature of the filter are combined to form the complex frequency domain feature of the filter corresponding to any complex frequency domain feature. The complex frequency domain feature of the filter is then subjected to inverse spectrum centering and two-dimensional inverse Fourier transform in sequence, and the real part is taken to obtain the frequency domain enhancement feature map corresponding to any complex frequency domain feature. The frequency domain enhancement feature map corresponding to the first complex frequency domain feature is used as the first frequency domain enhancement feature map, and the frequency domain enhancement feature map corresponding to the second complex frequency domain feature is used as the second frequency domain enhancement feature map; Wherein, the first channel-wise convolutional layer and the second channel-wise convolutional layer are both depthwise convolutional layers, and the number of groups in the depthwise convolutional layer is equal to the number of channels of the input feature; the convolutional kernel parameters of the first channel-wise convolutional layer and the second channel-wise convolutional layer are independent of each other.

[0075] like Figure 3 and Figure 4 As shown, the implementation process of this step includes the following sub-steps.

[0076] Two-dimensional Fourier transform and spectral centering processing; training device for the first spatial feature map With the second spatial feature map Perform two-dimensional Fourier transforms in the spatial dimensions, namely the height dimension and the width dimension.

[0077] In practical implementation, a two-dimensional fast Fourier transform can be used to improve computational efficiency. The output of the two-dimensional Fourier transform is a set of complex feature tensors, denoted as the initial complex frequency domain features.

[0078] After the two-dimensional Fourier transform, the initial complex frequency domain features are subjected to spectrum centering, which involves shifting the zero-frequency components from the corner positions of the spectrum to the central region of the spectrum. Spectrum centering can be implemented using FFTShift. The purpose of spectrum centering is to concentrate low-frequency components in the center of the spectrum and distribute high-frequency components around the periphery, facilitating the subsequent application of localized, learnable filtering in the frequency domain through neighborhood convolution.

[0079] After the above processing, the first complex frequency domain features are obtained respectively. Second complex frequency domain features This process can be represented as: ; in, This represents a two-dimensional Fourier transform or a two-dimensional fast Fourier transform. This indicates spectrum centering processing.

[0080] Independent filtering of the real and imaginary parts channel by channel; targeting the first complex frequency domain characteristics With the second complex frequency domain features For any complex frequency domain feature in the training device, the real part and the imaginary part are taken respectively.

[0081] Let any complex frequency domain characteristic be . In fact, part of the quantity is Re( The imaginary component is Im( The training device uses a first-channel progressive convolutional layer. GR For Re( Apply a learnable filter to obtain the real part features of the filter: ; And a second channel-by-channel convolutional layer is used. right By applying a learnable filter, the imaginary part of the filter feature is obtained: ; in, This represents the real part of the characteristic of a complex number. This represents the imaginary part of the characteristics of a complex number. This indicates a channel-by-channel learnable filter with the real part. This represents a channel-by-channel learnable filter with an imaginary part. and This can be achieved using deep convolutional layers, and the parameters of the two layers are independent of each other.

[0082] This grouping method enables each feature channel to learn an independent frequency response in the frequency domain that matches its own semantic function, avoiding forced sharing of frequency responses between different channels.

[0083] First channel-wise convolutional layer With the second channel-by-channel convolutional layer The convolutional kernel parameters are independent of each other. That is, for the same feature channel, the kernel parameters of the real part and the kernel parameters of the imaginary part are not shared, which makes the network have different modulation capabilities for the amplitude and phase components in the frequency domain. The kernel size of deep convolutional layers is usually an odd number, such as 3×3, 5×5, 7×7, etc., and this invention does not limit the kernel size.

[0084] Combining the filtered results and performing an inverse frequency domain transform; the training device will use the real part features of the filter. With the characteristics of the imaginary part of the filter The combination yields the complex frequency domain characteristics of the filter: ; in, j It represents the imaginary unit.

[0085] Then, inverse spectrum centering (e.g., IFFTShift) is performed on the filtered complex frequency domain features to restore the zero-frequency component from the center position to the corner position of the spectrum; then, a two-dimensional inverse Fourier transform is performed and the real part is taken to obtain the frequency domain enhanced feature map corresponding to the current complex frequency domain features: ; in, Indicates the inverse Fourier transform. This indicates inverse spectrum centering processing, such as IFFTShift processing.

[0086] After the above processing, the first frequency domain enhanced feature maps are obtained respectively. Second frequency domain enhanced feature map .

[0087] In one exemplary embodiment, the input spatial feature map has a size of 7×7 and 512 channels. The training device first performs a two-dimensional Fast Fourier Transform on a 7×7 scale to obtain 7×7 complex frequency domain features, and then performs FFTShift processing; subsequently, a deep convolutional layer with 512 groups and a kernel size of 3×3 is used. GR Filtering is performed on the real part, using the same number of groups of 512, a convolution kernel size of 3×3, and... Depth convolutional layers with independent parameters The imaginary part is filtered; then the filtered real and imaginary parts are combined in imaginary units, and IFFTShift and two-dimensional inverse fast Fourier transform are performed in sequence, and the real part is taken to obtain a frequency domain enhanced feature map with 512 channels and a spatial size of 7×7.

[0088] In this application, by decoupling the real and imaginary parts of the complex frequency domain features and passing them to two sets of parameter-independent deep convolutional layers for channel-by-channel filtering, the backbone network can finely adjust the frequency response curve of each channel in the frequency domain and distinguish between the real components related to amplitude information and the imaginary components related to phase information. This overcomes the frequency band selection bias caused by fixed high-pass, fixed low-pass, or real-imaginary shared filters, and improves the preservation quality of lesion-related fine structural information in the frequency domain enhanced feature map.

[0089] Optionally, the method further includes: At the beginning of the training phase of the backbone feature extraction network, the kernel parameters and biases of any one of the first and second channel-wise convolutional layers are initialized. The initialization process includes: initializing the weight at the center position of the convolution kernel of any channel-wise convolutional layer to one, initializing the weights at other positions in the convolution kernel of any channel-wise convolutional layer other than the center position to zero, and initializing the bias of any channel-wise convolutional layer to zero.

[0090] In this application, approximate identity mapping initialization refers to an initialization strategy that sets the parameters of the convolutional layer at the beginning of training to make its effect approximate an identity mapping. Specifically, for any convolutional layer of any depth, there exists a unique center position in its convolutional kernel. For example, for a 3×3 convolutional kernel, the center position is the element corresponding to coordinate (1,1); for a 5×5 convolutional kernel, the center position is the element corresponding to coordinate (2,2).

[0091] Before training begins, the training device initializes its parameters according to the following rules: the weight value at the center of the convolution kernel is initialized to one; the weight values ​​at all other positions in the convolution kernel except the center position are initialized to zero; and the bias term of the convolutional layer is initialized to zero.

[0092] Under these initialization conditions, the convolution operation of each channel is equivalent to performing an approximate original value transfer on the input components during the early stages of training. At the start of training, the frequency domain filtering layer does not impose significant destructive perturbations on the original spectral information, and the entire frequency domain branch approximates a transparent path.

[0093] As the training process progresses, the gradient backpropagation mechanism will guide the weights of the convolution kernels in the first and second channel-wise convolutional layers, except for the center position, to gradually deviate from zero, enabling the deep convolutional layers to gradually learn a frequency response pattern that adapts to the statistical characteristics of medical image samples.

[0094] By constraining the channel-wise filters of the real and imaginary parts to an approximate identity mapping in the early stage of training, the network avoids uncontrollable destruction of the original spectral information under the initial random weights. This allows the pre-trained representation capability of the backbone feature extraction network to be fully preserved in the early stage of training and to adapt to the frequency domain response in a progressive learning manner, which significantly improves the stability and convergence reliability of the training process.

[0095] Optionally, the step of generating a first global-local energy contrast activation map based on the first frequency domain enhanced feature map and generating a second global-local energy contrast activation map based on the second frequency domain enhanced feature map includes: For any one of the first and second frequency domain enhancement feature maps, channel aggregation is performed on the any one frequency domain enhancement feature map to obtain a single-channel energy map corresponding to the any one frequency domain enhancement feature map; For each spatial location in the single-channel energy map, a preset local neighborhood centered on the spatial location is determined, and the local energy variance corresponding to the spatial location is obtained based on the non-negative difference between the mean square energy value and the square of the mean energy value within the preset local neighborhood. A local energy variance map is formed by the local energy variances corresponding to all spatial locations in the single-channel energy map. Based on the local energy variance map, a global-local energy contrast activation map corresponding to any frequency domain enhancement feature map is generated. The global-local energy contrast activation map corresponding to the first frequency domain enhanced feature map is used as the first global-local energy contrast activation map, and the global-local energy contrast activation map corresponding to the second frequency domain enhanced feature map is used as the second global-local energy contrast activation map.

[0096] In this application, channel aggregation refers to the process of integrating the frequency domain enhancement feature maps of multiple channels along the channel dimension to obtain a single-channel feature map that reflects the overall energy level of each spatial location. Channel aggregation can be implemented in ways including, but not limited to, channel averaging, channel weighted averaging, channel max pooling, and compression using a learnable projection module.

[0097] The training device enhances the feature map in the first frequency domain. With the second frequency domain enhanced feature map Any frequency domain enhanced feature map Perform channel aggregation to obtain the corresponding single-channel energy map: ; in, ψ (·) represents the channel aggregation function. .

[0098] For example, when using channel averaging for aggregation, a single-channel energy map can be represented as: ; in, The first frequency domain enhanced feature map c One channel.

[0099] Training equipment for single-channel energy maps Each spatial location in ( i , j Determine a preset local neighborhood centered on this spatial location. .

[0100] The preset local neighborhood can be a 3×3 neighborhood, a 5×5 neighborhood, or a square neighborhood of other odd sizes; this invention does not impose any limitations.

[0101] After determining the preset local neighborhood, the training device calculates two statistics based on the energy values ​​within the preset local neighborhood: the squared mean of energy and the square of the energy mean.

[0102] Specifically, energy mean Defined as a preset local neighborhood The average value of the internal energy can be obtained by performing an average pooling operation on the single-channel energy map with a preset local neighborhood as the kernel: ; Energy square mean Defined as the result obtained by first performing an element-wise squaring operation on the single-channel energy map, and then performing an average pooling operation with a preset local neighborhood as the kernel: ; Here, ⊙ represents element-wise multiplication.

[0103] Subsequently, spatial location ( i , j The corresponding local energy variance is defined as a non-negative difference between the squared mean of energy and the square of the energy mean: ; The purpose of using the `max(·,0)` operation is to avoid small negative values ​​in the local energy variance due to numerical errors in floating-point operations, thereby ensuring the numerical stability of subsequent standardization processing based on this value. The local energy variance map is obtained by combining the local energy variances corresponding to all spatial locations in the single-channel energy map.

[0104] The training device is based on the local energy variance map. Furthermore, its statistics across the entire space are calculated, and based on this, the deviation of the local energy variance at each spatial location from the overall value is measured, thereby generating the corresponding global-local energy contrast activation map. .

[0105] By first compressing multi-channel frequency domain information into a single-channel energy representation through channel aggregation, and then calculating the local energy variance at spatial locations based on the second-order moment statistics of the local neighborhood, the resulting local energy variance map can stably and differentiably characterize the "local activity" of the energy distribution at each spatial location, providing a reliable feature basis for subsequent relative deviation measurement based on global statistics.

[0106] Optionally, generating a global-local energy contrast activation map corresponding to any frequency domain enhanced feature map based on the local energy variance map includes: Calculate the global mean and global dispersion statistics of the local energy variance map over the entire space. An energy comparison value is obtained based on the local energy variance, the global mean, and the global dispersion statistic; wherein the energy comparison value is directly proportional to the difference between the local energy variance and the global mean, inversely proportional to the sum of the global dispersion statistic and a preset constant, and the activation intensity is adjusted by a learnable scaling parameter; The energy contrast value is nonlinearly mapped to obtain the global-local energy contrast activation map corresponding to any frequency domain enhanced feature map.

[0107] Further, the local energy variance map is calculated. The global mean and global dispersion statistics across the entire space. Let the global mean be: ; Let the global dispersion statistic be: ; in, It can be variance, standard deviation, or a function of both. In one specific implementation, The global variance of the local energy variance map is taken; in another embodiment, The global standard deviation of the local energy variance map can be taken.

[0108] Based on the local energy variance, the global mean, and the global dispersion statistic, the energy comparison value is obtained: ; in, This indicates a learnable scaling parameter. This represents a preset constant used to avoid a denominator of zero. The energy comparison value is input into a nonlinear activation function to obtain a global-local energy comparison activation map. ; in, It can be the Sigmoid function, the Softmax function, the Tanh function, or other normalization functions. In one specific implementation, the Sigmoid function is used.

[0109] By introducing a standardized comparison mechanism based on global mean and global dispersion, the response at each spatial location no longer depends on absolute energy intensity, but reflects its degree of anomaly relative to the statistics of the entire feature map. This effectively suppresses strong energy interference such as reflection, boundary artifacts, and repetitive anatomical textures that exist in the form of common features across the entire map. Furthermore, the network is endowed with the ability to adaptively adjust the sharpness of the response through learnable scaling parameters.

[0110] Optionally, after generating the first global-local energy contrast activation map and the second global-local energy contrast activation map, the method further includes: For any spatial feature map in the first spatial feature map and the second spatial feature map, the global-local energy contrast activation map corresponding to the first spatial feature map is multiplied positionally with the first spatial feature map, and then added positionally with the first spatial feature map to obtain the residual enhancement feature map corresponding to the first spatial feature map; wherein, the global-local energy contrast activation map corresponding to the first spatial feature map is the first global-local energy contrast activation map, and the global-local energy contrast activation map corresponding to the second spatial feature map is the second global-local energy contrast activation map; The residual enhancement feature map corresponding to the first spatial feature map is used as the first residual enhancement feature map, and the residual enhancement feature map corresponding to the second spatial feature map is used as the second residual enhancement feature map. The residual enhanced feature map corresponding to the first spatial feature map and the residual enhanced feature map corresponding to the second spatial feature map are used for obtaining patch features corresponding to subsequent semantic anchors.

[0111] In this application, residual fusion refers to a fusion method that combines the activation map and the spatial feature map by multiplying position by position and adding the original features.

[0112] The introduction of residual structures avoids completely masking the original information in regions with low activation map responses, thus preserving the original discriminative ability of the backbone feature extraction network on the learned categories.

[0113] For the first spatial feature map With the second spatial feature map Any spatial feature map in The training device performs the following operation: [The operation involves] generating the spatial feature map... Corresponding global-local energy contrast activation map Broadcast along the channel dimension to The same number of channels are used; then position-by-position multiplication is performed to obtain weighted features; finally, the weighted features are compared with the original spatial feature map. By adding them according to their positions, we obtain the residual enhanced feature map corresponding to this spatial feature map: ; Where ⊙ represents element-wise multiplication. Broadcast along the channel dimension.

[0114] After the above processing, the first residual enhanced feature maps corresponding to the first spatial feature maps are obtained respectively. Second residual enhanced feature map corresponding to the second spatial feature map .

[0115] Subsequently, when selecting the set of semantic anchors, the corresponding patch features can be obtained from the first residual enhanced feature map and the second residual enhanced feature map after being expanded according to the patch grid. This ensures that the selected semantic anchors not only correspond to high response regions in spatial location, but their feature vectors have also been weighted and amplified by the activation response.

[0116] Optionally, selecting a first set of semantic anchor points from the first patch marker sequence based on the first global-local energy contrast activation map, and selecting a second set of semantic anchor points from the second patch marker sequence based on the second global-local energy contrast activation map, includes: For any one of the first global-local energy contrast activation maps and the second global-local energy contrast activation map, the any one global-local energy contrast activation map is divided into multiple patch regions according to the spatial grid corresponding to the patch marker sequence; For each patch region, the activation values ​​within the patch region are aggregated to obtain the patch response score corresponding to the patch region; Normalize all the patch response scores to obtain the patch response distribution corresponding to any global-local energy contrast activation map; Based on the patch response distribution, a preset number of patch tags with the highest response values ​​are selected from the corresponding patch tag sequence as the semantic anchor set corresponding to any global-local energy contrast activation graph; The set of semantic anchor points corresponding to the first global-local energy contrast activation map is used as the first semantic anchor point set, and the set of semantic anchor points corresponding to the second global-local energy contrast activation map is used as the second semantic anchor point set.

[0117] In this application, a global-local energy contrast activation map is used. The patch area is divided into multiple patch regions according to the patch grid corresponding to the patch marker sequence. Let the first... The space support area for each patch area is... Then the first The first sample The patch response score is: ; Training equipment for samples b The patch response scores corresponding to all patch regions are normalized to obtain the patch response distribution corresponding to the current global-local energy contrast activation map. ; When the denominator is less than the preset threshold, a uniform distribution can be used as the patch response distribution, or a uniform distribution can be used as the patch response distribution on non-background patches.

[0118] Select the highest-scoring patch response based on its distribution. Each patch serves as a semantic anchor, and its index set can be represented as: ; Based on the index set Extract a set of candidate semantic anchors from the patch features corresponding to the patch marker sequence or residual enhanced feature map: ; in, This indicates that the highest score is selected. One element, This indicates that the corresponding patch tag is extracted according to the index set.

[0119] After the above processing, the first global-local energy contrast activation map is... Obtain its corresponding first semantic anchor set For the second global-local energy contrast activation map Obtain its corresponding set of second semantic anchor points This is used for subsequent nearest neighbor matching steps.

[0120] In one specific implementation, In other embodiments, It can be other preset values, or it can be adaptively determined based on the total number of patches or the distribution of patch responses.

[0121] In this application, the activation graph is discretized by patch mesh, normalized, and then... The selection transforms the continuous spatial response into a sparse selection strategy for patch labels, enabling subsequent consistency constraints to accurately focus on several patch locations with high potential lesion relevance, thus avoiding semantic redundancy and training noise caused by applying uniform constraints to the entire map region.

[0122] Optionally, a nearest neighbor matching is performed on the first semantic anchor set and the second semantic anchor set to obtain a set of valid anchor matching pairs, including: A similarity matrix is ​​obtained based on the similarity between each first semantic anchor in the first semantic anchor set and each second semantic anchor in the second semantic anchor set; For any first semantic anchor in the first semantic anchor set, the nearest neighbor semantic anchor in the second semantic anchor set is determined based on the similarity matrix. When the nearest neighbor of the nearest semantic anchor point in the first semantic anchor point set is any one of the first semantic anchor points, and the similarity between any one of the first semantic anchor points and the nearest neighbor semantic anchor point is not lower than a preset similarity threshold, the any one of the first semantic anchor points and the nearest neighbor semantic anchor point are determined as a valid anchor point matching pair; The set of valid anchor point matching pairs is composed of all the determined valid anchor point matching pairs.

[0123] In this application, regarding the first b A medical image sample is used to train the device to calculate the first semantic anchor point set. Each first semantic anchor point in With the second semantic anchor set Each second semantic anchor in The similarity between them is used to construct a similarity matrix: ; in, Indicates the first The sample in the first enhanced view A semantic anchor point, Indicates the first The sample in the second enhanced view A semantic anchor point, This represents a similarity function. In one specific implementation, the similarity function is cosine similarity: ; Retain anchor point pairs that satisfy the condition of mutual nearest neighbor and whose similarity is not lower than a preset threshold to obtain a set of valid anchor point matching pairs: in, This indicates a preset similarity threshold. In one specific implementation, By using nearest neighbor constraints, incorrect anchor point matching caused by enhancement perturbations, cropping misalignment, or similar backgrounds can be reduced.

[0124] In this application, by introducing a bidirectional consistency constraint of mutual nearest neighbors and a dual screening mechanism of similarity threshold, pseudo-matching pairs that are unidirectionally optimal but semantically inconsistent caused by factors such as image enhancement perturbation, random cropping offset, and background texture similarity are effectively eliminated. This ensures that the subsequent consistency loss only applies to stable anchor point pairs that truly correspond to the semantics, significantly reducing the noise sensitivity of the training process.

[0125] Optionally, constructing an anchor consistency loss based on the set of effective anchor matching pairs includes: For each of the valid anchor point matching pairs in the set of valid anchor point matching pairs, the matching weight is determined based on the patch response scores corresponding to the two semantic anchor points in the valid anchor point matching pair. Based on the matching weight and the similarity between the two semantic anchors, calculate the single-pair consistency loss corresponding to the effective anchor matching pair; The anchor point consistency loss corresponding to the medical image sample is obtained by weighted summation of all the single-pair consistency losses corresponding to the medical image sample. The single-pair consistency loss is obtained based on the difference between the cosine similarity between the two semantic anchors, or based on the squared distance between the two semantic anchors.

[0126] In this application, for valid anchor point matching pairs The matching weight is determined based on the patch response scores corresponding to the two anchor points: ; The design of matching weights can also take other forms such as weighted sums, maximum values, minimum values, or learnable functions. Regardless of the method used, the value of the matching weight is positively correlated with the response strength of the two semantic anchors on their respective activation maps, so that the stronger the response of the matching pair, the greater its proportion in the final loss.

[0127] For each valid anchor point matching pair ( i , j The training device calculates the one-to-one consistency loss based on the similarity between the two semantic anchors in the matching pair.

[0128] The training device performs a weighted summation and normalization on all one-to-one consistency losses corresponding to the current medical image sample, using the matching weights as weights, to obtain the anchor point consistency loss corresponding to that medical image sample. Taking the weighted cosine consistency loss as an example, it can be expressed as: ; In this application, by linking the matching weights to the patch response scores, the consistency loss is made to give a greater gradient contribution to anchor pairs with stronger responses that are more likely to correspond to the real lesion locations. At the same time, two loss implementations, cosine similarity and squared distance, are provided to facilitate flexible selection for the feature scales of different backbone networks, which significantly improves the semantic specificity of consistency constraints.

[0129] To avoid a small number of unreliable matches affecting training, the anchor consistency loss can be calculated only for samples with at least a preset minimum number of valid matches. Let the set of valid samples be: The batch-level anchor consistency loss is: in, This indicates a preset minimum number of matches. In one specific implementation, In another implementation, a weighted distance consistency loss can also be used, that is, the loss in the formula... Replace with the squared distance between the two anchor features.

[0130] Optionally, updating the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor consistency loss to obtain the trained backbone feature extraction network includes: The anchor point consistency loss and the basic training loss are combined according to weights to obtain the total training loss; Based on the total training loss, the parameters of the backbone feature extraction network and the filtering parameters corresponding to the channel-by-channel learnable frequency domain filtering are updated to obtain the trained backbone feature extraction network. The basic training loss includes at least one of representation learning loss, classification loss, clustering loss, pseudo-label loss, and entropy regularization loss; the weight corresponding to the anchor consistency loss is gradually increased in a linear warm-up manner at the beginning of training, and is maintained at the preset maximum weight after reaching the preset number of warm-up rounds.

[0131] In this application, the training device will incorporate anchor consistency loss. Compared with basic training loss The total training loss is obtained by linearly combining the weights: ; in, These are the weights corresponding to the anchor consistency loss. Basic training loss. It can be at least one of the following: representation learning loss (e.g., contrastive learning loss, InfoNCE loss, cluster contrastive loss, etc.), classification loss (e.g., cross-entropy loss), clustering loss (e.g., DeepCluster type loss), pseudo-label loss (e.g., FixMatch-based loss), and entropy regularization loss. That is, the anchor consistency loss proposed in this invention can be used as an auxiliary constraint in conjunction with multiple main task losses.

[0132] Training equipment based on total training loss Gradient backpropagation is performed on the parameters of the backbone feature extraction network, and the parameters are updated using a preset optimizer (such as SGD, Adam, AdamW, etc.). The training process is carried out in batches and iteratively until the number of training epochs reaches the preset maximum or the validation set index meets the preset conditions, thereby obtaining the trained backbone feature extraction network.

[0133] To avoid imposing excessively strong consistency constraints on the backbone feature extraction network at the beginning of training (when the semantic anchors selected by the global-local energy contrast activation map are not yet stable), the training device adjusts the anchor consistency loss weights. The heating was gradually increased using a linear preheating method.

[0134] Specifically, in the first t During training rounds, weights Determine as follows: ; in, Indicates the current training round. Indicates the number of preheating cycles. This represents the maximum weight of the anchor point consistency loss. In one specific implementation, .

[0135] In this application, by using anchor consistency loss as an auxiliary term in conjunction with multiple basic training losses, and progressively introducing constraint strength through linear warm-up, the training dominance of the main task loss is preserved, while the backbone feature extraction network can fully obtain the local representation gain brought by consistency constraints after the anchor selection is stable, thereby improving the stability of the training process and the reproducibility of the final performance.

[0136] Optionally, the sources of medical image samples may include, but are not limited to, the following: Endoscopic images: video frames or static images from examination equipment such as gastrointestinal endoscopes, bronchoscopes, and cystoscopes, mainly used for the identification of lesions such as mucosal lesions, polyps, and tumors; CT images: slice images or three-dimensional volume data from computed tomography (CT) equipment, mainly used for the detection of lung nodules, liver lesions, bone lesions, etc. MRI images: multimodal slice images derived from magnetic resonance imaging equipment, mainly used for the analysis of brain, spine, joint, soft tissue tumors, etc. Ultrasound images: Two-dimensional or three-dimensional images derived from ultrasound imaging equipment, mainly used for the examination of the thyroid gland, breast, abdominal organs, etc. Fundus images: Color or fluorescence angiography images obtained from fundus cameras, mainly used for screening diabetic retinopathy, glaucoma, macular degeneration, etc. Pathological images: These are whole or partial slide images obtained from a pathological slide scanner and are mainly used for cell morphology analysis, tumor grading, and immunohistochemical analysis.

[0137] The output features of the trained backbone feature extraction network can be applied to at least one of the following tasks: Classification task: The output features are fed into the classification task head to obtain the probability distribution of lesion categories; Search task: Construct a search index for similar medical images based on output features, supporting image search, similar case retrieval, etc. Clustering task: Perform unsupervised clustering of unlabeled medical images based on output features to discover potential disease subtypes; Segmentation task: The output features are fed into the segmentation task head to obtain the pixel-level mask of the lesion region; Lesion localization task: The output features are sent to the localization task head, and the location coordinates or bounding box of the lesion are output. Unknown category discovery task: Identify new category samples not seen during the training phase based on output features, and support the gradual introduction of new categories; Assisted analysis task: Use the output features as input features for downstream doctor decision support systems to provide a quantifiable description of lesion features.

[0138] The training method provided by this invention does not impose specific limitations on the imaging modality of medical images. The output features of the trained backbone feature extraction network can be adapted to various downstream analysis tasks, making this solution widely applicable to intelligent medical image analysis systems in different clinical scenarios, and possessing good versatility and scalability.

[0139] The model training apparatus provided by the present invention is described below. The model training apparatus described below and the model training method described above can be referred to in correspondence.

[0140] Figure 7 This is a schematic diagram of the model training device provided by the present invention, as shown below. Figure 7 As shown, it includes: The acquisition module 710 is used to acquire medical image samples and generate a first enhanced view and a second enhanced view based on the medical image samples; The input module 720 is used to input the first enhanced view and the second enhanced view into the backbone feature extraction network respectively to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The transformation module 730 is used to perform frequency domain transformation on the first spatial feature map and the second spatial feature map respectively and perform channel-wise learnable frequency domain filtering to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part features and imaginary part features of the frequency domain features separately using independent filtering parameters; The generation module 740 is used to generate a first global-local energy contrast activation map based on the first frequency domain enhanced feature map, and to generate a second global-local energy contrast activation map based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; The selection module 750 is used to select a first semantic anchor set from the first patch marker sequence according to the first global-local energy contrast activation map; and select a second semantic anchor set from the second patch marker sequence according to the second global-local energy contrast activation map, and perform mutual nearest neighbor matching on the first semantic anchor set and the second semantic anchor set to obtain a set of effective anchor matching pairs; The construction module 760 is used to construct an anchor consistency loss based on the set of effective anchor matching pairs, and update the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor consistency loss, so as to obtain the trained backbone feature extraction network.

[0141] In this application, frequency domain transformation is performed on two enhanced views of the same medical image sample, and channel-wise learnable filtering is performed on the real and imaginary parts of the frequency domain features using independent filtering parameters. This enables the backbone feature extraction network to adaptively learn frequency responses related to lesion semantics across the entire spectrum, avoiding the biased loss of boundary details or low-frequency structures caused by fixed-band filtering. Simultaneously, a global-local energy contrast activation map is generated based on the deviation of local energy from global energy statistics, effectively suppressing global common interferences such as reflections, boundary artifacts, and repetitive anatomical textures, while highlighting local regions related to potential lesions. Furthermore, semantic anchors are selected from the patch marker sequence based on the global-local energy contrast activation map, and unreliable correspondences caused by disturbances, misalignments, or background similarity are screened between the semantic anchor sets of the two enhanced views through nearest neighbor matching. Consistency constraints are applied only to stable and locally semantically oriented effective anchor matching pairs, thereby promoting the backbone feature extraction network to form view-invariant and stable feature representations of lesion-related local regions, significantly improving the reliability of the medical image feature extraction model at the local lesion level.

[0142] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a model training method, which includes: acquiring medical image samples and generating a first enhanced view and a second enhanced view based on the medical image samples; The first enhanced view and the second enhanced view are respectively input into the backbone feature extraction network to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The first spatial feature map and the second spatial feature map are subjected to frequency domain transformation and channel-wise learnable frequency domain filtering respectively to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; A first global-local energy contrast activation map is generated based on the first frequency domain enhanced feature map, and a second global-local energy contrast activation map is generated based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; Based on the first global-local energy contrast activation map, a first semantic anchor set is selected from the first patch marker sequence; and based on the second global-local energy contrast activation map, a second semantic anchor set is selected from the second patch marker sequence. The first semantic anchor set and the second semantic anchor set are matched for nearest neighbors to obtain a set of effective anchor matching pairs. Based on the set of effective anchor matching pairs, an anchor consistency loss is constructed, and the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter are updated based on the anchor consistency loss to obtain the trained backbone feature extraction network.

[0143] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the model training method provided by the above methods, the method including: acquiring medical image samples, and generating a first enhanced view and a second enhanced view based on the medical image samples; The first enhanced view and the second enhanced view are respectively input into the backbone feature extraction network to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The first spatial feature map and the second spatial feature map are subjected to frequency domain transformation and channel-wise learnable frequency domain filtering respectively to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; A first global-local energy contrast activation map is generated based on the first frequency domain enhanced feature map, and a second global-local energy contrast activation map is generated based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; Based on the first global-local energy contrast activation map, a first semantic anchor set is selected from the first patch marker sequence; and based on the second global-local energy contrast activation map, a second semantic anchor set is selected from the second patch marker sequence. The first semantic anchor set and the second semantic anchor set are matched for nearest neighbors to obtain a set of effective anchor matching pairs. Based on the set of effective anchor matching pairs, an anchor consistency loss is constructed, and the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter are updated based on the anchor consistency loss to obtain the trained backbone feature extraction network.

[0145] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the model training methods provided by the methods described above, the method comprising: acquiring medical image samples, and generating a first enhanced view and a second enhanced view based on the medical image samples; The first enhanced view and the second enhanced view are respectively input into the backbone feature extraction network to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The first spatial feature map and the second spatial feature map are subjected to frequency domain transformation and channel-wise learnable frequency domain filtering respectively to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; A first global-local energy contrast activation map is generated based on the first frequency domain enhanced feature map, and a second global-local energy contrast activation map is generated based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; Based on the first global-local energy contrast activation map, a first semantic anchor set is selected from the first patch marker sequence; and based on the second global-local energy contrast activation map, a second semantic anchor set is selected from the second patch marker sequence. The first semantic anchor set and the second semantic anchor set are matched for nearest neighbors to obtain a set of effective anchor matching pairs. Based on the set of effective anchor matching pairs, an anchor consistency loss is constructed, and the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter are updated based on the anchor consistency loss to obtain the trained backbone feature extraction network.

[0146] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that, include: Acquire medical image samples, and generate a first enhanced view and a second enhanced view based on the medical image samples; The first enhanced view and the second enhanced view are respectively input into the backbone feature extraction network to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The first spatial feature map and the second spatial feature map are subjected to frequency domain transformation and channel-wise learnable frequency domain filtering respectively to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; A first global-local energy contrast activation map is generated based on the first frequency domain enhanced feature map, and a second global-local energy contrast activation map is generated based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; Based on the first global-local energy contrast activation map, a first semantic anchor set is selected from the first patch marker sequence; and based on the second global-local energy contrast activation map, a second semantic anchor set is selected from the second patch marker sequence. The first semantic anchor set and the second semantic anchor set are matched for nearest neighbors to obtain a set of effective anchor matching pairs. Based on the set of effective anchor matching pairs, an anchor consistency loss is constructed, and the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter are updated based on the anchor consistency loss to obtain the trained backbone feature extraction network.

2. The method according to claim 1, characterized in that, The method further includes: During the inference phase of the trained backbone feature extraction network, the medical image to be processed is acquired. The medical image to be processed is input into the trained backbone feature extraction network to obtain the image features corresponding to the medical image to be processed. Based on the image features, output the feature extraction results or output the prediction results based on the task header.

3. The method according to claim 1, characterized in that, The step of performing frequency domain transformation on the first spatial feature map and the second spatial feature map respectively, and then performing channel-by-channel learnable frequency domain filtering to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map includes: Two-dimensional Fourier transform and spectral centering are performed on the first spatial feature map and the second spatial feature map respectively to obtain the first complex frequency domain feature and the second complex frequency domain feature respectively; For any complex frequency domain feature among the first complex frequency domain feature and the second complex frequency domain feature, a first channel-wise convolutional layer is used to filter the real part feature of the any complex frequency domain feature to obtain the filtered real part feature corresponding to the any complex frequency domain feature; and a second channel-wise convolutional layer is used to filter the imaginary part feature of the any complex frequency domain feature to obtain the filtered imaginary part feature corresponding to the any complex frequency domain feature. The real part feature of the filter and the imaginary part feature of the filter are combined to form the complex frequency domain feature of the filter corresponding to any complex frequency domain feature. The complex frequency domain feature of the filter is then subjected to inverse spectrum centering and two-dimensional inverse Fourier transform in sequence, and the real part is taken to obtain the frequency domain enhancement feature map corresponding to any complex frequency domain feature. The frequency domain enhancement feature map corresponding to the first complex frequency domain feature is used as the first frequency domain enhancement feature map, and the frequency domain enhancement feature map corresponding to the second complex frequency domain feature is used as the second frequency domain enhancement feature map; Wherein, the first channel-wise convolutional layer and the second channel-wise convolutional layer are both depthwise convolutional layers, and the number of groups in the depthwise convolutional layer is equal to the number of channels of the input feature; the convolutional kernel parameters of the first channel-wise convolutional layer and the second channel-wise convolutional layer are independent of each other.

4. The method according to claim 3, characterized in that, The method further includes: At the beginning of the training phase of the backbone feature extraction network, the kernel parameters and biases of any one of the first and second channel-wise convolutional layers are initialized. The initialization process includes: initializing the weight at the center position of the convolution kernel of any channel-wise convolutional layer to one, initializing the weights at other positions in the convolution kernel of any channel-wise convolutional layer other than the center position to zero, and initializing the bias of any channel-wise convolutional layer to zero.

5. The method according to claim 1, characterized in that, The step of generating a first global-local energy contrast activation map based on the first frequency domain enhanced feature map and generating a second global-local energy contrast activation map based on the second frequency domain enhanced feature map includes: For any one of the first and second frequency domain enhancement feature maps, channel aggregation is performed on the any one frequency domain enhancement feature map to obtain a single-channel energy map corresponding to the any one frequency domain enhancement feature map; For each spatial location in the single-channel energy map, a preset local neighborhood centered on the spatial location is determined, and the local energy variance corresponding to the spatial location is obtained based on the non-negative difference between the mean square energy value and the square of the mean energy value within the preset local neighborhood. A local energy variance map is formed by the local energy variances corresponding to all spatial locations in the single-channel energy map. Based on the local energy variance map, a global-local energy contrast activation map corresponding to any frequency domain enhancement feature map is generated. The global-local energy contrast activation map corresponding to the first frequency domain enhanced feature map is used as the first global-local energy contrast activation map, and the global-local energy contrast activation map corresponding to the second frequency domain enhanced feature map is used as the second global-local energy contrast activation map.

6. The method according to claim 5, characterized in that, The step of generating a global-local energy contrast activation map corresponding to any frequency domain enhanced feature map based on the local energy variance map includes: Calculate the global mean and global dispersion statistics of the local energy variance map over the entire space. An energy comparison value is obtained based on the local energy variance, the global mean, and the global dispersion statistic; wherein the energy comparison value is directly proportional to the difference between the local energy variance and the global mean, inversely proportional to the sum of the global dispersion statistic and a preset constant, and the activation intensity is adjusted by a learnable scaling parameter; The energy contrast value is nonlinearly mapped to obtain the global-local energy contrast activation map corresponding to any frequency domain enhanced feature map.

7. The method according to claim 1, characterized in that, After generating the first global-local energy contrast activation map and the second global-local energy contrast activation map, the method further includes: For any spatial feature map in the first spatial feature map and the second spatial feature map, the global-local energy contrast activation map corresponding to the first spatial feature map is multiplied positionally with the first spatial feature map, and then added positionally with the first spatial feature map to obtain the residual enhancement feature map corresponding to the first spatial feature map; wherein, the global-local energy contrast activation map corresponding to the first spatial feature map is the first global-local energy contrast activation map, and the global-local energy contrast activation map corresponding to the second spatial feature map is the second global-local energy contrast activation map; The residual enhancement feature map corresponding to the first spatial feature map is used as the first residual enhancement feature map, and the residual enhancement feature map corresponding to the second spatial feature map is used as the second residual enhancement feature map. The first residual enhanced feature map and the second residual enhanced feature map are used to obtain the patch features corresponding to the subsequent semantic anchor points.

8. The method according to claim 1, characterized in that, Based on the first global-local energy contrast activation map, a first set of semantic anchor points is selected from the first patch labeling sequence, and based on the second global-local energy contrast activation map, a second set of semantic anchor points is selected from the second patch labeling sequence, including: For any one of the first global-local energy contrast activation maps and the second global-local energy contrast activation map, the any one global-local energy contrast activation map is divided into multiple patch regions according to the spatial grid corresponding to the patch marker sequence; For each patch region, the activation values ​​within the patch region are aggregated to obtain the patch response score corresponding to the patch region; Normalize all the patch response scores to obtain the patch response distribution corresponding to any global-local energy contrast activation map; Based on the patch response distribution, a preset number of patch tags with the highest response values ​​are selected from the corresponding patch tag sequence as the semantic anchor set corresponding to any global-local energy contrast activation graph; The set of semantic anchor points corresponding to the first global-local energy contrast activation map is used as the first semantic anchor point set, and the set of semantic anchor points corresponding to the second global-local energy contrast activation map is used as the second semantic anchor point set.

9. The method according to claim 8, characterized in that, Perform nearest neighbor matching on the first set of semantic anchors and the second set of semantic anchors to obtain a set of valid anchor matching pairs, including: A similarity matrix is ​​obtained based on the similarity between each first semantic anchor in the first semantic anchor set and each second semantic anchor in the second semantic anchor set; For any first semantic anchor in the first semantic anchor set, the nearest neighbor semantic anchor in the second semantic anchor set is determined based on the similarity matrix. When the nearest neighbor of the nearest semantic anchor point in the first semantic anchor point set is any one of the first semantic anchor points, and the similarity between any one of the first semantic anchor points and the nearest neighbor semantic anchor point is not lower than a preset similarity threshold, the any one of the first semantic anchor points and the nearest neighbor semantic anchor point are determined as a valid anchor point matching pair; The set of valid anchor point matching pairs is composed of all the determined valid anchor point matching pairs.

10. The method according to claim 9, characterized in that, Based on the set of effective anchor point matching, an anchor point consistency loss is constructed, including: For each of the valid anchor point matching pairs in the set of valid anchor point matching pairs, the matching weight is determined based on the patch response scores corresponding to the two semantic anchor points in the valid anchor point matching pair. Based on the matching weight and the similarity between the two semantic anchors, calculate the single-pair consistency loss corresponding to the effective anchor matching pair; The anchor point consistency loss corresponding to the medical image sample is obtained by weighted summation of all the single-pair consistency losses corresponding to the medical image sample. The single-pair consistency loss is obtained based on the difference between the cosine similarity between the two semantic anchors, or based on the squared distance between the two semantic anchors.

11. The method according to claim 1, characterized in that, The step of updating the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor point consistency loss to obtain the trained backbone feature extraction network includes: The anchor point consistency loss and the basic training loss are combined according to weights to obtain the total training loss; Based on the total training loss, the parameters of the backbone feature extraction network and the filtering parameters corresponding to the channel-by-channel learnable frequency domain filtering are updated to obtain the trained backbone feature extraction network. The basic training loss includes at least one of representation learning loss, classification loss, clustering loss, pseudo-label loss, and entropy regularization loss; the weight corresponding to the anchor consistency loss is gradually increased in a linear warm-up manner at the beginning of training, and is maintained at the preset maximum weight after reaching the preset number of warm-up rounds.

12. A model training device, characterized in that, include: An acquisition module is used to acquire medical image samples and generate a first enhanced view and a second enhanced view based on the medical image samples; The input module is used to input the first enhanced view and the second enhanced view into the backbone feature extraction network respectively to obtain the first spatial feature map and the first patch label sequence corresponding to the first enhanced view, and the second spatial feature map and the second patch label sequence corresponding to the second enhanced view. The transformation module is used to perform frequency domain transformation on the first spatial feature map and the second spatial feature map respectively and perform channel-wise learnable frequency domain filtering to obtain a first frequency domain enhanced feature map and a second frequency domain enhanced feature map; wherein, the channel-wise learnable frequency domain filtering processes the real part and imaginary part of the frequency domain features separately using independent filtering parameters; The generation module is used to generate a first global-local energy contrast activation map based on the first frequency domain enhanced feature map, and to generate a second global-local energy contrast activation map based on the second frequency domain enhanced feature map; wherein, both the first global-local energy contrast activation map and the second global-local energy contrast activation map are generated based on the degree of deviation of local energy from global energy statistics; The selection module is used to select a first set of semantic anchors from the first patch marker sequence based on the first global-local energy contrast activation map; and to select a second set of semantic anchors from the second patch marker sequence based on the second global-local energy contrast activation map, and to perform mutual nearest neighbor matching on the first set of semantic anchors and the second set of semantic anchors to obtain a set of effective anchor matching pairs; The construction module is used to construct an anchor consistency loss based on the set of effective anchor matching pairs, and update the parameters of the backbone feature extraction network and the filter parameters corresponding to the channel-by-channel learnable frequency domain filter based on the anchor consistency loss, so as to obtain the trained backbone feature extraction network.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the model training method as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 11.

Images