A sustainable learning multi-feature fusion method

Abstract

The application discloses a sustainable learning multi-feature fusion method, comprising the following steps: determining a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and respectively determining a plurality of target frequency domain feature sub-data of each target training picture; establishing a cross loss function of an image processing model according to a feature fusion process of the plurality of frequency domain feature sub-data in the image processing model; and training the image processing model to obtain an image processing model learning the target image field by taking minimizing the cross loss function as a training target. Through the synergistic effect of frequency domain feature extraction, double loss function construction and continuous optimization training, the dynamic adaptation capability and recognition accuracy of the image processing model in a multi-modal fusion task are improved, and the stability and expansibility of the model are enhanced.

Description

Technical Field

[0001] This application belongs to the field of image processing, specifically relating to a sustainable learning-based multi-feature fusion method and image processing methods, apparatus, storage media, devices, and computer program products based on sustainable learning-based multi-feature fusion. Background Technology

[0002] In fields such as computer vision and medical image processing, multimodal feature fusion technology is increasingly becoming a key supporting tool. Especially in clinical diagnosis and intraoperative navigation, physicians often need to simultaneously refer to multiple image modalities, such as high-resolution magnetic resonance imaging (HR-MRI), computed tomography (CT), magnetic resonance angiography (MRA), and digital subtraction angiography (DSA), to comprehensively understand the spatial structure and functional status of lesions. Each of these modalities has its own advantages and limitations; therefore, fusing their feature information has become an important path to improve diagnostic accuracy and processing efficiency.

[0003] Existing multimodal image fusion methods typically rely on static feature extraction and fusion frameworks. They employ convolutional neural networks (CNNs) to extract features from images of different modalities, extract high and low frequency information through frequency domain segmentation, Gaussian filters, and other means, and then use dense connection structures for fusion.

[0004] However, with the continuous expansion of multi-source data, traditional models struggle to adapt dynamically, leading to a decline in fusion performance over time. Secondly, the imbalance between the quantity and quality of samples makes the model prone to bias towards high-quality or high-frequency categories during training, affecting overall recognition performance. Summary of the Invention

[0005] This application aims to provide a sustainable learning-based multi-feature fusion method and an image processing method, apparatus, storage medium, device, and computer program product based on sustainable learning-based multi-feature fusion, which at least solves the problems of poor dynamic adaptability and recognition performance of traditional image processing models.

[0006] In a first aspect, embodiments of this application disclose a multi-feature fusion method for sustainable learning, including:

[0007] Multiple target training images are identified in the target image domain to be learned in multiple image domains, and multiple target frequency domain feature sub-data are determined for each target training image; each target training image is used to record a first target object in the target image domain through different image modalities; each target frequency domain feature sub-data has a corresponding frequency domain range;

[0008] Based on the feature fusion process performed on multiple frequency domain feature sub-data in the image processing model, a cross-loss function for the image processing model is established. This cross-loss function is jointly constrained by the equivalent distillation loss of the target training image on the image processing model and the focusing loss of the target image domain on all image domains. The equivalent distillation loss characterizes the knowledge distillation loss generated during the knowledge distillation process of using the target training image as extracted knowledge on the image processing model. The focusing loss characterizes the image sample differences between all training images in the target image domain and all training images in the image domain.

[0009] The image processing model is trained with the goal of minimizing the cross-loss function, so as to obtain the image processing model that has learned the target image domain.

[0010] Secondly, embodiments of this application also disclose an image processing method based on multi-feature fusion of sustainable learning, including:

[0011] Acquire multiple target images to be processed; each target image is used to record a second target object through a different image modality;

[0012] Multiple target images are used as input data and input into an image processing model trained according to the multi-feature fusion method of sustainable learning described in the first aspect, so as to obtain a feature fusion processed image of multiple target images.

[0013] Thirdly, embodiments of this application also disclose a multi-feature fusion device for sustainable learning, comprising:

[0014] The feature extraction module is used to determine multiple target training images in the target image domain to be learned in multiple image domains, and to determine multiple target frequency domain feature sub-data for each target training image; each target training image is used to record a first target object in the target image domain through different image modalities; each target frequency domain feature sub-data has a corresponding frequency domain range;

[0015] The function construction module is used to establish a cross-loss function for the image processing model based on the feature fusion process performed on multiple frequency domain feature sub-data in the image processing model. The cross-loss function is jointly constrained by the equivalent distillation loss of the target training image on the image processing model and the focusing loss of the target image domain on all image domains. The equivalent distillation loss is used to characterize the knowledge distillation loss generated during the knowledge distillation process of using the target training image as extracted knowledge on the image processing model. The focusing loss is used to characterize the image sample differences between all training images in the target image domain and all training images in the image domain.

[0016] The model training module is used to train the image processing model with the goal of minimizing the cross loss function, so as to obtain the image processing model that has learned the target image domain.

[0017] Fourthly, embodiments of this application also disclose an image processing apparatus based on multi-feature fusion of sustainable learning, comprising:

[0018] The image acquisition module is used to acquire multiple target images to be processed; each target image is used to record a second target object through a different image modality;

[0019] The image processing module is used to input multiple target images as input data into an image processing model trained according to the multi-feature fusion method of sustainable learning described in the first aspect, so as to obtain a feature fusion processed image of multiple target images.

[0020] Fifthly, embodiments of this application also disclose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the first or second aspect.

[0021] In a sixth aspect, embodiments of this application also disclose an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps as described in the first or second aspect.

[0022] In a seventh aspect, embodiments of this application also disclose a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps described in the first or second aspect.

[0023] In summary, in this embodiment, by selecting target training images from multiple image domains and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the image processing model's ability to express target objects and laying a feature foundation for the subsequent establishment of knowledge distillation and focusing optimization metrics. Furthermore, by introducing equivalent distillation loss and focusing loss to jointly constitute a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focusing loss dynamically adjusts the training weights according to the differences in sample quantity and quality, effectively alleviating the sample imbalance problem and enabling the model to... Maintaining stable performance during continuous learning avoids the decline in recognition accuracy that traditional models experience when data is updated. The training process simultaneously preserves old knowledge and adapts to new samples. Ultimately, minimizing the cross-loss function is used as the training objective, ensuring that the image processing model balances knowledge transfer from old tasks with sample differences in each training round. This enables continuous learning of the target image domain, improving the model's adaptability in multimodal image fusion tasks and enhancing its generalization ability across different image domains. This allows the model to maintain the stability of fusion results and the reliability of recognition performance even in scenarios with continuously expanding multi-source data. Therefore, the method based on this application, through the synergistic effect of frequency domain feature extraction, dual loss function construction, and continuous optimization training, improves the dynamic adaptability and recognition accuracy of the image processing model in multimodal fusion tasks. It reduces the risk of model performance degradation when facing challenges such as uneven data quality, imbalanced sample numbers, and continuous modality updates, enhancing the model's stability and scalability, thus providing a more robust and intelligent solution in image processing. Attached Figure Description

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0025] Figure 1 This is a flowchart illustrating the steps of a multi-feature fusion method for sustainable learning provided in an embodiment of this application.

[0026] Figure 2 This is a flowchart illustrating the steps of another sustainable learning multi-feature fusion method provided in this application embodiment;

[0027] Figure 3 This is a flowchart illustrating the steps of an image processing method based on sustainable learning and multi-feature fusion, as provided in an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the structure of a multi-feature fusion device for sustainable learning provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of an image processing device based on sustainable learning and multi-feature fusion provided in an embodiment of this application;

[0030] Figure 6 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0032] Figure 1 This application provides a multi-feature fusion method for sustainable learning, which specifically includes the following steps:

[0033] Step 101: Determine multiple target training images in the target image domain to be learned in multiple image domains, and determine multiple target frequency domain feature sub-data for each target training image.

[0034] Each target training image is used to record the first target object in the target image domain through different image modalities; each target frequency domain feature sub-data has a corresponding frequency domain range.

[0035] In some embodiments of this application, in order to identify and extract representative training data in multiple image domains to build the learning foundation for subsequent fusion models, multiple target training images in the target image domain to be learned are determined, and multiple target frequency domain feature sub-data are determined for each target training image. Each target training image is used to record the first target object in the target image domain through different image modalities; each target frequency domain feature sub-data has a corresponding frequency domain range. Frequency domain feature sub-data refers to the high-frequency and low-frequency information extracted from the original image through spectrum segmentation, used to characterize edge details and overall structure in the image, respectively. In this way, the image processing model can obtain more discriminative feature representation capabilities, providing a stable and diverse feature foundation for subsequent knowledge distillation and the construction of the focusing loss function.

[0036] In a specific example, a medical image processing task selects brain images as the target image domain and acquires multiple modalities of images, including HR-MRI, CT, MRA, and DSA. Each image can be segmented in the frequency domain to extract multiple frequency domain feature sub-data, and their corresponding frequency domain ranges can be labeled. In this way, the image processing model obtains a training dataset containing information from different modalities and frequencies, providing a multi-dimensional input foundation for subsequently building a fusion model, thereby improving the model's ability to identify and adapt to brain lesion structures.

[0037] Step 102: Based on the feature fusion process of multiple frequency domain feature sub-data in the image processing model, establish the cross loss function for the image processing model.

[0038] The cross-loss function is jointly constrained by the equivalent distillation loss of the target training image to the image processing model and the focusing loss of the target image domain to the entire image domain. The equivalent distillation loss is used to characterize the knowledge distillation loss generated in the process of using the target training image as extracted knowledge to perform knowledge distillation on the image processing model. The focusing loss is used to characterize the image sample differences between all training images in the target image domain and all training images in the entire image domain.

[0039] In some embodiments of this application, to establish an optimization mechanism that can simultaneously preserve old knowledge and adapt to new samples, thereby enhancing the continuous learning ability of image processing models in multimodal fusion tasks, a cross-loss function for the image processing model is established based on the feature fusion process performed on multiple frequency domain feature sub-data in the image processing model. This cross-loss function is jointly constrained by the equivalent distillation loss of the target training image on the image processing model and the focusing loss of the target image domain on all image domains. The equivalent distillation loss characterizes the knowledge distillation loss generated during the knowledge distillation process of the image processing model using the target training image as extracted knowledge. The focusing loss characterizes the image sample differences between all training images in the target image domain and all training images in all image domains. The equivalent distillation loss maintains the model's output distribution for old tasks, preventing the forgetting of the original knowledge structure during the training of new tasks. The focusing loss, by adjusting sample weights and soft truncation parameters, guides the model to focus on categories with fewer or lower-quality samples, thereby alleviating the sample imbalance problem. In this way, the image processing model obtains a training objective with dual constraints, which enables the model to take into account both knowledge transfer and sample differences during the fusion process, thereby improving the model's stability and generalization ability.

[0040] In a specific example, in a brain image fusion task, the training objective of the image processing model is constructed based on the extracted frequency domain feature sub-data. First, the output probability distribution of each target training image in the old and new models is calculated, and an equivalent distillation loss is constructed accordingly. Then, based on the differences in the quantity and quality of samples in each image domain, the weight parameters and soft cutoff factor in the focusing loss are set. In this way, the image processing model obtains a cross-loss function that incorporates both knowledge preservation and sample balance constraints, providing a stable and adaptive optimization objective for subsequent training, thereby improving the model's continuous learning ability in multimodal image fusion tasks.

[0041] Step 103: Train the image processing model with the goal of minimizing the cross loss function to obtain an image processing model that has learned the target image domain.

[0042] In some embodiments of this application, in order to guide the image processing model to continuously learn the target image domain by optimizing the objective function, thereby improving the model's adaptability and generalization ability in multimodal fusion tasks, the image processing model is trained with minimizing the cross-loss function as the training objective, thus obtaining an image processing model that has learned the target image domain. By minimizing this loss function, the model can dynamically adjust its parameters during training, so that the fusion result retains the original knowledge structure while adapting to new modalities and sample differences. Ultimately, the image processing model acquires the ability to continuously learn the target image domain, maintaining the stability of the fusion effect and the reliability of the recognition performance in scenarios where multi-source data is constantly expanding.

[0043] In a specific example, in a brain multimodal image fusion task, a pre-constructed cross-loss function was used as the training objective to iteratively train the image processing model. During training, the model continuously adjusted its parameters based on the frequency domain features of modalities such as HR-MRI, CT, MRA, and DSA images to minimize the sum of distillation and focusing losses. Ultimately, the model successfully learned the feature distribution patterns in the brain imaging domain and demonstrated stronger recognition and adaptability in subsequent fusion tasks, effectively addressing the challenges of inconsistent sample quality and continuous modal updates.

[0044] In summary, in this embodiment, by selecting target training images from multiple image domains and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the image processing model's ability to express target objects and laying a feature foundation for the subsequent establishment of knowledge distillation and focusing optimization metrics. Furthermore, by introducing equivalent distillation loss and focusing loss to jointly constitute a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focusing loss dynamically adjusts the training weights according to the differences in sample quantity and quality, effectively alleviating the sample imbalance problem and enabling the model to... Maintaining stable performance during continuous learning avoids the decline in recognition accuracy that traditional models experience when data is updated. The training process simultaneously preserves old knowledge and adapts to new samples. Ultimately, minimizing the cross-loss function is used as the training objective, ensuring that the image processing model balances knowledge transfer from old tasks with sample differences in each training round. This enables continuous learning of the target image domain, improving the model's adaptability in multimodal image fusion tasks and enhancing its generalization ability across different image domains. This allows the model to maintain the stability of fusion results and the reliability of recognition performance even in scenarios with continuously expanding multi-source data. Therefore, the method based on this application, through the synergistic effect of frequency domain feature extraction, dual loss function construction, and continuous optimization training, improves the dynamic adaptability and recognition accuracy of the image processing model in multimodal fusion tasks. It reduces the risk of model performance degradation when facing challenges such as uneven data quality, imbalanced sample numbers, and continuous modality updates, enhancing the model's stability and scalability, thus providing a more robust and intelligent solution in image processing.

[0045] Figure 2 This embodiment provides another sustainable learning multi-feature fusion method, which specifically includes the following steps:

[0046] Step 201: Determine multiple target training images in the target image domain to be learned in multiple image domains, and determine multiple target frequency domain feature sub-data for each target training image.

[0047] Each target training image is used to record the first target object in the target image domain through different image modalities; each target frequency domain feature sub-data has a corresponding frequency domain range.

[0048] The method shown in this step has been explained in step 101 and will not be repeated here.

[0049] Optionally, in order to determine multiple target frequency domain feature sub-data for each target training image, step 201 includes the following sub-steps:

[0050] Sub-step 2011: Based on the convolution of each target training image, obtain the target frequency domain feature data corresponding to each target training image.

[0051] In some embodiments of this application, in order to extract frequency domain feature data from the original image and provide a foundation for subsequent spectrum segmentation and feature sub-data generation, target frequency domain feature data corresponding to each target training image is obtained by convolving each target training image. Convolution is a common image processing method. By sliding the convolution kernel across the image and performing weighted summation, local structural information in the image can be extracted. In this application, convolution is used to initially extract the frequency domain feature data of the image, typically using a 3×3 convolution kernel to capture basic features such as edges and textures. Thus, the system obtains the frequency domain feature data for each target training image, providing operable input data for subsequent spectrum segmentation using a Gaussian filter, thereby improving the accuracy and efficiency of feature extraction.

[0052] In a specific example, when processing high-resolution MRI (HR-MRI) images of the brain, a 3×3 convolution kernel is applied to each image to extract primary frequency domain features. This operation performs a weighted summation within the neighborhood of each pixel in the image, generating a set of frequency domain feature data that reflects the local variations of the image at different spatial locations. Thus, each HR-MRI image is converted into a set of frequency domain feature data, laying the foundation for subsequent spectrum segmentation and high / low frequency information extraction, thereby improving the feature representation capability of the image processing model in brain structure recognition tasks. Specifically, the system can convert the generated high-frequency component F... 0H The data is transmitted to the high-frequency feature extraction unit, which consists of multiple residual-in-residual dense blocks (RRDBs) and an adaptive feature aggregation module to enhance the expressive power of high-frequency details. Simultaneously, the low-frequency information component F... 0L It is introduced into the low-frequency feature extraction unit, which contains only a small number of RRDB blocks and self-adjusting feature integration modules to control computational resources and avoid over-processing of low-frequency information.

[0053] Sub-step 2012 involves performing spectral segmentation on each target frequency domain feature data according to multiple preset Gaussian filters, to obtain multiple target frequency domain feature sub-data corresponding to each target training image.

[0054] Each Gaussian filter corresponds to a different Gaussian kernel.

[0055] In some embodiments of this application, to further refine the frequency domain feature data into multiple frequency band sub-data to enhance the image processing model's ability to recognize different frequency information, each target frequency domain feature data is spectrally segmented according to multiple preset Gaussian filters to obtain multiple target frequency domain feature sub-data corresponding to each target training image. Each Gaussian filter corresponds to a different Gaussian kernel. A Gaussian filter is a commonly used tool for image smoothing and frequency separation; its core parameters are the size and standard deviation of the Gaussian kernel, which determine the filter's response range to image frequencies. In this application, by setting multiple filters with different Gaussian kernel sizes, multi-level segmentation of the frequency domain feature data can be achieved, thereby extracting high-frequency and low-frequency components separately. Thus, the system obtains multiple frequency domain feature sub-data, each corresponding to the image's feature expression within a specific frequency range, providing a more discriminative input basis for subsequent feature fusion and loss function construction.

[0056] In a specific example, when processing the frequency domain feature data of brain CT images, three Gaussian filters are applied for spectral segmentation, with Gaussian kernel sizes of 3×3, 5×5, and 7×7, respectively. Each filter performs a convolution operation on the frequency domain feature data to extract high-frequency, mid-frequency, and low-frequency information from the image. Thus, each CT image is decomposed into three frequency domain feature sub-data, reflecting the structural details and texture features of the image at different frequency levels. This provides multi-scale input data for the subsequent feature fusion model, thereby improving the model's frequency sensitivity and fusion accuracy in brain lesion identification tasks.

[0057] Optionally, sub-step 2012 includes the following sub-steps:

[0058] Sub-step 20121 involves performing spectral filtering on the target frequency domain feature data using a first Gaussian filter among multiple Gaussian filters to obtain the first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and then performing spectral filtering on the target frequency domain feature data using multiple second Gaussian filters among multiple Gaussian filters to obtain the second target frequency domain feature decomposition sub-data of the target frequency domain feature data.

[0059] Among them, the first Gaussian filter is the Gaussian filter corresponding to the largest Gaussian kernel among the multiple Gaussian filters; the second Gaussian filter is the Gaussian filter other than the first Gaussian filter among the multiple Gaussian filters.

[0060] In some embodiments of this application, in order to perform multi-scale spectral filtering on the target frequency domain feature data to separate image features at different frequency levels and provide a foundation for the construction of subsequent frequency domain feature sub-data, the target frequency domain feature data is spectrally filtered using a first Gaussian filter among multiple Gaussian filters to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data. Then, the target frequency domain feature data is spectrally filtered using multiple second Gaussian filters among multiple Gaussian filters to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data. The first Gaussian filter is the Gaussian filter corresponding to the largest Gaussian kernel among the multiple Gaussian filters; the second Gaussian filter is the Gaussian filter other than the first Gaussian filter among the multiple Gaussian filters. Larger Gaussian kernels tend to retain low-frequency information and are suitable for capturing the overall structure of the image; while smaller Gaussian kernels are more sensitive to high-frequency information and are suitable for extracting edge and detail features in the image. In this way, the system obtains multi-frequency level decomposition sub-data, providing a hierarchical basis for subsequent difference calculation and frequency domain feature sub-data generation, thereby improving the image processing model's responsiveness to frequency changes and the resolution of feature representation.

[0061] In a specific example, when processing the frequency domain feature data of brain CT images, three Gaussian filters are applied for spectral filtering. The first Gaussian filter has a Gaussian kernel size of 7×7, corresponding to the low-frequency components of the image; the second Gaussian filters have Gaussian kernel sizes of 5×5 and 3×3, corresponding to the mid-frequency and high-frequency components. First, the first Gaussian filter is used to process the frequency domain feature data to obtain the first target frequency domain feature decomposition sub-data. Then, two second Gaussian filters are used to process the data to obtain the second target frequency domain feature decomposition sub-data. Ultimately, the system successfully extracts image features at multiple frequency levels, providing a multi-scale input foundation for subsequent construction of frequency domain feature sub-data and feature fusion operations, thereby improving the model's frequency resolution and fusion accuracy in brain structure recognition tasks.

[0062] Sub-step 20122: The difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data is determined as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and the difference between the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter is determined as the target frequency domain feature sub-data corresponding to each second Gaussian filter respectively.

[0063] In some embodiments of this application, in order to extract frequency domain feature sub-data corresponding to each Gaussian filter from the spectral decomposition results to achieve feature separation and representation at the frequency level, the difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data is determined as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter is determined as the target frequency domain feature sub-data corresponding to each second Gaussian filter respectively. The difference operation is used to highlight the parts of the original frequency domain feature data that were not retained by the first Gaussian filter, thereby enhancing the expressive ability of high-frequency details; while directly using the decomposition result of the second Gaussian filter retains the feature response of mid-to-high frequency components. The role of the Gaussian filter in the frequency domain is to perform weighted suppression or enhancement of different frequency components. Therefore, by combining the difference and decomposition results, target frequency domain feature sub-data at multiple frequency levels can be constructed. Finally, the system obtains a set of frequency domain feature sub-data corresponding to each Gaussian filter, providing multi-scale frequency domain input for subsequent feature fusion, classification, or recognition tasks, which helps to improve the model's sensitivity and expressive ability to image frequency changes.

[0064] In a specific example, when processing the frequency domain feature data of brain CT images, the system obtains first target frequency domain feature decomposition sub-data (generated by a 7×7 Gaussian kernel filter) and two second target frequency domain feature decomposition sub-data (generated by 5×5 and 3×3 Gaussian kernel filters, respectively). The difference between the original frequency domain feature data and the first decomposition sub-data is calculated to obtain the frequency domain feature sub-data corresponding to the first Gaussian filter; simultaneously, the two second decomposition sub-data are used as the frequency domain feature sub-data corresponding to their respective Gaussian filters. Ultimately, the system successfully constructs target frequency domain feature sub-data at three frequency levels, corresponding to low-frequency, mid-frequency, and high-frequency features, providing multi-scale frequency domain feature support for subsequent image structure recognition and lesion detection.

[0065] Optional, cross-loss function Characterized by the following formula:

[0066] ,

[0067] in, This represents the equivalent distillation loss. This indicates a focus on losses;

[0068] , representing the predicted probabilities of the old model and the new model for sample x belonging to class k, respectively, are smoothed using the temperature parameter T:

[0069] in, Let these represent the output probabilities of the old model and the new model on sample x, respectively. , These represent the original output values ​​of the new model and the old model for category k, respectively;

[0070] ,

[0071] in, Train sample weights for the target image. This is the soft truncation parameter (the soft truncation parameter is used to control the degree of attention given to difficult-to-classify samples; the larger the value, the more it is biased towards difficult samples). Let x be the predicted probability of the model that sample x belongs to class k. , These represent the training sample sets for the old category and the new category, respectively.

[0072] The purpose of this cross-loss function is to simultaneously achieve knowledge transfer and class focus during model training, thereby improving the model's generalization and class discrimination capabilities in mixed training scenarios with both old and new samples. Its design principle integrates distillation learning and focusing mechanisms: by using an equivalent distillation loss term to maintain the new model's inheritance of knowledge from the old model, and by using a focusing loss term to enhance the model's attention to difficult-to-classify samples, it improves classification accuracy while maintaining stability.

[0073] Optional, sample weights of the target training images It is positively correlated with the peak signal-to-noise ratio and / or structural similarity evaluation value of the target training image.

[0074] The initial purpose of sample weight design is to incorporate prior information about image quality into the loss function, enabling the model to dynamically adjust its learning intensity based on image quality differences during training, thereby improving its feature extraction capability for high-quality samples. The principle is to use the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as metrics to measure image quality, and establish a positive correlation between these metrics and the sample weight coefficient α. This means that samples with higher image quality have a larger weight in the loss function and a stronger influence on training. In this way, the model prioritizes the effective feature information carried by high-quality images during training, thereby improving the overall training effect and the model's generalization ability.

[0075] Optionally, the PSNR of the target training image is calculated according to the following formula:

[0076] ,

[0077] in, MAX represents the theoretical maximum value of the target training image, and H and W represent the height and width of the target training image, respectively. , These represent two different target training images;

[0078] The SSIM of the target training image is calculated using the following formula:

[0079] ,

[0080] in, , , , These represent the preset weight parameters. and , and are the mean and standard deviation of the target training image x and all training images y, respectively. Let x and y be the covariances. , , These are the preset adjustment parameters.

[0081] The initial purpose of establishing the image quality evaluation formula was to quantitatively evaluate the quality of target training images during the training process, thereby providing a basis for setting sample weights. Its principle is to use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Score (SSIM) to measure the reconstruction accuracy and structural fidelity of the image, respectively. PSNR mainly reflects the magnitude of pixel-level errors in the image and is suitable for evaluating image sharpness and compression quality; SSIM, on the other hand, measures the image's brightness (…). ), contrast ( ) and structure ( This method comprehensively evaluates the similarity between images across three dimensions, more closely resembling human visual perception. By using these two metrics together, the overall quality level of the target training images can be reflected, and their weights in the loss function can be adjusted accordingly.

[0082] Step 202: Based on the feature fusion process of multiple frequency domain feature sub-data in the image processing model, establish the cross loss function for the image processing model.

[0083] The cross-loss function is jointly constrained by the equivalent distillation loss of the target training image to the image processing model and the focusing loss of the target image domain to the entire image domain. The equivalent distillation loss is used to characterize the knowledge distillation loss generated in the process of using the target training image as extracted knowledge to perform knowledge distillation on the image processing model. The focusing loss is used to characterize the image sample differences between all training images in the target image domain and all training images in the entire image domain.

[0084] The method shown in this step has been explained in step 102 and will not be repeated here.

[0085] Optionally, multiple frequency domain feature sub-data points correspond to different Gaussian kernels. The feature fusion process of multiple frequency domain feature sub-data points in the image processing model is specifically implemented through the following process:

[0086] Sub-step 2020: Using the first frequency domain feature sub-data among multiple frequency domain feature sub-data as the initial value of the fusion process, adaptive feature fusion is performed on the second frequency domain feature sub-data among multiple frequency domain feature sub-data against the initial value.

[0087] Among them, the first Gaussian kernel corresponding to the first frequency domain feature sub-data is the largest Gaussian kernel among all Gaussian kernels.

[0088] In some embodiments of this application, to achieve multi-frequency hierarchical feature fusion in the image processing model and enhance the model's comprehensive ability to express image structure, the first frequency domain feature sub-data among multiple frequency domain feature sub-data is used as the initial value for the fusion process, and the second frequency domain feature sub-data among multiple frequency domain feature sub-data is used to adaptively fuse the initial value. The first Gaussian kernel corresponding to the first frequency domain feature sub-data is the largest Gaussian kernel among all Gaussian kernels. Adaptive feature fusion is a processing method that dynamically adjusts the fusion strategy based on the correlation and weights between features, typically implemented through attention mechanisms or weighted averaging, to retain key features and suppress redundant information. The largest Gaussian kernel corresponding to the first frequency domain feature sub-data mainly retains the low-frequency components of the image in the frequency domain, representing the overall structural information of the image; while the second frequency domain feature sub-data contains mid-to-high frequency components, representing the details and edge information of the image. Finally, the system obtains fused frequency domain feature data, which enhances the detail expression ability while maintaining the overall image structure, providing more discriminative feature input for subsequent image recognition, segmentation, or classification tasks.

[0089] In a specific example, when processing frequency domain feature data from brain CT images, the initial fusion values ​​are selected from the frequency domain feature data generated by a 7×7 Gaussian kernel. This data primarily contains low-frequency information about brain structures. Subsequently, frequency domain feature data generated by 5×5 and 3×3 Gaussian kernels are used as the second frequency domain feature data, and the initial values ​​are fused using an adaptive fusion module. This fusion module dynamically adjusts the fusion weights based on the response intensity and spatial distribution of each frequency domain feature data. Ultimately, the system obtains fused frequency domain feature data that, while preserving the overall brain contour, enhances the expression of lesion edges and texture details, providing more hierarchical feature support for subsequent lesion detection models.

[0090] Step 203: Train the image processing model with the goal of minimizing the cross loss function to obtain an image processing model that has learned the target image domain.

[0091] The method shown in this step has been explained in step 103 and will not be repeated here.

[0092] In summary, in this embodiment, by selecting target training images from multiple image domains and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the image processing model's ability to express target objects and laying a feature foundation for the subsequent establishment of knowledge distillation and focusing optimization metrics. Furthermore, by introducing equivalent distillation loss and focusing loss to jointly constitute a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focusing loss dynamically adjusts the training weights according to the differences in sample quantity and quality, effectively alleviating the sample imbalance problem and enabling the model to... Maintaining stable performance during continuous learning avoids the decline in recognition accuracy that traditional models experience when data is updated. The training process simultaneously preserves old knowledge and adapts to new samples. Ultimately, minimizing the cross-loss function is used as the training objective, ensuring that the image processing model balances knowledge transfer from old tasks with sample differences in each training round. This enables continuous learning of the target image domain, improving the model's adaptability in multimodal image fusion tasks and enhancing its generalization ability across different image domains. This allows the model to maintain the stability of fusion results and the reliability of recognition performance even in scenarios with continuously expanding multi-source data. Therefore, the method based on this application, through the synergistic effect of frequency domain feature extraction, dual loss function construction, and continuous optimization training, improves the dynamic adaptability and recognition accuracy of the image processing model in multimodal fusion tasks. It reduces the risk of model performance degradation when facing challenges such as uneven data quality, imbalanced sample numbers, and continuous modality updates, enhancing the model's stability and scalability, thus providing a more robust and intelligent solution in image processing.

[0093] Figure 3 This application provides an image processing method based on sustainable learning and multi-feature fusion, which specifically includes the following steps:

[0094] Step 301: Obtain multiple target images to be processed.

[0095] Each target image is used to record the second target object through different image modalities.

[0096] In some embodiments of this application, to provide input data for the image processing model to complete feature fusion and recognition tasks in subsequent steps, multiple target images are acquired. Each target image is used to record a second target object through a different image modality. Target images refer to image data collected in real-world application scenarios, typically including modal images such as HR-MRI, CT, MRA, and DSA. Each modal image records the structural or functional features of the second target object under different imaging mechanisms, exhibiting complementary information. Thus, the image processing model obtains multimodal input data, providing a multi-dimensional image foundation for subsequent feature fusion processing, which helps improve the model's accuracy and robustness in recognizing target objects.

[0097] In a specific example, during preoperative assessment of brain lesions, the user acquired image data from four modalities: HR-MRI, CT, MRA, and DSA, to record the spatial structure of cerebral blood vessels, blood flow status, and lesion distribution. Each image corresponds to the same secondary target object, namely a specific lesion region in the patient's brain. The user inputs these images as target images into the image processing system. The system identifies the source and correspondence of each modality of image and performs data preprocessing. In this way, the image processing model obtains multimodal target image input, providing rich and complementary image information for the subsequent fusion model, thereby improving the model's accuracy and adaptability in brain lesion identification tasks.

[0098] Step 302: Input multiple target images as input data into the image processing model trained by the sustainable learning multi-feature fusion method disclosed in the embodiments of this application, so as to obtain a feature fusion processed image of multiple target images.

[0099] In some embodiments of this application, in order to utilize a trained image processing model to perform feature fusion processing on multiple target images and generate an output image with multimodal information, multiple target images are used as input data and input into the image processing model trained according to the sustainable learning multi-feature fusion method disclosed in the embodiments of this application to obtain a feature fusion processed image of multiple target images. The image processing model is trained by minimizing the cross-loss function. This model has the ability to extract and fuse frequency domain features of images of different modalities and can maintain the memory of old knowledge and adapt to new samples during continuous learning. The feature fusion processed image refers to the image generated after fusing multiple modal images at the frequency domain level. This image simultaneously retains the structural information and detailed features of each modal image. Finally, the system obtains the fused image output, providing image data with higher information density and expressive power for subsequent recognition, analysis, or decision support.

[0100] In a specific example, during intraoperative navigation, the user inputs acquired brain HR-MRI, CT, MRA, and DSA images as target images into the image processing system. The system calls upon an image processing model trained based on the method described in this application to perform frequency domain feature extraction and fusion operations on these images. The model uses an adaptive feature fusion mechanism to complete the fusion process based on the frequency domain feature sub-data of each modality image and outputs a processed image based on the feature fusion. Thus, the user obtains a fused image that simultaneously presents the spatial distribution of brain structures, vascular orientation, and lesion details, providing a more comprehensive visual reference for intraoperative localization and intervention.

[0101] It is important to emphasize that during the actual operation of the model, the feature fusion process is performed in the same way as described in sub-step 2020: the system first uses low-frequency feature sub-data as the starting point for fusion, and then introduces mid- and high-frequency feature sub-data in sequence. Through an adaptive mechanism, it completes feature fusion at multiple frequency levels, thereby enhancing the expression of details while maintaining the overall structure of the image, and realizing the effective integration and utilization of frequency domain features in the model.

[0102] For example, the system can employ a hierarchical structure for feature extraction and fusion: the first layer has 64 input channels, indicating that the initial input data contains 64-dimensional feature information in the frequency or spatial domain; after convolution or other processing, this layer outputs 32 channels, compressing the feature dimension to 32 dimensions to extract more discriminative features. The input of the second layer consists of the initial input and the output of the first layer, forming a fused input structure to enhance the model's ability to jointly express the original and intermediate features.

[0103] In summary, in this embodiment, by selecting target training images from multiple image domains and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the image processing model's ability to express target objects and laying a feature foundation for the subsequent establishment of knowledge distillation and focusing optimization metrics. Furthermore, by introducing equivalent distillation loss and focusing loss to jointly constitute a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focusing loss dynamically adjusts the training weights according to the differences in sample quantity and quality, effectively alleviating the sample imbalance problem and enabling the model to... Maintaining stable performance during continuous learning avoids the decline in recognition accuracy that traditional models experience when data is updated. The training process simultaneously preserves old knowledge and adapts to new samples. Ultimately, minimizing the cross-loss function is used as the training objective, ensuring that the image processing model balances knowledge transfer from old tasks with sample differences in each training round. This enables continuous learning of the target image domain, improving the model's adaptability in multimodal image fusion tasks and enhancing its generalization ability across different image domains. This allows the model to maintain the stability of fusion results and the reliability of recognition performance even in scenarios with continuously expanding multi-source data. Therefore, the method based on this application, through the synergistic effect of frequency domain feature extraction, dual loss function construction, and continuous optimization training, improves the dynamic adaptability and recognition accuracy of the image processing model in multimodal fusion tasks. It reduces the risk of model performance degradation when facing challenges such as uneven data quality, imbalanced sample numbers, and continuous modality updates, enhancing the model's stability and scalability, thus providing a more robust and intelligent solution in image processing.

[0104] like Figure 4 As shown in the embodiments of this application, a multi-feature fusion device 40 for sustainable learning is also disclosed, comprising:

[0105] The feature extraction module 401 is used to determine multiple target training images in the target image domain to be learned in multiple image domains, and to determine multiple target frequency domain feature sub-data for each target training image; each target training image is used to record the first target object in the target image domain through different image modalities; each target frequency domain feature sub-data has a corresponding frequency domain range;

[0106] The function construction module 402 is used to establish a cross-loss function for the image processing model based on the feature fusion process performed on multiple frequency domain feature sub-data in the image processing model. The cross-loss function is jointly constrained by the equivalent distillation loss of the target training image to the image processing model and the focusing loss of the target image domain to the entire image domain. The equivalent distillation loss is used to characterize the knowledge distillation loss generated in the process of using the target training image as extracted knowledge to perform knowledge distillation on the image processing model. The focusing loss is used to characterize the image sample differences between all training images in the target image domain and all training images in the entire image domain.

[0107] The model training module 403 is used to train the image processing model with the goal of minimizing the cross loss function, so as to obtain an image processing model that has learned the target image domain.

[0108] Optionally, the feature extraction module 401 includes:

[0109] The convolutional submodule is used to obtain target frequency domain feature data corresponding to each target training image by convolving each target training image;

[0110] The segmentation submodule is used to perform spectral segmentation on each target frequency domain feature data according to multiple preset Gaussian filters, so as to obtain multiple target frequency domain feature sub-data corresponding to each target training image; each Gaussian filter corresponds to a different Gaussian kernel.

[0111] Optionally, the segmented submodules include:

[0112] The filtering unit is used to perform spectral filtering on the target frequency domain feature data through a first Gaussian filter among multiple Gaussian filters to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and to perform spectral filtering on the target frequency domain feature data through multiple second Gaussian filters among multiple Gaussian filters to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data; the first Gaussian filter is the Gaussian filter corresponding to the largest Gaussian kernel among the Gaussian kernels corresponding to the multiple Gaussian filters; the second Gaussian filters are the Gaussian filters other than the first Gaussian filter among the multiple Gaussian filters;

[0113] The determining unit is used to determine the difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and to determine the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter as the target frequency domain feature sub-data corresponding to each second Gaussian filter respectively.

[0114] like Figure 5As shown in the embodiments of this application, an image processing apparatus 50 based on sustainable learning and multi-feature fusion is also disclosed, comprising:

[0115] Image acquisition module 501 is used to acquire multiple target images to be processed; each target image is used to record a second target object through a different image modality;

[0116] Image processing module 502 is used to input multiple target images as input data into an image processing model trained by the sustainable learning multi-feature fusion method disclosed in the embodiments of this application, so as to obtain a feature fusion processed image of multiple target images.

[0117] In summary, in this embodiment, by selecting target training images from multiple image domains and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the image processing model's ability to express target objects and laying a feature foundation for the subsequent establishment of knowledge distillation and focusing optimization metrics. Furthermore, by introducing equivalent distillation loss and focusing loss to jointly constitute a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focusing loss dynamically adjusts the training weights according to the differences in sample quantity and quality, effectively alleviating the sample imbalance problem and enabling the model to... Maintaining stable performance during continuous learning avoids the decline in recognition accuracy that traditional models experience when data is updated. The training process simultaneously preserves old knowledge and adapts to new samples. Ultimately, minimizing the cross-loss function is used as the training objective, ensuring that the image processing model balances knowledge transfer from old tasks with sample differences in each training round. This enables continuous learning of the target image domain, improving the model's adaptability in multimodal image fusion tasks and enhancing its generalization ability across different image domains. This allows the model to maintain the stability of fusion results and the reliability of recognition performance even in scenarios with continuously expanding multi-source data. Therefore, the method based on this application, through the synergistic effect of frequency domain feature extraction, dual loss function construction, and continuous optimization training, improves the dynamic adaptability and recognition accuracy of the image processing model in multimodal fusion tasks. It reduces the risk of model performance degradation when facing challenges such as uneven data quality, imbalanced sample numbers, and continuous modality updates, enhancing the model's stability and scalability, thus providing a more robust and intelligent solution in image processing.

[0118] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described sustainable learning multi-feature fusion method and the image processing method based on sustainable learning multi-feature fusion, achieving the same technical effects. To avoid repetition, these details are not repeated here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] Figure 6 This is a block diagram of an electronic device 700 provided in an embodiment of this application. For example, the electronic device 700 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0120] Reference Figure 6 The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0121] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the aforementioned sustainable learning-based multi-feature fusion method and image processing method based on sustainable learning. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0122] Memory 704 is used to store various types of data to support the operation of electronic device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0123] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0124] Multimedia component 708 includes a screen that provides an output interface between electronic device 700 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When electronic device 700 is in an operating mode, such as shooting mode or multimedia mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0125] Audio component 710 is used to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) used to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0126] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0127] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or a component of electronic device 700, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0128] Communication component 716 facilitates wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 7G), or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0129] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the sustainable learning multi-feature fusion method and the sustainable learning-based multi-feature fusion image processing method provided in the embodiments of this application.

[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to complete the aforementioned sustainable learning multi-feature fusion method and the sustainable learning-based multi-feature fusion image processing method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0131] In an exemplary embodiment, the electronic device 700 may also be provided as a server, including a processing component 702, which further includes one or more processors 720, and memory resources represented by a memory 704 for storing instructions executable by the processing component 702, such as application programs. The application programs stored in the memory 704 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 702 is configured to execute instructions to perform the sustainable learning multi-feature fusion method and the sustainable learning-based multi-feature fusion image processing method provided in the embodiments of this application.

[0132] Electronic device 700 may also include a power supply component 706 configured to perform power management of electronic device 700, a wired or wireless network interface 718 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 712. Electronic device 700 may operate on an operating system stored in memory 704, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0133] This application also provides a computer program product, including a computer program, a sustainable learning multi-feature fusion method and an image processing method based on sustainable learning multi-feature fusion implemented when the computer program is executed by a processor.

[0134] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0135] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0137] It will be readily apparent to those skilled in the art that any combination of the above embodiments is feasible. Therefore, any combination of the above embodiments is an implementation scheme of this application. However, due to space limitations, this specification will not describe them in detail here.

[0138] The sustainable learning-based multi-feature fusion method and image processing method based on sustainable learning-based multi-feature fusion provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. Based on the above description, the required structure for constructing a system with the scheme of this application is obvious. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0139] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0140] Similarly, it should be understood that, for the purpose of simplification and aiding understanding of one or more aspects of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the application. However, this approach of disclosure should not be construed as reflecting an intention that the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the claims, the application aspect comprises fewer features than all of the features of the single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the application.

[0141] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0142] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0143] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sustainable learning multi-feature fusion method and the sustainable learning-based multi-feature fusion image processing method according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0144] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the sustainable learning multi-feature fusion method and the sustainable learning-based multi-feature fusion image processing method of the present application embodiments.

[0145] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0146] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0147] It should be noted that, for the sake of simplicity, the method embodiments of this application are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of this application.

[0148] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for embodiments of systems or devices, since they are basically similar to the method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0149] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for sustainable learning of multi-feature fusion, characterized in that, The method comprises the steps of: determining a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and respectively determining a plurality of target frequency domain feature sub-data of each target training picture; each target training picture is used to record a first target object in the target image field through different picture modalities; each target frequency domain feature sub-data has a corresponding frequency domain range; establishing a cross-loss function of an image processing model according to a feature fusion process of a plurality of target frequency domain feature sub-data in the image processing model; the cross-loss function is jointly constrained by an equivalent distillation loss of the target training picture on the image processing model and a focus loss of the target image field on all image fields; the equivalent distillation loss is used to represent a knowledge distillation loss generated in the process of extracting knowledge from the target training picture to perform knowledge distillation on the image processing model; the focus loss is used to represent the image sample difference of all training pictures in the target image field to all training pictures in all image fields; training the image processing model to obtain the image processing model learned in the target image field, with minimizing the cross-loss function as the training target; the method further comprises the steps of: obtaining target frequency domain feature data corresponding to each target training picture according to convolution of each target training picture; performing frequency spectrum segmentation on each target frequency domain feature data according to a plurality of preset Gaussian filters to obtain a plurality of target frequency domain feature sub-data corresponding to each target training picture; each Gaussian filter corresponds to a different Gaussian kernel; the method further comprises the steps of: performing frequency spectrum filtering on the target frequency domain feature data through a first Gaussian filter in the plurality of Gaussian filters to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and performing frequency spectrum filtering on the target frequency domain feature data through a plurality of second Gaussian filters in the plurality of Gaussian filters to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data; the first Gaussian filter is a Gaussian filter corresponding to the largest Gaussian kernel in the Gaussian kernels corresponding to the plurality of Gaussian filters; the second Gaussian filter is a Gaussian filter other than the first Gaussian filter in the plurality of Gaussian filters; determining the difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and determining the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter as the target frequency domain feature sub-data corresponding to each second Gaussian filter.

2. The method of claim 1, wherein, The plurality of frequency domain feature sub-data respectively correspond to different Gaussian kernels, and a feature fusion process performed by the plurality of frequency domain feature sub-data in an image processing model includes: Taking a first frequency domain feature sub-data in the plurality of frequency domain feature sub-data as an initial value of the fusion process, and performing adaptive feature fusion on the initial value by a second frequency domain feature sub-data in the plurality of frequency domain feature sub-data; the first Gaussian kernel corresponding to the first frequency domain feature sub-data is the largest Gaussian kernel among all the Gaussian kernels.

3. A method for image processing based on sustainable learning and multi-feature fusion, characterized in that, The method comprises: obtaining a plurality of target pictures to be processed; each of the target pictures is used to record a second target object by different picture modalities; inputting the plurality of target pictures as input data into an image processing model trained by the sustainable learning multi-feature fusion method according to any one of claims 1 to 2 to obtain a feature fusion processing image of the plurality of target pictures.

4. A multi-feature fusion device capable of sustainable learning, characterized in that, The method comprises: a feature extraction module configured to determine a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and determine a plurality of target frequency domain feature sub-data of each of the target training pictures; each of the target training pictures is used to record a first target object in the target image field by different picture modalities; each of the target frequency domain feature sub-data has a corresponding frequency domain range; a function construction module configured to establish a cross-loss function of the image processing model according to a feature fusion process performed by the plurality of frequency domain feature sub-data in the image processing model; the cross-loss function is jointly constrained by an equivalent distillation loss of the target training pictures on the image processing model and a focus loss of the target image field on all image fields; the equivalent distillation loss is used to represent a knowledge distillation loss generated in a process of extracting knowledge from the target training pictures to perform knowledge distillation on the image processing model; the focus loss is used to represent image sample difference of all training pictures in the target image field with respect to all training pictures in all image fields; a model training module configured to train the image processing model to obtain the image processing model learned in the target image field by taking minimizing the cross-loss function as a training target; The feature extraction module comprises: a convolution sub-module configured to obtain target frequency domain feature data corresponding to each target training picture according to convolution of each target training picture; a segmentation sub-module configured to perform frequency spectrum segmentation on each target frequency domain feature data according to a plurality of preset Gaussian filters to obtain a plurality of target frequency domain feature sub-data corresponding to each target training picture; each Gaussian filter corresponds to a different Gaussian kernel; The segmentation sub-module comprises: The filtering unit is configured to perform spectral filtering on the target frequency domain feature data by a first Gaussian filter in the plurality of Gaussian filters to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and perform spectral filtering on the target frequency domain feature data by a plurality of second Gaussian filters in the plurality of Gaussian filters to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data; the first Gaussian filter is a Gaussian filter corresponding to a largest Gaussian kernel in Gaussian kernels corresponding to the plurality of Gaussian filters; the second Gaussian filter is a Gaussian filter other than the first Gaussian filter in the plurality of Gaussian filters. The determining unit is configured to determine a difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data as target frequency domain feature sub-data corresponding to the first Gaussian filter, and determine the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter as target frequency domain feature sub-data corresponding to each second Gaussian filter.

5. An image processing device based on sustainable learning and multi-feature fusion, characterized in that, The method comprises: The picture acquisition module is configured to acquire a plurality of target pictures to be processed; Each of the target pictures is used to record a second target object by a different picture modality; The picture processing module is configured to input the plurality of target pictures as input data into an image processing model trained by the method for sustainable learning and multi-feature fusion according to any one of claims 1 to 2, to obtain feature fusion processing images of the plurality of target pictures.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1 to 3.

7. An electronic device, comprising: The computer program product stores a computer program, and the computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 3.

8. A computer program product, characterised in that, The computer program product stores a computer program, and the computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 3.

Images