Multi-feature fusion method for sustainable learning

Abstract

The invention discloses a multi-feature fusion method for sustainable learning, and the method comprises the steps: determining a plurality of target training pictures in a to-be-learned target image field in a plurality of image fields, and determining a plurality of pieces of target frequency domain feature sub-data of each target training picture; establishing a cross loss function of the image processing model according to a feature fusion process of the plurality of frequency domain feature sub-data in the image processing model; and training the image processing model by taking minimization of the cross loss function as a training target so as to obtain an image processing model which learns a target image field. According to the method, through the synergistic effect of frequency domain feature extraction, double loss function construction and continuous optimization training, the dynamic adaptive capacity and recognition precision of the image processing model in the multi-modal fusion task are improved, and the stability and expansibility of the model are enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to a multi-feature fusion method based on sustainable learning and an image processing method, device, storage medium, equipment and computer program product based on multi-feature fusion of sustainable learning. BACKGROUND

[0002] In the field of computer vision and medical image processing, the multi-modal feature fusion technology of images is increasingly becoming a key support means. Especially in clinical diagnosis and intraoperative navigation, doctors often need to refer to multiple image modalities, such as high-resolution magnetic resonance imaging (HR-MRI), computed tomography (CT), magnetic resonance angiography (MRA), and digital subtraction angiography (DSA), to fully understand the spatial structure and functional state of the lesion. These modalities each have advantages and limitations, so fusing their feature information is an important path to improve diagnostic accuracy and processing efficiency.

[0003] Existing multi-modal image fusion methods usually rely on static feature extraction and fusion frameworks, using convolutional neural networks (CNN) to extract features from different modal images, and extracting high and low frequency information through frequency domain segmentation, Gaussian filter, etc. Then, a dense connection structure is used for fusion.

[0004] However, in the face of the continuous expansion of multi-source data, traditional models are difficult to achieve dynamic adaptation, resulting in a decline in fusion effect over time; secondly, the imbalance of sample quantity and quality makes the model prone to bias towards high-quality or high-frequency categories during training, affecting overall recognition performance. SUMMARY

[0005] The present application aims to provide a multi-feature fusion method based on sustainable learning and an image processing method, device, storage medium, equipment and computer program product based on multi-feature fusion of sustainable learning, which at least solves the problems of dynamic adaptability and poor recognition performance of traditional image processing models.

[0006] In a first aspect, the embodiments of the present application disclose a multi-feature fusion method based on sustainable learning, comprising: a plurality of target training pictures in a target image field to be learned in a plurality of image fields are determined, and a plurality of target frequency domain feature sub-data of each of the target training pictures are respectively determined; each of the target training pictures is used to record a first target object in the target image field through different picture modalities; and each of the target frequency domain feature sub-data has a corresponding frequency domain range; According to a feature fusion process of the plurality of frequency domain feature sub-data in an image processing model, a cross-loss function of the image processing model is established; the cross-loss function is jointly constrained by an equivalent distillation loss of the target training picture on the image processing model and a focus loss of the target image field on all image fields; the equivalent distillation loss is used to represent a knowledge distillation loss generated in a process of extracting knowledge of the target training picture to perform knowledge distillation on the image processing model; and the focus loss is used to represent image sample difference of all training pictures in the target image field on all training pictures in all image fields. The image processing model is trained with a training target of minimizing the cross-loss function, to obtain the image processing model learned in the target image field.

[0007] In a second aspect, the embodiments of the present application further disclose a multi-feature fusion image processing method based on sustainable learning, comprising: a plurality of target pictures to be processed are obtained; each of the target pictures is used to record a second target object through different picture modalities; the plurality of target pictures are input into an image processing model trained by the multi-feature fusion method based on sustainable learning according to the first aspect as input data, to obtain feature fusion processing images of the plurality of target pictures.

[0008] In a third aspect, the embodiments of the present application further disclose a multi-feature fusion device based on sustainable learning, comprising: a feature extraction module is configured to determine a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and determine a plurality of target frequency domain feature sub-data of each of the target training pictures; each of the target training pictures is used to record a first target object in the target image field through different picture modalities; and each of the target frequency domain feature sub-data has a corresponding frequency domain range. a function construction module, configured to construct a cross-loss function of the image processing model according to a feature fusion process performed on the plurality of frequency domain feature sub-data in the image processing model; the cross-loss function is jointly constrained by an equivalent distillation loss of the target training picture on the image processing model and a focus loss of the target image field on all image fields; the equivalent distillation loss is used to represent a knowledge distillation loss generated in a process of performing knowledge distillation on the image processing model by taking the target training picture as extracted knowledge; and the focus loss is used to represent an image sample difference of all training pictures in the target image field on all training pictures in all image fields; a model training module, configured to train the image processing model with a training target of minimizing the cross-loss function, to obtain the image processing model learned in the target image field.

[0009] In a fourth aspect, the embodiments of the present application further disclose an image processing device based on sustainable learning and multi-feature fusion, comprising: a picture acquisition module, configured to acquire a plurality of target pictures to be processed; each of the target pictures is used to record a second target object by a different picture modality; a picture processing module, configured to input a plurality of the target pictures as input data into the image processing model trained by the method of sustainable learning and multi-feature fusion according to the first aspect, to obtain feature fusion processing images of the plurality of target pictures.

[0010] In a fifth aspect, the embodiments of the present application further disclose a computer readable storage medium, which stores a computer program; when the computer program is executed by a processor, the steps of the first aspect or the second aspect are implemented.

[0011] In a sixth aspect, the embodiments of the present application further disclose an electronic device, which comprises a processor, a memory, and a computer program stored on the memory and executable on the processor; when the computer program is executed by the processor, the steps of the first aspect or the second aspect are implemented.

[0012] In a seventh aspect, the embodiments of the present application further disclose a computer program product, which stores a computer program; when the computer program is executed by a processor, the steps of the first aspect or the second aspect are implemented.

[0013] In summary, in the embodiments of the present application, by selecting target training pictures in multiple image fields and extracting corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the expression ability of the image processing model for target objects, laying a feature foundation for the establishment of subsequent knowledge distillation indicators and focus optimization indicators. Then, by introducing equivalent distillation loss and focus loss to form a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, and the focus loss dynamically adjusts the training weight according to the sample quantity and quality difference, effectively alleviating the sample imbalance problem, so that the model maintains stable performance in the continuous learning process, avoiding the phenomenon of recognition accuracy decline of the traditional model when the data is updated. In the training process, the old knowledge is maintained and the new samples are adapted at the same time; finally, the minimization of the cross-loss function is taken as the training target, so that the image processing model can consider the knowledge transfer and sample difference of new and old tasks in each training, thereby realizing continuous learning of the target image field, not only improving the adaptability of the model in the multi-modal image fusion task, but also enhancing its generalization ability in different image fields. So that the model can maintain the stability of the fusion effect and the reliability of the recognition performance in the face of the scene of continuous expansion of multi-source data. Therefore, based on the method of the embodiments of the present application, through the synergistic effect of frequency domain feature extraction, double-loss function construction and continuous optimization training, the dynamic adaptation ability and recognition accuracy of the image processing model in the multi-modal fusion task are improved, and the risk of model performance degradation is reduced when facing challenges such as uneven data quality, unbalanced sample quantity and continuous update of modal, enhancing the stability and expansibility of the model, thereby providing a more robust and intelligent solution in the image processing process. BRIEF DESCRIPTION OF DRAWINGS

[0014] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the present application. Moreover, the same reference numerals are used throughout the same figures. In the drawings: Figure 1 is a step flow chart of a multi-feature fusion method provided by an embodiment of the present application; Figure 2 is a step flow chart of another multi-feature fusion method provided by an embodiment of the present application; Figure 3 is a step flow chart of an image processing method based on multi-feature fusion of sustainable learning provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of a multi-feature fusion device provided by an embodiment of the present application; Figure 5 Fig. 1 is a structural schematic diagram of an image processing device based on sustainable learning and multi-feature fusion provided by an embodiment of the present application; Figure 6 Fig. 2 is a block diagram of an electronic device. DETAILED DESCRIPTION

[0015] Exemplary embodiments of the present application will be described in more detail by referring to the attached drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be accurately conveyed to those skilled in the art.

[0016] Figure 1 Fig. 1 is a structural schematic diagram of an image processing device based on sustainable learning and multi-feature fusion provided by an embodiment of the present application; Step 101, determining a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and respectively determining a plurality of target frequency domain feature sub-data of each target training picture.

[0017] Each target training picture is used to record a first target object in the target image field through different picture modalities; each target frequency domain feature sub-data has a corresponding frequency domain range.

[0018] In some embodiments of the present application, in order to identify and extract representative training data in a plurality of image fields to construct a learning basis for subsequent fusion models, a plurality of target training pictures in a target image field to be learned in a plurality of image fields are determined, and a plurality of target frequency domain feature sub-data of each target training picture are respectively determined. Each target training picture is used to record a first target object in the target image field through different picture modalities; each target frequency domain feature sub-data has a corresponding frequency domain range. The frequency domain feature sub-data refers to high-frequency and low-frequency information extracted from the original image through frequency spectrum segmentation, which is used to represent the edge details and overall structure in the image, respectively. In this way, the image processing model can obtain more discriminative feature expression capability, providing a stable and diversified feature basis for subsequent knowledge distillation and focused loss function construction.

[0019] In a specific example, brain images are selected as the target image field in a medical image processing task, and multiple modal images including HR-MRI, CT, MRA and DSA are obtained therefrom. Each image can be subjected to frequency domain segmentation processing to extract multiple frequency domain feature sub-data, and the corresponding frequency domain range is labeled. In this way, the image processing model obtains a training data set containing different modal and frequency information, providing a multi-dimensional input basis for subsequent fusion model construction, thereby improving the recognition ability and adaptability of the model to brain lesion structures.

[0020] In step 102, a cross-loss function for the image processing model is established according to the feature fusion process of the multiple frequency domain feature sub-data in the image processing model.

[0021] The cross-loss function is jointly constrained by the equivalent distillation loss of the target training picture to the image processing model and the focus loss of the target image field to all image fields. The equivalent distillation loss represents the knowledge distillation loss generated in the process of using the target training picture as extracted knowledge to perform knowledge distillation on the image processing model. The focus loss represents the image sample difference between all training pictures in the target image field and all training pictures in all image fields.

[0022] In some embodiments of the present application, in order to establish an optimization mechanism that can simultaneously retain old knowledge and adapt to new samples to improve the continuous learning ability of the image processing model in a multi-modal fusion task, a cross-loss function for the image processing model is established according to the feature fusion process of the multiple frequency domain feature sub-data in the image processing model. The cross-loss function is jointly constrained by the equivalent distillation loss of the target training picture to the image processing model and the focus loss of the target image field to all image fields. The equivalent distillation loss represents the knowledge distillation loss generated in the process of using the target training picture as extracted knowledge to perform knowledge distillation on the image processing model. The focus loss represents the image sample difference between all training pictures in the target image field and all training pictures in all image fields. The equivalent distillation loss avoids forgetting the original knowledge structure during the new task training process by retaining the output distribution of the model for the old task. The focus loss adjusts the sample weight and the soft truncation parameter to guide the model to focus on the categories with fewer samples or lower quality, thereby alleviating the sample imbalance problem. In this way, the image processing model obtains a training target with dual constraints, enabling the model to consider both knowledge transfer and sample difference during the fusion process, thereby improving the stability and generalization ability of the model.

[0023] In a specific example, in a brain image fusion task, based on the extracted frequency domain feature sub-data, the training target of the image processing model is constructed. Then the output probability distribution of each target training picture in the old model and the new model is calculated first, and the equivalent distillation loss is constructed accordingly; then, according to the differences in the number and quality of samples in each image field, the weight parameters and soft truncation factors in the focus loss are set. In this way, the image processing model obtains a cross loss function containing dual constraints of knowledge preservation and sample balance, which provides a stable and adaptive optimization target for the subsequent training process, thereby improving the continuous learning ability of the model in the multi-modal image fusion task.

[0024] Step 103, training the image processing model with the minimum cross loss function as the training target to obtain an image processing model learned in the target image field.

[0025] In some embodiments of the present application, in order to guide the image processing model to complete the continuous learning of the target image field by optimizing the objective function, thereby improving the adaptability and generalization ability of the model in the multi-modal fusion task, the image processing model is trained with the minimum cross loss function as the training target to obtain an image processing model learned in the target image field. By minimizing the loss function, the model can dynamically adjust the parameters during training, so that the fusion result not only retains the original knowledge structure, but also adapts to the differences in new modalities and samples. Finally, the image processing model obtains the continuous learning ability for the target image field, and can maintain the stability of the fusion effect and the reliability of the recognition performance in the scene of continuously expanding multi-source data.

[0026] In a specific example, in a brain multi-modal image fusion task, the image processing model is iteratively trained using the constructed cross loss function as the training target. During the training process, the model continuously adjusts the parameters to minimize the sum of the distillation loss and the focus loss according to the frequency domain features of the modal images such as HR-MRI, CT, MRA and DSA. Finally, the model successfully learns the feature distribution rule of the brain image field, and shows stronger recognition ability and adaptability in subsequent fusion tasks, which can effectively cope with the challenges of uneven sample quality and continuous update of modalities.

[0027] In summary, in the embodiment of the present application, by selecting target training pictures in multiple image fields and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the expression ability of the image processing model for target objects, laying a feature foundation for the establishment of subsequent knowledge distillation indicators and focus optimization indicators. Then, by introducing equivalent distillation loss and focus loss to form a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focus loss dynamically adjusts the training weight according to the sample quantity and quality difference, effectively alleviating the sample imbalance problem, so that the model maintains stable performance in the continuous learning process, avoiding the phenomenon of recognition accuracy decline of traditional models when data is updated. In the training process, the old knowledge is maintained and the new samples are adapted at the same time; finally, the minimization of the cross-loss function is taken as the training target, so that the image processing model can consider the knowledge transfer and sample difference of new and old tasks in each training round, thereby realizing continuous learning of the target image field, not only improving the adaptability of the model in the multi-modal image fusion task, but also enhancing its generalization ability in different image fields. So that the model can maintain the stability of the fusion effect and the reliability of the recognition performance in the face of the scene of continuous expansion of multi-source data. Therefore, based on the method of the embodiment of the present application, through the synergistic effect of frequency domain feature extraction, double-loss function construction and continuous optimization training, the dynamic adaptation ability and recognition accuracy of the image processing model in the multi-modal fusion task are improved, and the risk of model performance degradation is reduced when facing challenges such as uneven data quality, unbalanced sample quantity and continuous update of modal, enhancing the stability and expansibility of the model, thereby providing a more robust and intelligent solution in the image processing process.

[0028] Figure 2 Another method for sustainable learning and multi-feature fusion provided by the present embodiment, specifically comprising the following steps: Step 201, determining a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and determining a plurality of target frequency domain feature sub-data for each target training picture.

[0029] Wherein, each target training picture is used to record a first target object in the target image field through different picture modalities; each target frequency domain feature sub-data has a corresponding frequency domain range.

[0030] The method shown in this step has been described in step 101, which will not be repeated here.

[0031] Optionally, in order to determine the plurality of target frequency domain feature sub-data of each target training picture, step 201 includes the following sub-steps: Sub-step 2011 , obtaining target frequency domain feature data corresponding to each target training picture according to the convolution of each target training picture.

[0032] In some embodiments of the present application, in order to extract frequency domain feature data from the original image and provide a basis for subsequent spectrum segmentation and feature sub-data generation, the target frequency domain feature data corresponding to each target training picture will be obtained based on the convolution of each target training picture. Convolution operation is a common image processing method. By sliding the convolution kernel on the image and performing weighted summation, the local structural information in the image can be extracted. In the present application, the convolution operation is used to preliminarily extract the frequency domain feature data of the image, usually using a convolution kernel of size 3×3 to capture basic features such as edges and textures of the image. In this way, the system obtains the frequency domain feature data of each target training picture, which provides operational input data for the subsequent spectrum segmentation using a Gaussian filter, thereby improving the accuracy and efficiency of feature extraction.

[0033] In a specific example, when processing brain HR-MRI images, a convolution operation is performed on each image using a 3×3 convolution kernel to extract primary frequency domain features from the image. This operation performs a weighted summation within each pixel neighborhood of the image to generate a set of frequency domain feature data that reflects the local changes in the image at different spatial locations. In this way, each HR-MRI image is converted into a set of frequency domain feature data, laying the foundation for subsequent spectrum segmentation and high and low frequency information extraction, thereby improving the feature expression ability of the image processing model in brain structure recognition tasks. Specifically, the system can generate the high-frequency component F 0H The high-frequency feature extraction unit is composed of multiple residual-in-residual dense blocks (RRDB) and adaptive feature aggregation modules to enhance the expression of high-frequency details. 0L It is introduced into the low-frequency feature extraction unit, which only contains a small number of RRDB blocks and a self-adjusting feature integration module to control computing resources and avoid excessive processing of low-frequency information.

[0034] In sub-step 2012, spectrum segmentation is performed on each target frequency domain feature data according to a plurality of preset Gaussian filters to obtain a plurality of target frequency domain feature sub-data corresponding to each target training image.

[0035] Among them, each Gaussian filter corresponds to a different Gaussian kernel.

[0036] In some embodiments of the present application, in order to further refine the frequency domain feature data into sub-data of multiple frequency bands, to enhance the recognition ability of the image processing model for different frequency information, a plurality of Gaussian filters are used to perform frequency spectrum segmentation on each target frequency domain feature data respectively, to obtain a plurality of target frequency domain feature sub-data corresponding to each target training picture respectively. Each Gaussian filter corresponds to a different Gaussian kernel. Gaussian filter is a tool commonly used for image smoothing and frequency separation, and its core parameters are the size and standard deviation of the Gaussian kernel, which determine the response range of the filter to image frequency. In the present application, by setting a plurality of filters with different Gaussian kernel sizes, multi-level segmentation of frequency domain feature data can be achieved, so as to extract high-frequency and low-frequency components respectively. In this way, the system obtains a plurality of frequency domain feature sub-data, each sub-data corresponding to the feature expression of the image in a specific frequency range, providing a more discriminative input basis for subsequent feature fusion and loss function construction.

[0037] In a specific example, when processing the frequency domain feature data of brain CT images, three Gaussian filters are applied for frequency spectrum segmentation respectively, and the Gaussian kernel sizes of these filters are 3x3, 5x5 and 7x7 respectively. Each filter performs convolution operation on the frequency domain feature data, and extracts high-frequency, medium-frequency and low-frequency information of the image respectively. In this way, each CT image is decomposed into three frequency domain feature sub-data, which respectively reflect the structural details and texture features of the image at different frequency levels, providing multi-scale input data for the subsequent feature fusion model, thereby improving the frequency sensitivity and fusion accuracy of the model in the brain lesion recognition task.

[0038] Optionally, the sub-step 2012 comprises the following sub-steps: Sub-step 20121, performing frequency spectrum filtering on the target frequency domain feature data by a first Gaussian filter in the plurality of Gaussian filters to obtain a first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and performing frequency spectrum filtering on the target frequency domain feature data by a plurality of second Gaussian filters in the plurality of Gaussian filters to obtain a second target frequency domain feature decomposition sub-data of the target frequency domain feature data.

[0039] Among them, the first Gaussian filter is the Gaussian filter corresponding to the largest Gaussian kernel among the Gaussian kernels corresponding to the plurality of Gaussian filters; the second Gaussian filter is the Gaussian filter in the plurality of Gaussian filters other than the first Gaussian filter.

[0040] In some embodiments of the present application, in order to perform multi-scale spectral filtering on the target frequency domain feature data to separate image features of different frequency levels and provide a basis for subsequent construction of frequency domain feature sub-data, the target frequency domain feature data is filtered by a first Gaussian filter in a plurality of Gaussian filters to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and the target frequency domain feature data is filtered by a plurality of second Gaussian filters in the plurality of Gaussian filters to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data. Among them, the first Gaussian filter is a Gaussian filter corresponding to the largest Gaussian kernel in the Gaussian kernels corresponding to the plurality of Gaussian filters; the second Gaussian filter is a Gaussian filter in the plurality of Gaussian filters other than the first Gaussian filter. A larger Gaussian kernel tends to retain low-frequency information and is suitable for capturing the overall structure of an image; while a smaller Gaussian kernel is more sensitive to high-frequency information and is suitable for extracting edge and detail features in an image. In this way, the system obtains a plurality of frequency level decomposition sub-data, which provides a hierarchical basis for subsequent difference calculation and frequency domain feature sub-data generation, thereby improving the response capability of the image processing model to frequency changes and the resolution of feature expression.

[0041] In one specific example, when processing frequency domain feature data of a brain CT image, three Gaussian filters are applied for spectral filtering respectively, wherein the Gaussian kernel size of the first Gaussian filter is 7x7, corresponding to the low-frequency component of the image; the Gaussian kernel size of the second Gaussian filter is 5x5 and 3x3, corresponding to the medium-frequency and high-frequency components. The first target frequency domain feature decomposition sub-data is obtained by using the first Gaussian filter to process the frequency domain feature data; then the second target frequency domain feature decomposition sub-data is obtained by using two second Gaussian filters respectively. Finally, the system successfully extracts a plurality of frequency level image features, which provides a multi-scale input basis for subsequent construction of frequency domain feature sub-data and execution of feature fusion operation, thereby improving the frequency resolution capability and fusion accuracy of the model in the brain structure recognition task.

[0042] Sub-step 20122, determining the difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and determining the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter as the target frequency domain feature sub-data corresponding to each second Gaussian filter.

[0043] In some embodiments of the present application, in order to extract the frequency domain feature sub-data corresponding to each Gaussian filter from the spectral decomposition result, to realize feature separation and expression on the frequency level, the difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data is determined as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and the second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter is determined as the target frequency domain feature sub-data corresponding to each second Gaussian filter respectively. The difference operation is used to highlight the part of the original frequency domain feature data that is not retained by the first Gaussian filter, thereby enhancing the expression ability of high frequency details; while directly using the decomposition result of the second Gaussian filter, the feature response of the medium and high frequency components is retained. The role of the Gaussian filter in the frequency domain is to weight the inhibition or enhancement of different frequency components, so through the combination of the difference and the decomposition result, multiple frequency level target frequency domain feature sub-data can be constructed. Finally, the system obtains a set of frequency domain feature sub-data corresponding to each Gaussian filter, providing multi-scale frequency domain input for subsequent feature fusion, classification or recognition tasks, which helps to improve the sensitivity and expression ability of the model to image frequency changes.

[0044] In a specific example, when processing the frequency domain feature data of a brain CT image, the first target frequency domain feature decomposition sub-data (generated by a 7x7 Gaussian kernel filter) and two second target frequency domain feature decomposition sub-data (generated by 5x5 and 3x3 Gaussian kernel filters respectively) have been obtained. The difference between the original frequency domain feature data and the first decomposition sub-data is calculated to obtain the frequency domain feature sub-data corresponding to the first Gaussian filter; at the same time, the two second decomposition sub-data are respectively taken as the frequency domain feature sub-data corresponding to the respective Gaussian filters. Finally, the system successfully constructs three frequency level target frequency domain feature sub-data, corresponding to low frequency, medium frequency and high frequency features respectively, providing multi-scale frequency domain feature support for subsequent image structure recognition and lesion detection.

[0045] Optionally, the cross-loss function It is represented by the following formula: , Wherein, represents the equivalent distillation loss, represents the focus loss; , respectively, represent the prediction probability of the old model and the new model that the sample x belongs to the class k, which is smoothed by the temperature parameter T: Wherein, respectively represent the output probability of the old model and the new model on the sample x, , respectively represent the original output value of the new model and the old model to the class k; , wherein, is the sample weight of the target training picture, is the soft truncation parameter (the soft truncation parameter is used to control the degree of attention to difficult samples, and the greater the value is, the more biased to difficult samples), is the prediction probability of the model that the sample x belongs to the class k, , respectively represent the training sample set of the old class and the new class.

[0046] The cross-loss function is initially designed to achieve knowledge transfer and class focusing during model training, thereby improving the generalization ability and class discrimination ability of the model in the mixed training scene of new and old samples. The design principle integrates distillation learning and focusing mechanism: the equivalent distillation loss term is used to maintain the inheritance of the knowledge of the old model by the new model, and the focusing loss term is used to enhance the attention of the model to difficult samples, thereby improving the classification accuracy while maintaining stability.

[0047] Optionally, the sample weight of the target training picture is positively correlated with the peak signal-to-noise ratio (PSNR) and / or the structural similarity index measure (SSIM) value of the target training picture.

[0048] The sample weight is initially designed to introduce prior information of image quality into the loss function, so that the model can dynamically adjust the learning intensity according to the quality difference of the image during the training process, thereby improving the feature extraction ability of high-quality samples. The principle is to use the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM) as indicators to measure the quality of the image, and establish a positive correlation between the sample weight coefficient and the image quality, so that the sample with higher image quality has greater weight in the loss function and has stronger influence on training. In this way, the model can pay more attention to the effective feature information carried by high-quality images during the training process, thereby improving the overall training effect and the generalization ability of the model.

[0049] Optionally, the PSNR of the target training picture is calculated according to the following formula: , wherein, MAX is the theoretical maximum value of the target training picture, H and W represent the height and width of the target training picture, respectively, , respectively represent two different target training pictures; The SSIM of the target training picture is calculated according to the following formula: , in, , , , Represent the preset weight parameters respectively, and 、 and are the mean and standard deviation of the target training image x and all training images y, respectively. is the covariance of x and y, 、 、 are the preset adjustment parameters.

[0050] The original intention of establishing the image quality evaluation formula was to quantitatively evaluate the quality of the target training images during the training process, thereby providing a basis for setting sample weights. The principle is to use the peak signal-to-noise ratio (PSNR) and the structural similarity score (SSIM) to measure the image reconstruction accuracy and structural fidelity respectively. PSNR mainly reflects the error size of the image at the pixel level and is suitable for evaluating image clarity and compression quality; SSIM is based on the brightness ( ), contrast ( ) and structure ( ) comprehensively evaluates the similarity between images along three dimensions, more closely resembling human visual perception. The combined use of these two metrics comprehensively reflects the quality of the target training image and adjusts its weight in the loss function accordingly.

[0051] Step 202: establishing a cross-loss function for the image processing model based on a feature fusion process of a plurality of frequency domain feature sub-data in the image processing model.

[0052] Among them, the cross loss function is jointly constrained by the equivalent distillation loss of the target training image to the image processing model, and the focus loss of the target image field to the entire image field; the equivalent distillation loss is used to characterize the knowledge distillation loss generated in the process of using the target training image as the extracted knowledge and performing knowledge distillation on the image processing model; the focus loss is used to characterize the image sample differences between all training images in the target image field and the training images in all image fields.

[0053] The method shown in this step has been described in step 102 and will not be repeated here.

[0054] Optionally, the multiple frequency domain feature sub-data respectively correspond to different Gaussian kernels, and the feature fusion process of the multiple frequency domain feature sub-data in the image processing model is specifically implemented by the following process: Sub-step 2020, taking the first frequency domain feature sub-data in the plurality of frequency domain feature sub-data as the initial value of the fusion process, adaptively fusing the second frequency domain feature sub-data in the plurality of frequency domain feature sub-data to the initial value.

[0055] Among them, the first Gaussian kernel corresponding to the first frequency domain feature sub-data is the largest Gaussian kernel in all Gaussian kernels.

[0056] In some embodiments of the present application, in order to realize the fusion of multi-frequency hierarchical features in the image processing model, to enhance the comprehensive expression ability of the model to the image structure, the first frequency domain feature sub-data in the plurality of frequency domain feature sub-data is taken as the initial value of the fusion process, and the second frequency domain feature sub-data in the plurality of frequency domain feature sub-data is adaptively fused to the initial value. Among them, the first Gaussian kernel corresponding to the first frequency domain feature sub-data is the largest Gaussian kernel in all Gaussian kernels. Adaptive feature fusion is a processing method that dynamically adjusts the fusion strategy according to the correlation and weight between features, which is usually realized through attention mechanism or weighted average to retain key features and suppress redundant information. The largest Gaussian kernel corresponding to the first frequency domain feature sub-data mainly retains the low-frequency components of the image in the frequency domain, representing the overall structural information of the image; while the second frequency domain feature sub-data contains medium-high frequency components, representing the details and edge information of the image. Finally, the system obtains the fused frequency domain feature data, which enhances the detail expression ability on the basis of maintaining the overall structure of the image, providing more discriminative feature input for subsequent image recognition, segmentation or classification tasks.

[0057] In a specific example, when processing the frequency domain feature sub-data of the brain CT image, first, the frequency domain feature sub-data generated by the 7x7 Gaussian kernel is selected as the initial value of fusion, which mainly contains the low-frequency information of the brain structure. Then, the frequency domain feature sub-data generated by the 5x5 and 3x3 Gaussian kernels are respectively taken as the second frequency domain feature sub-data, and the initial value is fused by the adaptive fusion module. The fusion module dynamically adjusts the fusion weight according to the response intensity and spatial distribution of each frequency domain feature sub-data. Finally, the system obtains the fused frequency domain feature data, which enhances the expression ability of lesion edge and texture details while retaining the overall outline of the brain, providing more hierarchical feature support for the subsequent lesion detection model.

[0058] Step 203, taking minimizing the cross-loss function as the training target, training the image processing model to obtain the image processing model learned in the target image field.

[0059] The method shown in this step has been described in step 103, which will not be repeated here.

[0060] In summary, in the embodiment of the present application, by selecting target training pictures in multiple image fields and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the expression ability of the image processing model for target objects, laying a feature foundation for the establishment of subsequent knowledge distillation indicators and focus optimization indicators; then by introducing equivalent distillation loss and focus loss to form a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focus loss dynamically adjusts the training weight according to the sample quantity and quality difference, effectively alleviating the sample imbalance problem, so that the model maintains stable performance in the continuous learning process, avoiding the phenomenon of recognition accuracy decline of traditional models when data is updated, and realizing the preservation of old knowledge and adaptation to new samples in the training process; finally, minimizing the cross-loss function is taken as the training target, so that the image processing model can consider the knowledge transfer and sample difference between new and old tasks in each training round, thereby realizing continuous learning of the target image field, not only improving the adaptability of the model in the multi-modal image fusion task, but also enhancing its generalization ability between different image fields, so that the model can maintain the stability of the fusion effect and the reliability of the recognition performance in the face of the scene of continuous expansion of multi-source data. Therefore, based on the method of the embodiment of the present application, through the synergistic effect of frequency domain feature extraction, double-loss function construction and continuous optimization training, the dynamic adaptation ability and recognition accuracy of the image processing model in the multi-modal fusion task are improved, and in the face of challenges such as uneven data quality, unbalanced sample quantity and continuous update of modal, the risk of model performance degradation is reduced, the stability and expansibility of the model are enhanced, thereby providing a more robust and intelligent solution in the image processing process.

[0061] Figure 3 is an image processing method for multi-feature fusion based on sustainable learning provided by the embodiment of the present application, which specifically includes the following steps: Step 301, obtaining a plurality of target pictures to be processed.

[0062] Each target picture is used to record a second target object through different picture modalities.

[0063] In some embodiments of the present application, in order to provide input data for the image processing model to be processed in subsequent steps to complete the feature fusion and recognition task, a plurality of target pictures to be processed are obtained, wherein each target picture is used to record a second target object by a different picture modality. The target picture refers to image data collected in an actual application scenario, which usually includes HR-MRI, CT, MRA, and DSA modal images. Each modality image records the structural or functional features of the second target object under different imaging mechanisms, and has information complementarity. In this way, the image processing model obtains multi-modal input data, providing a multi-dimensional image basis for subsequent feature fusion processing, which helps to improve the recognition accuracy and processing robustness of the model for the target object.

[0064] In a specific example, when a user evaluates brain lesions before surgery, the user collects image data of four modalities of HR-MRI, CT, MRA, and DSA to record the spatial structure, blood flow state, and lesion distribution of the brain blood vessels. Each image corresponds to the same second target object, i.e., a specific lesion area of the patient's brain. The user inputs these images as target pictures into the image processing system, the system identifies the source and corresponding relationship of each modality image, and completes data preprocessing. In this way, the image processing model obtains multi-modal target picture input, providing rich and complementary image information for the subsequent fusion model, thereby improving the accuracy and adaptability of the model in the brain lesion recognition task.

[0065] In step 302, a plurality of target pictures are input as input data into an image processing model trained according to the disclosed sustainable learning multi-feature fusion method, to obtain a feature fusion processing image of the plurality of target pictures.

[0066] In some embodiments of the present application, in order to utilize the trained image processing model to perform feature fusion processing on a plurality of target pictures to generate an output image with multi-modal information, a plurality of target pictures are input as input data into an image processing model trained according to the disclosed sustainable learning multi-feature fusion method, to obtain a feature fusion processing image of the plurality of target pictures. The image processing model is trained by minimizing the cross-loss function, and the model has the ability to extract and fuse frequency domain features of different modality images, and can maintain the memory of old knowledge and adapt to new samples in the process of continuous learning. The feature fusion processing image is an image generated by fusing a plurality of modality images at the frequency domain level, which retains the structural information and detailed features of each modality image. Finally, the system obtains the fused image output, which provides image data with higher information density and expression ability for subsequent recognition, analysis, or auxiliary decision-making.

[0067] In a specific example, a user inputs the collected brain HR-MRI, CT, MRA and DSA images as target pictures into the image processing system during intraoperative navigation. The system calls the image processing model trained based on the method of the present application to perform frequency domain feature extraction and fusion operation on these images. The model uses the adaptive feature fusion mechanism to complete the fusion processing according to the frequency domain feature sub-data of each modality image, and outputs the processed image based on the feature fusion. In this way, the user obtains a fusion processed image, which simultaneously presents the spatial distribution of brain structure, blood vessel direction and lesion details, providing a more comprehensive visual reference for intraoperative positioning and intervention.

[0068] It should be emphasized that in the actual operation of the model, the feature fusion process is also performed in the same way as described in sub-step 2020: the system first takes the low-frequency feature sub-data as the starting point for fusion, and then introduces the medium-high frequency feature sub-data in turn, and completes the feature fusion of multiple frequency levels through the adaptive mechanism, thereby enhancing the detail expression while maintaining the overall structure of the image, realizing the effective integration and utilization of the frequency domain features in the model.

[0069] For example, the system can use a hierarchical structure for feature extraction and fusion: the first layer has 64 input channels, indicating that the initial input data contains 64-dimensional feature information in the frequency domain or spatial domain; after convolution or other processing, the output channel number of this layer is 32, that is, the feature dimension is compressed to 32 dimensions to extract more discriminative features. The input of the second layer is composed of the initial input and the output of the first layer, forming a fusion input structure, which is used to enhance the joint expression ability of the model for original features and intermediate features.

[0070] In summary, in the embodiment of the present application, by selecting target training pictures in multiple image fields and extracting corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the expression ability of the image processing model for target objects, laying a feature foundation for the establishment of subsequent knowledge distillation indicators and focus optimization indicators; then by introducing equivalent distillation loss and focus loss to form a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focus loss dynamically adjusts the training weight according to the sample quantity and quality difference, effectively alleviating the sample imbalance problem, so that the model maintains stable performance in the continuous learning process, avoiding the phenomenon of recognition accuracy decline of the traditional model when the data is updated, and realizing the preservation of old knowledge and adaptation to new samples in the training process; finally, minimizing the cross-loss function is taken as the training target, so that the image processing model can balance the knowledge transfer and sample difference between new and old tasks in each training round, thereby realizing continuous learning of the target image field, not only improving the adaptability of the model in the multi-modal image fusion task, but also enhancing its generalization ability between different image fields, so that the model can maintain the stability of the fusion effect and the reliability of the recognition performance in the face of the scene of continuous expansion of multi-source data. Therefore, based on the method of the embodiment of the present application, through the synergistic effect of frequency domain feature extraction, double-loss function construction and continuous optimization training, the dynamic adaptation ability and recognition accuracy of the image processing model in the multi-modal fusion task are improved, and in the face of challenges such as uneven data quality, unbalanced sample quantity and continuous update of modal, the risk of performance degradation of the model is reduced, the stability and expansibility of the model are enhanced, thereby providing a more robust and intelligent solution in the image processing process.

[0071] As shown in Figure 4 The embodiment of the present application also discloses a multi-feature fusion device 40 capable of sustainable learning, comprising: A feature extraction module 401 is configured to determine a plurality of target training pictures in a target image field to be learned in a plurality of image fields, and determine a plurality of target frequency domain feature sub-data of each target training picture respectively; each target training picture is used to record a first target object in the target image field through different picture modalities; each target frequency domain feature sub-data has a corresponding frequency domain range; The function construction module 402 is configured to construct a cross loss function of the image processing model according to a feature fusion process performed by the plurality of frequency domain feature sub-data in the image processing model; the cross loss function is jointly constrained by an equivalent distillation loss of the target training picture to the image processing model and a focus loss of the target image field to all image fields; the equivalent distillation loss is used to represent a knowledge distillation loss generated in a process of distilling knowledge from the target training picture to the image processing model; and the focus loss is used to represent an image sample difference of all training pictures in the target image field to training pictures in all image fields. The model training module 403 is configured to train the image processing model with a training target of minimizing the cross loss function, so as to obtain the image processing model learned in the target image field.

[0072] Optionally, the feature extraction module 401 comprises: The convolution sub-module is configured to obtain target frequency domain feature data corresponding to each target training picture according to convolution of each target training picture. The segmentation sub-module is configured to perform frequency spectrum segmentation on each target frequency domain feature data according to a plurality of preset Gaussian filters, so as to obtain a plurality of target frequency domain feature sub-data corresponding to each target training picture; each Gaussian filter corresponds to a different Gaussian kernel.

[0073] Optionally, the segmentation sub-module comprises: The filtering unit is configured to perform frequency spectrum filtering on the target frequency domain feature data through a first Gaussian filter in the plurality of Gaussian filters, so as to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and perform frequency spectrum filtering on the target frequency domain feature data through a plurality of second Gaussian filters in the plurality of Gaussian filters, so as to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data; the first Gaussian filter is a Gaussian filter corresponding to the largest Gaussian kernel in the Gaussian kernels corresponding to the plurality of Gaussian filters; the second Gaussian filter is a Gaussian filter other than the first Gaussian filter in the plurality of Gaussian filters. The determination unit is configured to determine a difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and determine each second target frequency domain feature decomposition sub-data corresponding to each second Gaussian filter as the target frequency domain feature sub-data corresponding to each second Gaussian filter.

[0074] As shown in Figure 5 The application further discloses an image processing device 50 based on sustainable learning and multi-feature fusion, which comprises: The picture acquisition module 501 is configured to acquire a plurality of target pictures to be processed, each of which is used to record a second target object by a different picture modality; The picture processing module 502 is configured to input the plurality of target pictures as input data into an image processing model trained by the sustainable learning multi-feature fusion method disclosed in the embodiments of the present application to obtain a feature fusion processed image of the plurality of target pictures.

[0075] In summary, in the embodiments of the present application, by selecting target training pictures in multiple image fields and extracting their corresponding frequency domain feature sub-data, the differences and complementarities of different modal images in the frequency domain are fully captured, thereby improving the expression ability of the image processing model for the target object, laying a feature foundation for the establishment of subsequent knowledge distillation indicators and focus optimization indicators; then by introducing equivalent distillation loss and focus loss to form a cross-loss function: the distillation loss ensures that the model does not forget the original knowledge structure when introducing new modal images, while the focus loss dynamically adjusts the training weight according to the sample quantity and quality difference, effectively alleviating the sample imbalance problem, so that the model maintains stable performance in the process of continuous learning, avoiding the phenomenon of recognition accuracy decline of traditional models when data is updated, and realizing the preservation of old knowledge and adaptation to new samples in the training process; finally, minimizing the cross-loss function is taken as the training target, so that the image processing model can consider the knowledge transfer and sample difference between new and old tasks in each training round, thereby realizing the continuous learning of the target image field, not only improving the adaptability of the model in the multi-modal image fusion task, but also enhancing its generalization ability in different image fields, so that the model can maintain the stability of the fusion effect and the reliability of the recognition performance in the face of the scene of continuous expansion of multi-source data. Therefore, based on the method of the embodiments of the present application, through the synergistic effect of frequency domain feature extraction, double-loss function construction and continuous optimization training, the dynamic adaptability and recognition accuracy of the image processing model in the multi-modal fusion task are improved, and in the face of challenges such as uneven data quality, uneven sample quantity and continuous update of modal, the risk of model performance degradation is reduced, the stability and expansibility of the model are enhanced, thereby providing a more robust and intelligent solution in the image processing process.

[0076] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the above-mentioned sustainable learning multi-feature fusion method and the image processing method based on the sustainable learning multi-feature fusion, and the same technical effects can be achieved. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0077] Figure 6 is a block diagram of an electronic device 700 provided by the embodiment of the present application. For example, the electronic device 700 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0078] Referring to Figure 6 , the electronic device 700 can include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0079] The processing component 702 usually controls overall operations of the electronic device 700, such as operations associated with displaying, making phone calls, data communications, camera operations and recording operations. The processing component 702 can include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned sustainable learning multi-feature fusion method and the image processing method based on the sustainable learning multi-feature fusion. In addition, the processing component 702 can include one or more modules to facilitate interaction between the processing component 702 and other components. For example, the processing component 702 can include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.

[0080] The memory 704 is used to store various types of data to support operations of the electronic device 700. Examples of the data include instructions for any application or method operating on the electronic device 700, contact data, phonebook data, messages, pictures, multimedia, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0081] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0082] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the demarcation of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a capture mode or a multimedia mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0083] The audio component 710 is used to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC) that receives external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0084] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0085] The sensor component 714 includes one or more sensors for providing status assessments for various aspects of the electronic device 700. For example, the sensor component 714 can detect an open / closed position of the electronic device 700, relative positioning of components, such as a display and a keypad of the electronic device 700, changes in position of the electronic device 700 or a component of the electronic device 700, presence or absence of user contact with the electronic device 700, orientation or acceleration / deceleration / g-force and temperature changes of the electronic device 700. The sensor component 714 can include an optical sensor for detecting ambient light, a proximity sensor configured to detect proximity of an object without any physical touch, or a CMOS or CCD image sensors for use in imaging applications. In some embodiments, the sensor component 714 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0086] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, a cellular network (e.g., 2G, 3G, 4G or 7G), or a combination thereof. In an example embodiment, the communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 716 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques and other techniques.

[0087] In an example embodiment, the electronic device 700 can be implemented using one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements to implement the method for multi-feature fusion based on sustainable learning and the image processing method based on multi-feature fusion based on sustainable learning.

[0088] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 704 including instructions, is also provided. The instructions can be executed by the processor 720 of the electronic device 700 to implement the method for multi-feature fusion based on sustainable learning and the image processing method based on multi-feature fusion based on sustainable learning. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0089] In exemplary embodiments, the electronic device 700 can also be provided as a server, including a processing component 702 further comprising one or more processors 720, and memory resources represented by the memory 704 for storing instructions, such as application programs, executable by the processing component 702. The application programs stored in the memory 704 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 702 is configured to execute the instructions to perform the method for sustainable learning based multi-feature fusion and the method for image processing based on sustainable learning based multi-feature fusion provided by the embodiments of the present application.

[0090] The electronic device 700 can also include a power component 706 configured to perform power management of the electronic device 700, a wired or wireless network interface 718 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 712. The electronic device 700 can operate based on an operating system stored in the memory 704, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0091] The embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the method for sustainable learning based multi-feature fusion and the method for image processing based on sustainable learning based multi-feature fusion.

[0092] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application indicated by the following claims.

[0093] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the attached drawings, and that various changes in form and detail can be made therein without departing from the scope thereof. The scope of the application should only be limited by the appended claims.

[0094] Each of the various embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between various embodiments can be mutually referred to.

[0095] It is readily apparent to those skilled in the art that any combination of the various embodiments described above is possible and thus any combination of the various embodiments described above is an embodiment of the present application, but due to the page limit, the detailed description of the specification will not be described here.

[0096] The sustainable learning based multi-feature fusion method and the image processing method based on the sustainable learning based multi-feature fusion provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can also be used with the teachings herein. The structure for a system having the aspects of the present application is apparent from the above description. Furthermore, the present application is not intended to be limited to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present application described herein, and any references below to specific languages are provided for disclosure of enablement of the best mode of the present application.

[0097] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0098] Similarly, it is to be understood that the embodiments of the present application can be readily combined with one another, and that features of one embodiment can be interchanged with features of another embodiment. It is also to be understood that the techniques of the present application can be applied to any embodiment of the present application, and the present application can be implemented in hardware, software, or a combination thereof.

[0099] As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, can readily be utilized as the basis for the designing of other structures, methods, and systems for carrying out the several purposes of the present application. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the present application.

[0100] Furthermore, those skilled in the art will recognize that the embodiments described herein, while including certain features, are meant to be illustrative of various embodiments of the application and are not meant to limit the scope of the application. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0101] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components of the sustainable learning based multi-feature fusion method and the image processing method based on sustainable learning based multi-feature fusion according to the embodiments of the present application. The present application can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer readable medium or can have one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0102] In yet another embodiment of the present application, a computer program product containing instructions which, when executed on a computer, cause the computer to carry out the sustainable learning based multi-feature fusion method and the image processing method based on sustainable learning based multi-feature fusion according to the embodiments of the present application is also provided.

[0103] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0104] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0105] It should be noted that, for the sake of simplicity, the method embodiments of the present application are described as a series of action combinations. However, those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0106] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the embodiments of the system or the device, since they are basically similar to the embodiments of the method, the description is relatively simple, and the relevant parts can be referred to the part of the description of the embodiments of the method.

[0107] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-feature fusion method for sustainable learning, characterized by: include: Determining a plurality of target training images in a target image domain to be learned in the plurality of image domains, and respectively determining a plurality of target frequency domain feature sub-data for each of the target training images; Each of the target training images is used to record the first target object in the target image field through different image modalities; each of the target frequency domain feature sub-data has a corresponding frequency domain range; A cross-loss function for the image processing model is established based on a feature fusion process of the plurality of frequency domain feature sub-data in the image processing model; the cross-loss function is jointly constrained by an equivalent distillation loss of the target training image with respect to the image processing model, and a focusing loss of the target image domain with respect to all image domains; the equivalent distillation loss is used to characterize the knowledge distillation loss generated in the process of performing knowledge distillation on the image processing model using the target training image as extracted knowledge; the focusing loss is used to characterize the image sample differences between all training images in the target image domain and all training images in the image domain; The image processing model is trained with minimizing the cross loss function as a training goal to obtain the image processing model that has learned the target image domain.

2. The multi-feature fusion method for sustainable learning according to claim 1, characterized in that: The determining of a plurality of target frequency domain feature sub-data of each target training picture includes: Obtain target frequency domain feature data corresponding to each target training picture according to the convolution of each target training picture; Spectral segmentation is performed on each of the target frequency domain feature data according to a plurality of preset Gaussian filters to obtain a plurality of target frequency domain feature sub-data corresponding to each of the target training images; each of the Gaussian filters corresponds to a different Gaussian kernel.

3. The multi-feature fusion method for sustainable learning according to claim 2, characterized in that: The performing spectrum segmentation on each of the target frequency domain feature data according to the preset multiple Gaussian filters includes: spectrally filtering the target frequency domain feature data through a first Gaussian filter among the plurality of Gaussian filters to obtain first target frequency domain feature decomposition sub-data of the target frequency domain feature data, and spectrally filtering the target frequency domain feature data through a plurality of second Gaussian filters among the plurality of Gaussian filters to obtain second target frequency domain feature decomposition sub-data of the target frequency domain feature data; the first Gaussian filter is a Gaussian filter corresponding to the largest Gaussian kernel among the Gaussian kernels respectively corresponding to the plurality of Gaussian filters; the second Gaussian filter is a Gaussian filter other than the first Gaussian filter among the plurality of Gaussian filters; The difference between the target frequency domain feature data and the first target frequency domain feature decomposition sub-data is determined as the target frequency domain feature sub-data corresponding to the first Gaussian filter, and the second target frequency domain feature decomposition sub-data corresponding to each of the second Gaussian filters is determined as the target frequency domain feature sub-data corresponding to each of the second Gaussian filters.

4. The multi-feature fusion method for sustainable learning according to claim 1, characterized in that: The plurality of frequency domain feature sub-data respectively correspond to different Gaussian kernels, and the feature fusion process of the plurality of frequency domain feature sub-data in the image processing model includes: The first frequency domain feature sub-data among the multiple frequency domain feature sub-data is used as the initial value of the fusion process, and the second frequency domain feature sub-data among the multiple frequency domain feature sub-data is adaptively fused with the initial value; the first Gaussian kernel corresponding to the first frequency domain feature sub-data is the largest Gaussian kernel among all the Gaussian kernels.

5. An image processing method based on multi-feature fusion of sustainable learning, characterized in that: include: Get multiple target images to be processed; Each of the target images is used to record a second target object through a different image modality; The plurality of target images are used as input data and input into an image processing model trained by the multi-feature fusion method for sustainable learning according to any one of claims 1 to 4 to obtain feature fusion processed images of the plurality of target images.

6. A multi-feature fusion device for sustainable learning, characterized in that: include: a feature extraction module, configured to determine a plurality of target training images in a target image domain to be learned from the plurality of image domains, and respectively determine a plurality of target frequency domain feature sub-data for each of the target training images; Each of the target training images is used to record the first target object in the target image field through different image modalities; each of the target frequency domain feature sub-data has a corresponding frequency domain range; A function construction module is used to establish a cross-loss function for the image processing model based on the feature fusion process of the multiple frequency domain feature sub-data in the image processing model; the cross-loss function is jointly constrained by the equivalent distillation loss of the target training image to the image processing model and the focusing loss of the target image domain to all image domains; the equivalent distillation loss is used to characterize the knowledge distillation loss generated in the process of performing knowledge distillation on the image processing model using the target training image as extracted knowledge; the focusing loss is used to characterize the image sample differences between all training images in the target image domain and all training images in the image domain; A model training module is used to train the image processing model with minimizing the cross loss function as the training goal, so as to obtain the image processing model that has learned the target image domain.

7. An image processing device based on multi-feature fusion of sustainable learning, characterized in that: include: An image acquisition module is used to acquire multiple target images to be processed; Each of the target images is used to record a second target object through a different image modality; An image processing module is used to input the multiple target images as input data into an image processing model trained by the multi-feature fusion method of sustainable learning according to any one of claims 1 to 4, so as to obtain feature fusion processed images of the multiple target images.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 5 when executed by the processor.

10. A computer program product, characterized in that The computer program product stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images