Image processing model training method and device, electronic equipment, storage medium and program product
By using multi-channel encoding and decorrelation operations, the correlation of image features is calculated, and image samples are fused and decoded, which solves the problem of single training dimension of image processing models and improves the performance of image processing models.
Patent Information
- Application Number
- CN202510590984.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-10
AI Technical Summary
The training dimension of existing image processing models is limited, resulting in poor image processing performance.
By performing multi-channel encoding on image samples, extracting multiple image features, and performing decorrelation operations, calculating the correlation between features, determining the first loss value, fusing features, decoding, and training the model by combining the first and second loss values.
It improves the feature independence and accuracy of the image processing model, thereby enhancing image processing performance.
Smart Images

Figure CN123071691A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a training method, apparatus, electronic device, storage medium, and program product for an image processing model. Background Technology
[0002] Image processing refers to the use of computers to perform operations such as denoising, enhancement, restoration, segmentation, and feature extraction on images to improve image quality, enrich information content, enhance image interpretation and recognition, and meet specific analytical needs. Image processing technology is widely used in many fields such as medicine, remote sensing, industry, and art, helping to extract useful information from images and support decision-making and analysis.
[0003] In training image processing models, related technologies typically train the model directly using the image labels carried by image samples. However, due to the single training dimension, the image processing performance of the trained target image processing model is poor. Summary of the Invention
[0004] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image processing model, which can effectively improve the image processing performance of the target image processing model.
[0005] The technical solution of this application embodiment is implemented as follows:
[0006] This application provides a method for training an image processing model, including:
[0007] The image samples are encoded in multiple channels to obtain multiple image features corresponding to one channel, and the image processing model is used to perform decorrelation operation on the multiple image features to obtain multiple new image features.
[0008] Determine the degree of correlation between any two of the new image features, and determine a first loss value based on the degree of correlation;
[0009] Multiple new image features are fused to obtain fused features, and the fused features are decoded to obtain a decoded image of the image sample;
[0010] Based on the decoded image and the image sample, a second loss value is determined, and the image processing model is trained based on the first loss value and the second loss value.
[0011] This application provides an image processing method, including:
[0012] The target image processing model encodes the image to be processed in multiple channels to obtain multiple image features corresponding to one channel. Then, the target image processing model performs decorrelation operation on the multiple image features of the multiple channels to obtain multiple new image features.
[0013] The target image processing model is used to fuse the multiple new image features to obtain fused features, and the fused features are then decoded to obtain the decoded image of the image.
[0014] The target image processing model is trained using the image processing model training method described above.
[0015] This application provides a training apparatus for an image processing model, comprising:
[0016] The decorrelation module is used to encode image samples in multiple channels to obtain multiple image features corresponding to one channel, and then perform decorrelation operation on the multiple image features through the image processing model to obtain multiple new image features;
[0017] The first loss module is used to determine the degree of correlation between any two of the new image features, and to determine a first loss value based on the degree of correlation.
[0018] A decoding module is used to fuse multiple new image features to obtain fused features, and to decode the fused features to obtain a decoded image of the image sample;
[0019] The training module is used to determine a second loss value based on the decoded image and the image samples, so as to train the image processing model based on the first loss value and the second loss value.
[0020] In the above scheme, the multi-channel encoding is implemented through the encoding layer of the image processing model, and the encoding layer includes multiple serial sub-encoding layers; the decorrelation module is further used to extract features from the image samples to obtain initial image features of the image samples; through the multiple serial sub-encoding layers, the initial image features are iteratively encoded multiple times to obtain multiple image features corresponding to one channel respectively; the decorrelation module is further used to take the image features output by each sub-encoding layer as layer image features, and perform the following processing on the layer image features output by multiple sub-encoding layers respectively: perform decorrelation operation on the layer image features of multiple different channels output by the sub-encoding layers to obtain multiple new image features of the sub-encoding layers.
[0021] In the above scheme, the decorrelation module is further configured to determine, for each sub-coding layer, the degree of correlation between any two new image features among the multiple new image features of the sub-coding layer; the first loss module is further configured to determine, for each sub-coding layer, a first sub-loss value based on the corresponding degree of correlation; and to determine a first loss value based on the first sub-loss value of each sub-coding layer.
[0022] In the above scheme, the multiple new image features of the sub-coding layer include a first new image feature and a second new image feature; the first loss module is further used to determine the covariance between the first new image feature and the second new image feature, and the standard deviations corresponding to the first new image feature and the second new image feature respectively; obtain the product value of the standard deviations corresponding to the first new image feature and the second new image feature respectively, and divide the covariance between the first new image feature and the second new image feature by the product value to obtain the correlation degree between the first new image feature and the second new image feature.
[0023] In the above scheme, the first loss module is further configured to construct a cross-correlation matrix of the sub-coding layer based on the correlation degree, wherein the cross-correlation matrix includes the correlation degree between any two new image features among the multiple new image features of the sub-coding layer; obtain the spectral norm of the cross-correlation matrix; and determine the first sub-loss value of the sub-coding layer based on the spectral norm of the cross-correlation matrix.
[0024] In the above scheme, the decoding module is further configured to perform the following processing on each of the sub-coding layers: fuse multiple new image features of the sub-coding layer to obtain the fused features of the sub-coding layer; the decoding module is further configured to perform feature transformation on the fused features of each of the sub-coding layers to obtain the transformed features of the sub-coding layer; fuse the transformed features of each of the sub-coding layers to obtain the target fused features, and decode the target fused features to obtain the decoded image of the image sample.
[0025] In the above scheme, when the sub-coding layer is the first sub-coding layer in the coding layers, the above decoding module is further used to obtain the random mask feature of the fusion feature of the sub-coding layer, wherein the feature dimension of the random mask feature of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; and to fuse the random mask feature of the sub-coding layer into the fusion feature of the sub-coding layer to obtain the transformation feature of the sub-coding layer.
[0026] In the above scheme, when the sub-coding layer is not the first sub-coding layer in the coding layer, the decoding module is further configured to perform two different linear transformations on the transformation features of the fusion features of the first sub-coding layer in the coding layer to obtain key features and value features; obtain the random mask features of the fusion features of the sub-coding layer, wherein the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion features of the sub-coding layer; fuse the random mask features of the sub-coding layer into the fusion features of the sub-coding layer to obtain candidate transformation features of the sub-coding layer; perform a linear transformation on the candidate transformation features to obtain the query features corresponding to the fusion features; and fuse the query features corresponding to the fusion features, the key features, and the value features to obtain the transformation features of the sub-coding layer.
[0027] In the above scheme, the decorrelation module is further used to perform decorrelation operation on the multiple image features through the image processing model to obtain multiple candidate image features; and to perform standardization processing on the multiple candidate image features with dimensional differences to obtain the multiple new image features without the dimensional differences.
[0028] In the above scheme, the candidate image features include multiple feature elements. The aforementioned decorrelation module is further configured to perform the following processing on each of the candidate image features: determine the mean of the feature values of the multiple feature elements and the standard deviation of the feature values of the multiple feature elements; subtract the mean from the feature value of each feature element in the candidate image features to obtain the target image features; divide the feature value of each feature element in the target image features by the standard deviation to obtain the new image features.
[0029] In the above scheme, the first loss module is further configured to perform the following processing for any two new image features: determine the covariance between the new image features and the standard deviation of each new image feature; obtain the product value of the standard deviations of each new image feature; divide the covariance between the new image features by the product value to obtain the correlation degree between any two new image features.
[0030] In the above scheme, the first loss module is further configured to construct a cross-correlation matrix among the plurality of new image features based on the correlation degree, wherein the cross-correlation matrix includes the correlation degree between any two of the plurality of new image features; obtain the spectral norm of the cross-correlation matrix, and determine a first loss value based on the spectral norm of the cross-correlation matrix.
[0031] In the above scheme, the training module is further used to obtain the label image of the image sample, perform feature extraction on the label image to obtain label image features; perform feature extraction on the decoded image to obtain decoded image features, and determine the similarity between the label image features and the decoded image features, and determine the similarity as the second loss value.
[0032] This application provides an image processing apparatus, including:
[0033] The encoding module is used to encode the image to be processed through a target image processing model in multiple channels to obtain multiple image features corresponding to one channel, and to perform decorrelation operation on the multiple image features of the multiple channels through the target image processing model to obtain multiple new image features;
[0034] The decoding module is used to fuse the multiple new image features through the target image processing model to obtain fused features, and to decode the fused features to obtain the decoded image of the image; wherein the target image processing model is trained using the above-mentioned image processing model training method.
[0035] This application provides an electronic device, including:
[0036] Memory is used to store executable instructions or computer programs.
[0037] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method and image processing method of the image processing model provided in the embodiments of this application.
[0038] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the training method and image processing method of the image processing model provided in this application.
[0039] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image processing model training method and image processing method described in this application embodiment.
[0040] The embodiments of this application have the following beneficial effects:
[0041] By processing image samples, multiple image features corresponding to each channel are extracted. Since image features may contain redundant information, the image processing model performs decorrelation operations on these features to reduce linear dependencies, resulting in multiple new image features. This improves the independence of these new features, enabling the image processing model to learn the intrinsic structure of the image samples more effectively. The correlation between any two new image features is calculated. Since the correlation reflects the strength and direction of the linear relationship between the new features, a first loss value is determined based on the correlation. This first loss value measures the correlation between features, guiding the image processing model to learn more independent and discriminative features. After obtaining multiple new image features, they are fused to integrate information from different channels, resulting in a comprehensive fused feature. Then, the fused feature is decoded to reconstruct the decoded image of the image sample. By comparing the decoded image with the original image sample, a second loss value is determined. This second loss value measures the image processing performance of the model, i.e., whether the model can accurately decode the image from the fused features. The image processing model is then trained based on the first and second loss values. By training with the first and second loss values, the image processing model can simultaneously improve the independence of image features and the accuracy of image processing, thereby effectively improving the image processing performance of the target image processing model. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the architecture of the training system for the image processing model provided in this application embodiment;
[0043] Figure 2 This is a schematic diagram of the structure of an electronic device for training an image processing model provided in an embodiment of this application;
[0044] Figure 3 This is a schematic diagram of the structure of an electronic device for image processing provided in an embodiment of this application. Figure 2 ;
[0045] Figure 4 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 1 ;
[0046] Figure 5 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 2 ;
[0047] Figure 6 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 3 ;
[0048] Figure 7 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 4 ;
[0049] Figure 8 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 5 ;
[0050] Figure 9 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 6 ;
[0051] Figure 10 This is a flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ;
[0052] Figure 11 This is a schematic diagram of the coding layer structure of the image processing model provided in this application embodiment;
[0053] Figure 12 This is a schematic diagram illustrating the principle of the decorrelation operation provided in the embodiments of this application;
[0054] Figure 13 This is a schematic diagram illustrating the principle of feature conversion provided in the embodiments of this application;
[0055] Figure 14 This is a schematic diagram illustrating the principle of the first sub-loss value provided in the embodiments of this application;
[0056] Figure 15 This is a schematic diagram illustrating the effect of the medical images provided in the embodiments of this application;
[0057] Figure 16 This is a schematic diagram illustrating the principle of the training method for the image processing model provided in the embodiments of this application;
[0058] Figure 17 This is a schematic diagram illustrating the principle of the covariance matrix provided in the embodiments of this application;
[0059] Figure 18 This is a visual distribution diagram of the feature distribution provided in the embodiments of this application. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0061] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0062] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0064] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0065] 1) Artificial Intelligence (AI): This is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Pre-trained models, also known as large models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.
[0066] 2) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition. It can be understood that the real-time global illumination rendering method in this application is based on computer vision technology to process images in virtual scenes, thereby effectively improving the efficiency and effect of lighting rendering in virtual scenes.
[0067] 3) Cross-Correlation Matrix: The cross-correlation matrix is used in signal processing and statistics to describe the relationship between two image features. It contains the degree of correlation (i.e., correlation coefficient) between two image features at different time delays. Specifically, if there is an image feature X and another image feature Y, their cross-correlation matrix will show the similarity or correlation between X and Y at different time delays. The cross-correlation matrix is very useful for analyzing the synchronicity, time delay, and interaction between image features.
[0068] 4) Spectral Norm: The spectral norm, also known as the operator 2-norm, is a standard method for measuring the size of a matrix. It focuses on the maximum amplification a matrix can achieve when acting on a vector. In practical applications, the spectral norm is commonly used in control and optimization theory, numerical analysis, and other fields. The spectral norm is defined as the maximum amplification factor a matrix can achieve when acting on a unit vector; specifically, it is the largest singular value of the matrix. The calculation of the spectral norm involves the singular value decomposition (SVD) of the matrix, where the spectral norm is the largest singular value in a diagonal matrix.
[0069] 5) Random Masked Features: In data processing and machine learning, this involves randomly masking or covering certain features in the original data to generate new feature representations. This method is commonly used in tasks such as self-supervised learning, data augmentation, and feature extraction. The working principle of random masked features is as follows: First, a subset of features is selected for masking. These features can be randomly selected or selected according to certain rules. Then, the unmasked features are used to predict the masked features, thereby learning the internal structure of the data and the relationships between features.
[0070] 6) Covariance: Covariance is a statistic that measures the strength and direction of the linear relationship between two variables. It indicates the degree to which the two variables change simultaneously. If the covariance of two variables is positive, it means that they tend to increase or decrease simultaneously; if the covariance is negative, it means that when one variable increases, the other variable tends to decrease. The magnitude of covariance depends on the units of the variables, therefore it is not a standardized measure.
[0071] 7) Standard Deviation: Standard deviation is a statistic that measures the dispersion of data distribution, representing the average distance between data points and the mean. A larger standard deviation indicates a more dispersed data distribution; a smaller standard deviation indicates a more concentrated data distribution. Standard deviation is the square root of variance and is commonly used to describe the volatility of data.
[0072] 8) Degree of Correlation: The degree of correlation can be characterized by the correlation coefficient, which is a standardized statistic that measures the strength and direction of the linear relationship between two variables. It standardizes the covariance by dividing the covariance by the product of the standard deviations of the two variables. The correlation coefficient ranges from -1 to +1, where +1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no linear correlation.
[0073] During the implementation of the embodiments of this application, the applicant discovered the following problems with the related technology:
[0074] In training image processing models, related technologies typically train the model directly using the image labels carried by image samples. However, due to the single training dimension, the image processing performance of the trained target image processing model is poor.
[0075] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image processing model, which can effectively improve the image processing performance of the target image processing model. The exemplary application of the image processing model training system provided in this application is described below.
[0076] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the training system 100 for the image processing model provided in this application embodiment. The terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.
[0077] Terminal 400 is used by a user to access client 410 and display decoded images on a graphical interface 410-1 (graphical interface 410-1 is shown as an example). Terminal 400 and server 200 are interconnected via wired or wireless network.
[0078] In some embodiments, server 200 can be a standalone physical server, a server cluster or business system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, in-vehicle terminal, etc., but is not limited to these. The electronic device provided in this application embodiment can be implemented as a terminal or a server. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.
[0079] In some embodiments, server 200 performs multi-channel encoding on image samples to obtain multiple image features corresponding to one channel, performs decorrelation operation on the multiple image features to obtain multiple new image features, determines a first loss value based on the degree of correlation, and determines a second loss value based on the decoded image and image samples, and sends the first loss value and the second loss value to terminal 400. Terminal 400 trains the image processing model based on the first loss value and the second loss value to obtain the target image processing model.
[0080] In other embodiments, terminal 400 performs multi-channel encoding on image samples to obtain multiple image features corresponding to one channel, performs decorrelation operation on the multiple image features to obtain multiple new image features, determines a first loss value based on the degree of correlation, and determines a second loss value based on the decoded image and image samples, and sends the first loss value and the second loss value to server 200. Server 200 trains the image processing model based on the first loss value and the second loss value to obtain the target image processing model.
[0081] In some embodiments, within the medical field, image processing models can be used to analyze X-ray, CT scan, and MRI images. By multi-channel encoding these images, features of different tissues and structures are extracted, and decorrelation operations are performed to improve feature independence. Then, by calculating the correlation between features and determining a first loss value, the model can learn more independent and discriminative features. Fusing and decoding these features allows for the reconstruction of clearer and more accurate medical images, aiding physicians in making more accurate diagnoses.
[0082] In some embodiments, in image denoising and enhancement tasks, image processing models can be used to remove noise and blur from images and enhance image details and contrast. By multi-channel encoding of the image, extracting features from different channels, and performing decorrelation operations, redundant information between features can be reduced. Then, by calculating the correlation between features and determining a first loss value, the model can learn more independent and discriminative features. Fusing features and decoding them can reconstruct a clearer and sharper image, improving image quality and usability.
[0083] In some embodiments, in object detection and recognition tasks, image processing models can be used to detect and recognize objects in images. By multi-channel encoding of the image, extracting features of different objects, and performing decorrelation operations, the independence of features can be improved. Then, by calculating the correlation between features and determining a first loss value, the model can learn more independent and discriminative features. Fusing and decoding the features can reconstruct a more accurate and clearer target image, improving the accuracy of object detection and recognition.
[0084] In some embodiments, in image style transfer tasks, image processing models can be used to transfer the style of one image to another. By multi-channel encoding of the image, features of different styles are extracted, and decorrelation operations are performed to reduce redundant information between features. Then, by calculating the correlation between features and determining a first loss value, the model can learn more independent and discriminative features. By fusing the features and decoding them, an image with the target style can be generated, achieving the effect of style transfer.
[0085] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for training an image processing model provided in an embodiment of this application, wherein, Figure 2 The electronic device 500 shown can be Figure 1 Server 200 or terminal 400 in the middle, Figure 2The illustrated electronic device 500 includes at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.
[0086] Processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0087] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 430.
[0088] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.
[0089] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
[0090] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;
[0091] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, such as Bluetooth, WiFi, and Universal Serial Bus (USB).
[0092] In some embodiments, the training apparatus for the image processing model provided in this application can be implemented in software. Figure 2 A training device 455 for an image processing model stored in memory 450 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a decorrelation module 4551, a first loss module 4552, a decoding module 4553, and a training module 4554. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.
[0093] See Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device for image processing provided in an embodiment of this application. Figure 2 ,in, Figure 3 The electronic device 600 shown can be Figure 1 Server 200 or terminal 400 in the middle, Figure 3 The illustrated electronic device 600 includes at least one processor 530, a memory 550, and at least one network interface 520. The various components in the electronic device 600 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 540.
[0094] Processor 530 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.
[0095] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 530.
[0096] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.
[0097] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
[0098] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;
[0099] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, such as Bluetooth, WiFi, and Universal Serial Bus (USB).
[0100] In some embodiments, the image processing apparatus provided in this application can be implemented in software. Figure 3 An image processing apparatus 555 stored in memory 550 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: encoding module 5551 and decoding module 5552. These modules are logically linked and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.
[0101] In other embodiments, the training device for the image processing model provided in this application can be implemented in hardware. As an example, the training device for the image processing model provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method for the image processing model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.
[0102] In some embodiments, the terminal or server can implement the training method and image processing method of the image processing model provided in this application by running a computer program or computer-executable instructions. For example, the computer program can be a native program in the operating system (e.g., a dedicated image deblurring program) or a software module, such as a training module that can be embedded in any program (e.g., an instant messaging client, a photo album program, an electronic map client, a navigation client); for example, it can be a native application (APP), that is, a program that needs to be installed in the operating system to run. In summary, the above-mentioned computer program can be any form of application, module, or plugin.
[0103] The training method of the image processing model provided in the embodiments of this application will be described in conjunction with exemplary applications and implementations of the server or terminal provided in the embodiments of this application.
[0104] See Figure 4 , Figure 4 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 1 , will combine Figure 4 Steps 101 to 108 are described below. The training method of the image processing model provided in this application embodiment can be implemented by the server or the terminal alone, or by the server and the terminal working together. The following description will take the implementation by the server alone as an example.
[0105] In step 101, the image samples are multi-channel encoded to obtain multiple image features corresponding to one channel.
[0106] In some embodiments, multi-channel coding is an image processing technique that encodes different channels (such as color channels, texture channels, etc.) of an image sample to extract multiple image features corresponding to each channel. In other words, each channel corresponds to one image feature, which means multiple image features corresponding to each channel. This can capture information from different aspects of the image, thereby improving the accuracy of image recognition and analysis.
[0107] In some embodiments, an image sample refers to a single image used to train or test an image processing model. In machine learning and deep learning, image samples are commonly used to train models to recognize specific features or patterns. In image processing, a channel refers to a color component or feature component in an image. For example, a typical color image has three channels: red, green, and blue (RGB). Each channel represents information about the corresponding color component in the image. By processing information from different channels, different features of the image can be extracted.
[0108] In some embodiments, in multi-channel encoding scenarios, a channel can be a color component, but also includes other possible feature components such as texture, edges, and shape. By separating image data into different channels, the information in each channel can be processed and analyzed independently, thereby extracting the specific features of each channel. For example, a color channel in a color image refers to the representation of different color components in the image. For instance, an RGB image has three channels: red, green, and blue, each corresponding to the intensity value of a color component. A feature channel, in the feature extraction process, refers to the representation of different feature types in the image. For example, a texture channel is used to represent texture information in the image, an edge channel is used to represent edge information in the image, and a shape channel is used to represent shape information in the image.
[0109] In some embodiments, a channel is a fundamental concept in image processing and computer vision, referring to the decomposition of image data into multiple independent components, each representing a different feature or attribute of the image. This decomposition allows for individual processing and analysis of information from each channel, enabling more efficient extraction and utilization of various information within the image. In color images, color channels are the most common type. For example, an RGB image has three color channels: red, green, and blue. Each pixel in an RGB image has three corresponding values, representing the intensity of that pixel in the red, green, and blue channels, respectively. By adjusting the values of these channels, the overall color and hue of the image can be altered. Besides color channels, feature channels are also commonly used concepts in image processing. Feature channels are used to represent specific features in an image, such as texture, edges, and shape. For example, a texture channel might contain information about texture patterns in the image, an edge channel might contain information about edges and contours in the image, and a shape channel might contain information about the shape of objects in the image. By processing and analyzing these feature channels, the shape, texture, and other visual features of objects in the image can be extracted. The concept of channels is not limited to image processing; it is also widely used in signal processing, audio processing, and many other fields. In these fields, a channel typically refers to the different components of a signal, with each channel representing a specific aspect or characteristic of the signal. By decomposing a signal into multiple channels, the information in each channel can be processed and analyzed independently, thereby improving the efficiency and accuracy of signal processing.
[0110] In some embodiments, the multi-channel encoding is implemented through the encoding layer of the image processing model, the encoding layer including multiple serial sub-encoding layers, and the above step 101 can be implemented as follows: extracting features from the image sample to obtain the initial image features of the image sample; performing multi-channel encoding on the initial image features multiple times through the multiple serial sub-encoding layers to obtain the multiple image features corresponding to one channel respectively.
[0111] In some embodiments, the server performs feature extraction on the input image samples through a convolution operation, where the convolution kernel (filter) slides across the image to detect various features in the image, such as edges and textures. The result of feature extraction is initial image features, which are the representation of the image samples in the feature space.
[0112] In some embodiments, the coding layer consists of multiple sequential sub-coding layers. Each sub-coding layer contains a series of convolutional layers, pooling layers, and possibly normalization layers, working together to extract increasingly abstract features. For initial image features, the server iteratively encodes them through multiple sequential sub-coding layers. Each time a sub-coding layer is passed, the image features are further processed and abstracted, analyzing the image at different levels, from low-level pixel information to high-level semantic information, extracting rich and discriminative image features through multi-channel encoding.
[0113] In some embodiments, feature extraction refers to processing image samples using the coding layer of an image processing model to extract feature representations that can describe the image content. The coding layer typically contains multiple sequential sub-coding layers, which progressively extract low- to high-level features of the image through a series of convolution, pooling, and normalization operations. The initial image features are the result of preliminary processing by the coding layer, while multi-channel coding further processes these initial features to extract specific features for each channel.
[0114] In some embodiments, the above-mentioned process of iteratively encoding the initial image features multiple times through the multiple serial sub-coding layers to obtain the multiple image features corresponding to one channel can be implemented as follows: The initial image features are encoded for the first time through the first sub-coding layer to obtain the first image feature of each channel; the i-th image feature is encoded for the (i+1)-th time through the i-th sub-coding layer to obtain the (i+1)-th image feature of each channel, where 1 ≤ i ≤ N-1, and N is a positive integer greater than 2; i is iterated until i equals N-1, and the first image feature value is determined as the N-th image feature.
[0115] In some embodiments, initial image features are input into the first sub-coding layer, which typically includes convolutional layers, pooling layers, and normalization layers for further extraction and abstraction of the input features. The first sub-coding layer processes the data to obtain the first image feature for each channel. The i-th image feature is then input into the i-th sub-coding layer for the (i+1)-th multi-channel encoding. Here, i is an iteration variable, starting from 1 and increasing by 1 each time, until i equals N-1, where N is a positive integer greater than 2, representing the total number of sub-coding layers. In each iteration, the sub-coding layer processes the input image features, extracting higher-level features. This process analyzes the image at different levels, from low-level pixel information to high-level semantic information. By encoding and processing the features of each channel separately, specific features for each channel can be extracted. These features are typically high-dimensional vectors containing detailed information about the image in a specific channel. After N-1 iterations of encoding, the N-th image feature is obtained. The N-th image feature is the final result after processing by all sub-coding layers and contains rich information about the image.
[0116] As an example, see Figure 11 , Figure 11 This is a schematic diagram of the coding layer structure of the image processing model provided in this application embodiment. Feature extraction is performed on image samples to obtain initial image features; through the first sub-coding layer ( Figure 11 The sub-coding layer 1 shown performs the first multi-channel encoding on the initial image features, obtaining the first image feature (H1*F1*W1) for each channel; then, through the second sub-coding layer (… Figure 11 The sub-coding layer 2 shown performs a second multi-channel encoding on the first image features, obtaining the second image features (H2*F2*W2) for each channel; through the third sub-coding layer ( Figure 11 The sub-coding layer 3 shown performs a third multi-channel encoding on the second image features, obtaining the third image features (H3*F3*W3) for each channel; through the fourth sub-coding layer ( Figure 11 The sub-coding layer 4 shown above performs a fourth multi-channel encoding on the third image features to obtain the fourth image features (H4*F4*W4) for each channel.
[0117] Thus, through these sub-coding layers, the initial image features are iteratively encoded in multiple channels. In each iteration, the sub-coding layer focuses on extracting specific types of features, such as color, texture, and shape. This iterative process allows multiple image features corresponding to a single channel to capture different aspects of the image information. Multiple serial sub-coding layers iterate through the initial image features in multiple channels. In each iteration, different sub-coding layers can analyze and transform features from different angles and scales. This multi-channel approach fully explores different types of features in the image, enriching the expressive power of the features. Each sub-coding layer operates at different scales and feature spaces. The first sub-coding layer might extract local detail features of the image using small convolutional kernels. As iterations progress, subsequent sub-coding layers synthesize previous features and further extract more abstract and semantic features. Multi-channel encoding, like observing an image from multiple perspectives, allows the final multiple image features corresponding to a single channel to comprehensively and deeply describe the image.
[0118] In step 102, decorrelation operation is performed on the multiple image features using an image processing model to obtain multiple new image features.
[0119] In some embodiments, decorrelation operations are performed on multiple new image features to ensure that the resulting new image features are pairwise positively interactive and uncorrelated. Decorrelation operations are used to reduce or eliminate the correlation between image features. Decorrelation operations are typically used to process multiple image features so that the resulting new image features are pairwise orthogonal. Orthogonality means that these features are mathematically independent, their inner product is zero, meaning that a change in one feature does not affect a change in another feature. Decorrelation operations can improve the accuracy of image recognition and classification because each feature provides unique information that helps distinguish different types of images. Decorrelation operations can also reduce redundant information between features, improving the model's generalization ability.
[0120] In some embodiments, step 102 above can be implemented as follows: taking the image features output by each sub-coding layer as layer image features, and performing the following processing on the layer image features output by multiple sub-coding layers respectively: performing decorrelation operation on the layer image features of multiple different channels output by the sub-coding layers to obtain multiple new image features of the sub-coding layers.
[0121] In some embodiments, during the multi-channel encoding process performed by the server, the image features output by each sub-coding layer are called layer image features. The server extracts features from image samples through multiple sequential sub-coding layers. Each sub-coding layer outputs a layer image feature, which contains information about the image at different levels. Each layer image feature contains information from multiple channels. The server separates this channel information to obtain multiple layer image features for different channels. For the multiple layer image features output by each sub-coding layer, the server performs a decorrelation operation. The purpose of this operation is to reduce or eliminate the correlation between different channel features, making the resulting new image features pairwise orthogonal. Through the decorrelation operation, the server obtains multiple new image features for each sub-coding layer. These new features are mathematically independent, and their inner product is zero, meaning that a change in one feature does not affect a change in another feature. In some cases, the server may fuse the new image features from different sub-coding layers to obtain a comprehensive image representation.
[0122] Continuing from the previous example, the image features output by the first sub-coding layer are used as layer image features. The following processing is performed on the layer image features output by the first sub-coding layer: For the first image feature of each channel (the layer image feature output by the first sub-coding layer), a decorrelation operation is performed to obtain multiple new image features of the first sub-coding layer.
[0123] Continuing from the previous example, the image features output by the i-th sub-coding layer are used as layer image features. The following processing is performed on the layer image features output by the i-th sub-coding layer: For the i-th image feature of each channel (the layer image feature output by the 1st sub-coding layer), a decorrelation operation is performed to obtain multiple new image features of the i-th sub-coding layer.
[0124] Following the previous example, see [link to example]. Figure 12 , Figure 12 This is a schematic diagram illustrating the principle of decorrelation operation provided in the embodiments of this application, which converts the image features output by the first sub-coding layer ( Figure 12 The F1 shown is used as the layer image feature. The following processing is performed on the layer image features output from the first sub-coding layer: For the first image feature of each channel (the layer image feature output from the first sub-coding layer), Figure 12 The F1 code shown is used to perform decorrelation to obtain multiple new image features of the first sub-coding layer. Figure 12 The figure shown ).
[0125] Following the previous example, see [link to example]. Figure 12 The image features output by the second sub-coding layer ( Figure 12As shown in F2), as the layer image feature, the following processing is performed on the layer image feature output by the second sub-coding layer: For the second image feature of each channel (the layer image feature output by the second sub-coding layer), Figure 12 As shown in F2), perform decorrelation operation to obtain multiple new image features of the second sub-coding layer. Figure 12 The figure shown ).
[0126] Following the previous example, see [link to example]. Figure 12 The image features output by the third sub-coding layer ( Figure 12 As shown in F3), as the layer image feature, the following processing is performed on the layer image feature output by the third sub-coding layer: For the third image feature of each channel (the layer image feature output by the third sub-coding layer), Figure 12 As shown in F3), perform decorrelation operation to obtain multiple new image features of the third sub-coding layer. Figure 12 The figure shown ).
[0127] Following the previous example, see [link to example]. Figure 12 The image features output by the 4th sub-coding layer ( Figure 12 As shown in F4), as the layer image feature, the following processing is performed on the layer image feature output by the 4th sub-coding layer: For the 4th image feature of each channel (the layer image feature output by the 4th sub-coding layer), Figure 12 As shown in F4), a decorrelation operation is performed to obtain multiple new image features of the fourth sub-coding layer. Figure 12 The figure shown ).
[0128] Thus, by using the image features output by each sub-coding layer as layer image features and performing decorrelation operations on the layer image features output by multiple sub-coding layers respectively, the performance of image recognition and classification can be significantly improved. The decorrelation operation reduces or eliminates the correlation between features from different channels, making each new image feature mathematically independent, thereby improving the discriminativeness and uniqueness of the new image features. The orthogonality between the new image features enhances the sensitivity of the image processing model to subtle differences in image samples, reduces feature redundancy, and thus improves the generalization ability of the image processing model.
[0129] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 2 , Figure 4 Step 102 shown can be achieved through Figure 5 Steps 1021 to 1022 shown are implemented.
[0130] In step 1021, the image processing model is used to perform decorrelation operation on the multiple image features to obtain multiple candidate image features.
[0131] In some embodiments, since the image processing model has not yet been trained, the multiple candidate image features obtained by performing decorrelation operation on multiple image features through the image processing model cannot be accurately decorrelated, that is, the multiple candidate image features obtained are not pairwise orthogonal. After the image processing model is trained in the future, the target image processing model can be used to achieve accurate decorrelation operation, so that the obtained decorrelated image features are pairwise orthogonal.
[0132] In some embodiments, decorrelation operations are used to reduce redundant information between features, improving feature discriminativeness and model generalization ability. However, this process depends on the training state of the image processing model, which performs decorrelation operations on these initial features to reduce the correlation between features. However, since the model has not yet been trained, the decorrelation operation may not accurately achieve decorrelation between features. As a result, multiple candidate image features may have high correlation, i.e., they are not pairwise orthogonal. To improve this situation, the image processing model is trained using the image processing model training method provided in the embodiments of this application. During training, the image processing model learns how to extract and represent features more effectively, while optimizing the parameters of the decorrelation operation to reduce the correlation between features. After training, the image processing model (now called the target image processing model) can perform decorrelation operations more accurately, and the image features processed by the target image processing model are pairwise orthogonal, i.e., they are mathematically independent, and a change in one feature does not affect a change in another feature.
[0133] Following the previous example, see [link to example]. Figure 12 The image features output by the 4th sub-coding layer ( Figure 12 As shown in F4), as the layer image feature, the following processing is performed on the layer image feature output by the 4th sub-coding layer through the image processing model: For the 4th image feature of each channel (the layer image feature output by the 4th sub-coding layer), Figure 12 As shown in F4), a decorrelation operation is performed to obtain multiple new image features of the fourth sub-coding layer. Figure 12 The figure shown ).
[0134] Following the previous example, see [link to example]. Figure 12 The image features output by the third sub-coding layer ( Figure 12As shown in F3), as the layer image feature, the following processing is performed on the layer image feature output by the third sub-coding layer through the image processing model: For the third image feature of each channel (the layer image feature output by the third sub-coding layer), Figure 12 As shown in F3), perform decorrelation operation to obtain multiple new image features of the third sub-coding layer. Figure 12 The figure shown ).
[0135] In step 1022, the multiple candidate image features with dimensional differences are standardized to obtain the multiple new image features without the dimensional differences.
[0136] In some embodiments, standardization is used to adjust features with different dimensions to have the same dimension, typically by converting them to a standard normal distribution with a mean of 0 and a standard deviation of 1. In image processing, standardization can improve the training efficiency and accuracy of a model because it eliminates the dimensional differences between features, allowing the model to learn the importance of each feature more fairly.
[0137] In some embodiments, the candidate image features include multiple feature elements. The standardization process for the multiple candidate image features with dimensional differences to obtain multiple new image features without such dimensional differences can be implemented as follows: For each candidate image feature, the following processing is performed: determining the mean of the feature values of the multiple feature elements and the standard deviation of the feature values of the multiple feature elements; subtracting the mean from the feature value of each feature element in the candidate image features to obtain the target image feature; and dividing the feature value of each feature element in the target image feature by the standard deviation to obtain the new image feature.
[0138] In some embodiments, in image processing, candidate image features typically contain multiple feature elements, each with its own specific feature value. These feature values may have different dimensions; for example, some features may represent color intensity, while others may represent texture complexity. To make these features comparable during image processing model training and to improve the performance of the image processing model, the candidate image features can be standardized. For each candidate image feature, the mean (μ) and standard deviation (σ) of the feature values of all its feature elements are first calculated. The mean represents the central tendency of the feature values, while the standard deviation describes the dispersion of the feature values. The target image feature is obtained by subtracting the corresponding mean from the feature value of each feature element in the candidate image feature. This process centers the feature values to zero, eliminating feature offsets. The standardized new image feature is obtained by dividing the feature value of each feature element in the target image feature by the corresponding standard deviation. This step adjusts the scale of the feature values to unit standard deviation, making the distribution range of the feature values more consistent. Each candidate image feature is transformed into a new image feature without dimensional differences. These new features are numerically standardized, with a mean of zero and a standard deviation of one. This makes them equally weighted and comparable in image processing model training. Standardization helps improve the training efficiency and accuracy of image processing models because it eliminates the dimensional differences between candidate image features, allowing the image processing model to learn the importance of each feature more fairly.
[0139] As an example, the expression for the above new image features can be:
[0140]
[0141] Where μ indicates the mean of the eigenvalues of multiple feature elements in the candidate image features, δ indicates the standard deviation of the eigenvalues of multiple feature elements, and F' indicates the candidate image features. Used to indicate new image features.
[0142] Continuing from the previous example, the image features output by the third sub-coding layer ( Figure 12 As shown in F3), as the layer image feature, the following processing is performed on the layer image feature output by the third sub-coding layer through the image processing model: For the third image feature of each channel (the layer image feature output by the third sub-coding layer), Figure 12 As shown in F3), perform decorrelation operation to obtain multiple candidate image features of the third sub-coding layer, determine the mean and standard deviation of the feature values of multiple feature elements; subtract the mean from the feature value of each feature element in the candidate image features to obtain the target image features; divide the feature value of each feature element in the target image features by the standard deviation to obtain the new image features. Figure 12 The figure shown ).
[0143] Thus, by standardizing multiple candidate image features with dimensional differences, the dimensional differences between feature values can be eliminated, ensuring that each new image feature has the same scale and distribution numerically. The mean and standard deviation of the feature values of multiple feature elements in each candidate image feature are calculated. Then, the mean is subtracted from the feature value of each feature element to obtain the target image feature. Finally, the feature value of each feature element in the target image feature is divided by the standard deviation to obtain the standardized new image feature. Standardization ensures that the mean of the new image feature is zero and the standard deviation is one, thereby eliminating dimensional differences between features and improving feature comparability and model training efficiency. Simultaneously, standardization can also reduce the correlation between features, improving the generalization ability and accuracy of the image processing model.
[0144] Thus, by performing decorrelation operations on multiple image features through an image processing model, redundant information between features can be effectively reduced, improving feature discriminative power and the model's generalization ability. However, since image features typically have different dimensions, this may lead the model to assign excessive weights to certain features during training, thereby affecting model performance. Therefore, standardization of multiple candidate image features with different dimensions is necessary. Standardization transforms all features to the same dimension, typically converting them to a standard normal distribution with a mean of 0 and a standard deviation of 1. This helps the image processing model learn the importance of each feature more fairly, improving the training efficiency and accuracy of the image processing model.
[0145] In step 103, the degree of correlation between any two of the new image features is determined.
[0146] In some embodiments, the correlation coefficient described above is a statistical indicator that measures the strength and direction of the linear relationship between two variables. The correlation coefficient value ranges from -1 to +1, where: +1 indicates a perfectly positive linear relationship between the two variables, meaning an increase in one variable is accompanied by an increase in the other; -1 indicates a perfectly negative linear relationship between the two variables, meaning an increase in one variable is accompanied by a decrease in the other; and 0 indicates no linear relationship between the two variables, meaning their changes are independent. In the cross-correlation matrix, each element represents the degree of correlation between a pair of variables. The correlation coefficient is typically calculated based on the Pearson correlation coefficient, assuming a linear relationship between the variables and that the variables follow a normal distribution.
[0147] In some embodiments, when multi-channel encoding of image samples is achieved through multiple sequential sub-coding layers of an image processing model, the correlation degree between any two new image features refers to the correlation degree between any two new image features among the multiple new image features of the sub-coding layer. Determining the correlation degree between any two new image features means calculating the correlation degree between any two new image features among the multiple new image features output by the sub-coding layer when multi-channel encoding of image samples is achieved through multiple sequential sub-coding layers of an image processing model. The correlation degree is a statistical indicator used to measure the strength and direction of the linear relationship between two variables. In image processing, the correlation degree can help understand the relationship between features in different channels and their role in image recognition and classification tasks. By calculating the correlation degree, it is possible to discover which features have a high correlation, thereby optimizing the feature selection or dimensionality reduction process and improving the performance and accuracy of the model.
[0148] In some embodiments, step 103 above can be implemented as follows: for each sub-coding layer, determine the degree of correlation between any two new image features among the plurality of new image features of the sub-coding layer.
[0149] In some embodiments, in deep learning image processing models, particularly those containing multiple sequential sub-coding layers, each sub-coding layer extracts different features of the image. For each sub-coding layer, the correlation between any two of the new image features is determined. For each feature vector, the mean and standard deviation of all its elements are calculated. This is for subsequent standardization processing, ensuring that the elements of each feature vector have the same dimensions. The mean is subtracted from each element of the feature vector, and then divided by the standard deviation to obtain the standardized feature vector. This step ensures that the elements of all feature vectors have zero mean and unit variance. For any two new image features in the sub-coding layer, the correlation between them is calculated.
[0150] As an example, the multiple new image features of the sub-coding layer include new image feature A, new image feature B, and new image feature C. Determining the correlation between any two new image features among the multiple new image features of the sub-coding layer means determining the correlation between new image feature A and new image feature B, the correlation between new image feature B and new image feature C, the correlation between new image feature A and new image feature C, the correlation between new image feature C and new image feature C, the correlation between new image feature A and new image feature A, and the correlation between new image feature B and new image feature B.
[0151] In some embodiments, see Figure 6, Figure 6 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 3 , Figure 4 Step 103 shown can be performed for any two of the new image features. Figure 6 Steps 1031 to 1032 shown are implemented.
[0152] In step 1031, the covariance between the new image features and the standard deviation of each of the new image features are determined.
[0153] In some embodiments, the covariance between new image features is used to indicate the synchronicity of changes between the new image features. If the covariance of two new image features is positive, it means that when the value of one feature increases, the value of the other feature also tends to increase; if the covariance is negative, it means that when the value of one new image feature increases, the value of the other new image feature tends to decrease. The magnitude of the covariance reflects the strength of the linear relationship between the two new image features.
[0154] In some embodiments, standard deviation is a measure of the degree of variation of an individual new image feature. It represents the average distance between the feature value and the feature mean. A larger standard deviation indicates a more dispersed distribution of feature values; a smaller standard deviation indicates a more concentrated distribution of feature values. Standard deviation is a dimensionless indicator because it is calculated based on the original data, eliminating the influence of dimensions.
[0155] In step 1032, the product of the standard deviations of each of the new image features is obtained, and the covariance between the new image features is divided by the product value to obtain the correlation between any two of the new image features.
[0156] In some embodiments, correlation is an indicator used to measure the strength and direction of the linear relationship between two variables. To determine the correlation between new image features, the covariance between any two new image features needs to be calculated, and the standard deviation of each new image feature needs to be calculated. This involves calculating the square root of the average of the squares of the differences between each feature value and its mean. The standard deviations of the two features are then multiplied to obtain the product of the standard deviations. The correlation value is then divided by the product of the standard deviations to obtain the correlation between the two features. The correlation value ranges from -1 to +1. +1 indicates a perfectly positive linear relationship between the two features, -1 indicates a perfectly negative correlation, and 0 indicates no linear relationship. By calculating the correlation, the strength and direction of the relationship between any two new image features can be understood.
[0157] As an example, the expression for the degree of correlation between any two new image features mentioned above can be:
[0158]
[0159] Where cov(i,j) is used to indicate the covariance between new image feature i and new image feature j, δ i δ is used to indicate the standard deviation of new image feature i. j X is used to indicate the standard deviation of new image feature j. ij Used to indicate the degree of correlation between new image feature i and new image feature j.
[0160] Thus, by determining the covariance between new image features and the standard deviation of each new image feature, and further calculating their product of standard deviations, the correlation between any two new image features can be obtained. Calculating the correlation not only eliminates the influence of feature dimensions, making the relationships between different features comparable, but also provides an important basis for feature selection and dimensionality reduction. The correlation degree can help identify redundant features, reduce model complexity, and improve the generalization ability and prediction accuracy of image processing models.
[0161] In some embodiments, the plurality of new image features of the sub-coding layer include a first new image feature and a second new image feature. The determination of the correlation degree between any two of the plurality of new image features of the sub-coding layer can be achieved by: determining the covariance between the first new image feature and the second new image feature, and the standard deviations corresponding to the first new image feature and the second new image feature respectively; obtaining the product of the standard deviations corresponding to the first new image feature and the second new image feature respectively; and dividing the covariance between the first new image feature and the second new image feature by the product value to obtain the correlation degree between the first new image feature and the second new image feature.
[0162] In some embodiments, the covariance between the first new image feature and the second new image feature is calculated. The covariance reflects whether the changes in the feature values of the first and second new image features are synchronized; that is, when one feature value increases, does the other feature value tend to increase or decrease? The standard deviations of the first and second new image features are calculated separately. The standard deviation measures the degree of deviation of an individual feature value from its mean, i.e., how volatile the feature value is. The larger the standard deviation, the more dispersed the distribution of the feature values; the smaller the standard deviation, the more concentrated the distribution of the feature values. The standard deviations of the first and second new image features are multiplied to obtain the product of the standard deviations. This product is used as the denominator to standardize the covariance, ensuring that the correlation is a dimensionless indicator that can be compared between different features. The correlation between the first and second new image features is divided by the product of the standard deviations to obtain the correlation between the two features. The correlation value ranges from -1 to +1, where +1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no linear relationship.
[0163] In some embodiments, the degree of correlation includes the i-th degree of correlation. The determination of the degree of correlation between any two new image features can be achieved by traversing i and performing the following process to determine the i-th degree of correlation between any two new image features.
[0164] In some embodiments, determining the degree of correlation between any two i-th new image features can be achieved by performing the following processing on any two i-th new image features: determining the covariance between the i-th new image features and the standard deviation of each i-th new image feature; dividing the covariance between the i-th new image features by the product of the standard deviations of each i-th new image feature to obtain the degree of correlation between any two i-th new image features.
[0165] Thus, by determining the covariance and standard deviation of the first and second new image features in the sub-coding layer, and calculating the product of their standard deviations, the correlation between the two features is obtained by dividing the covariance by this product. This not only eliminates the influence of feature dimensions, making the relationships between different features comparable, but also provides an important basis for feature selection and dimensionality reduction. In image processing and machine learning, the degree of correlation can help identify redundant features, reduce the complexity of image processing models, and improve the generalization ability and prediction accuracy of image processing models.
[0166] In step 104, a first loss value is determined based on the degree of correlation.
[0167] In some embodiments, see Figure 7 , Figure 7 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 4 , Figure 4 Step 104 shown can be achieved through Figure 7 Steps 1041 to 1042 shown are implemented.
[0168] In step 1041, for each sub-coding layer, a first sub-loss value is determined based on the corresponding degree of correlation.
[0169] In some embodiments, step 1041 above can be implemented as follows: for each sub-coding layer, based on the degree of correlation between any two new image features among the multiple new image features of the sub-coding layer, the first sub-loss value of the sub-coding layer is determined.
[0170] As an example, the sub-coding layer has multiple new image features, including new image feature A, new image feature B, and new image feature C. Based on the correlation between new image feature A and new image feature B, the correlation between new image feature B and new image feature C, and the correlation between new image feature A and new image feature C, the first sub-loss value of the sub-coding layer is determined.
[0171] In some embodiments, step 1041 above can be implemented as follows: based on the correlation degree, construct the cross-correlation matrix of the sub-coding layer, the cross-correlation matrix including the correlation degree between any two new image features among the multiple new image features of the sub-coding layer; obtain the spectral norm of the cross-correlation matrix, and determine the first sub-loss value of the sub-coding layer based on the spectral norm of the cross-correlation matrix.
[0172] In some embodiments, the construction of the cross-correlation matrix of the sub-coding layer based on the degree of correlation can be achieved as follows: the cross-correlation matrix of the sub-coding layer is constructed based on the degree of correlation between any two new image features among multiple new image features of the sub-coding layer.
[0173] In some embodiments, a cross-correlation matrix is a statistical matrix used to describe the linear relationship between multiple variables. In image processing, particularly in deep learning models, the cross-correlation matrix is used to represent the correlation between multiple new image features in a sub-coding layer. Specifically, each element in the cross-correlation matrix represents the degree of correlation between a pair of new image features; these correlation degrees measure the strength and direction of the linear relationship between the two feature values. The calculated correlation degrees are then filled into the cross-correlation matrix. The diagonal elements of the matrix represent the correlation degree of each feature with itself, typically 1, because a feature is perfectly correlated with itself. The off-diagonal elements represent the correlation degree between different features. The cross-correlation matrix is a symmetric matrix.
[0174] As an example, when the new image features of the sub-coding layer include new image feature A, new image feature B, and new graphic feature C, the expression for the cross-correlation matrix of the sub-coding layer can be:
[0175]
[0176] Where W is used to indicate the cross-correlation matrix of the sub-coding layer, X AA X is used to indicate the degree of correlation between new image feature A and new image feature A'. BB X is used to indicate the degree of correlation between new image feature B and new image feature B'. CC X is used to indicate the degree of correlation between new image feature C and new image feature C. BA X is used to indicate the degree of correlation between new image feature B and new image feature A. AB X is used to indicate the degree of correlation between new image feature A and new image feature B. AC X is used to indicate the degree of correlation between new image feature A and new image feature C. BC X is used to indicate the degree of correlation between new image feature B and new image feature C. CA X is used to indicate the degree of correlation between new image feature C and new image feature A. CB Used to indicate the degree of correlation between new image feature C and new image feature B.
[0177] In some embodiments, the spectral norm of a cross-correlation matrix is an important concept in matrix theory, describing the maximum stretchability of a matrix across all unit vectors. The spectral norm of a cross-correlation matrix refers to the maximum output length that the matrix can produce when acting on unit vectors. This value is equal to the matrix's largest singular value, which is the square root of the matrix's eigenvalues.
[0178] As an example, the expression for the first sub-loss value of the sub-coding layer can be:
[0179]
[0180] Where L1i indicates the first sub-loss value of the i-th sub-coding layer, σ indicates the spectral norm function, and Σ μ Used to indicate the cross-correlation matrix. Used to indicate the transpose of the cross-correlation matrix.
[0181] Following the previous example, see [link to example]. Figure 14 , Figure 14 This is a schematic diagram illustrating the principle of the first sub-loss value provided in the embodiments of this application, based on the cross-correlation matrix ( Figure 14 The Σ shown μ ) spectral norm ( Figure 14 The spectral norm function σ shown determines the first sub-loss value of the sub-coding layer. Specifically, the cross-correlation matrix and its transpose are multiplied to obtain the reference matrix, the spectral norm of the reference matrix is determined, and the spectral norm of the reference matrix is then multiplied by... Figure 14 The first loss value is obtained by multiplying the shown symmetric matrix (with zero elements on the main diagonal and 1 elements elsewhere). Since the main diagonal elements of the cross-correlation matrix are fixed values, the eigenvalues of the main diagonal elements can be eliminated by multiplying it with the symmetric matrix (with zero elements on the main diagonal and 1 elements elsewhere), thus effectively reducing the amount of subsequent computation.
[0182] Thus, by constructing the cross-correlation matrix of the sub-coding layer based on the degree of correlation, the linear relationships between multiple new image features can be comprehensively captured. This not only helps in understanding the interactions between features but also provides crucial information for feature selection and dimensionality reduction. The spectral norm of the cross-correlation matrix, as the largest singular value of the matrix, reflects the overall size and potential complexity of the matrix. By obtaining the spectral norm of the cross-correlation matrix and using it as part of the first sub-loss value of the sub-coding layer, the complexity of the image processing model can be effectively controlled, preventing overfitting.
[0183] In step 1042, a first loss value is determined based on the first sub-loss value of each of the sub-coding layers.
[0184] In some embodiments, step 1042 can be implemented as follows: summing the first sub-loss values of each sub-coding layer to obtain the first loss value.
[0185] As an example, the expression for the first loss value above can be:
[0186]
[0187] Where L1 is used to indicate the first loss value, and L1i is used to indicate the first sub-loss value of each sub-coding layer.
[0188] Thus, for each sub-coding layer, by determining the first sub-loss value based on the corresponding correlation level, the strength of the linear relationship between new image features in that sub-coding layer can be accurately measured, and the spectral norm of the cross-correlation matrix can be calculated, thereby controlling the complexity of the image processing model and preventing overfitting. By combining the first sub-loss values of each sub-coding layer to determine the first loss value, not only can the overall performance of the image processing model be comprehensively evaluated, but the optimization direction of the image processing model can also be guided.
[0189] In some embodiments, when multi-channel encoding of multiple image samples is achieved through a single encoding layer, step 104 above can also be implemented as follows: based on the correlation degree, construct a cross-correlation matrix among the multiple new image features, the cross-correlation matrix including the correlation degree between any two of the multiple new image features; obtain the spectral norm of the cross-correlation matrix, and determine a first loss value based on the spectral norm of the cross-correlation matrix.
[0190] In some embodiments, the spectral norm of the cross-correlation matrix is an important concept in matrix theory, describing the maximum stretching power of a matrix across all unit vectors. The spectral norm of the cross-correlation matrix refers to the maximum output length that the matrix can produce when applied to unit vectors. This value is equal to the matrix's largest singular value, which is the square root of the matrix's eigenvalues. The cross-correlation matrix is a matrix used in statistics to describe linear relationships between multiple variables. In image processing, particularly in deep learning models, the cross-correlation matrix is used to represent the correlation between multiple new image features in a sub-coding layer. Specifically, each element in the cross-correlation matrix represents the degree of correlation between a pair of new image features; these correlation degrees measure the strength and direction of the linear relationship between the two eigenvalues. The calculated correlation degrees are filled into the cross-correlation matrix. The diagonal elements of the matrix represent the correlation degree of each feature with itself, typically 1, because a feature is perfectly correlated with itself. The off-diagonal elements represent the correlation degree between different features. The cross-correlation matrix is a symmetric matrix.
[0191] As an example, the expression for the first loss value above can be:
[0192]
[0193] Where σ is used to indicate the spectral norm function, Σ μ Used to indicate the cross-correlation matrix. Used to indicate the transpose of the cross-correlation matrix.
[0194] In some embodiments, the determination of the first loss value based on the degree of correlation can be achieved by: traversing i and performing the following process: determining the first sub-loss value corresponding to the i-th degree of correlation based on the i-th degree of correlation; and determining the first loss value based on each of the first sub-loss values.
[0195] In some embodiments, determining the degree of correlation between any two i-th new image features can be achieved by performing the following processing on any two i-th new image features: determining the covariance between the i-th new image features and the standard deviation of each i-th new image feature; dividing the covariance between the i-th new image features by the product of the standard deviations of each i-th new image feature to obtain the degree of correlation between any two i-th new image features.
[0196] In some embodiments, the determination of the first sub-loss value corresponding to the i-th correlation degree based on the i-th correlation degree can be achieved as follows: constructing an i-th cross-correlation matrix among multiple i-th new image features based on the i-th correlation degree; and determining the first sub-loss value corresponding to the i-th correlation degree based on the spectral norm of the i-th cross-correlation matrix.
[0197] Thus, when using a single encoding layer to encode multiple image samples across multiple channels, constructing a cross-correlation matrix based on correlation levels comprehensively captures the linear relationships between multiple new image features. The spectral norm of the cross-correlation matrix, as the largest singular value of the matrix, reflects the overall correlation between new image features and the complexity of the model. By obtaining the spectral norm of the cross-correlation matrix and using it as part of the first loss value, the complexity of the image processing model can be effectively controlled, preventing overfitting. This ensures that the image processing model maintains a good fit to the training data while possessing stronger generalization ability, thereby exhibiting higher accuracy and robustness in practical applications.
[0198] In step 105, multiple new image features are fused to obtain fused features.
[0199] In some embodiments, when multi-channel coding is implemented through multiple serial sub-coding layers of an image processing model, step 105 above can be implemented as follows: perform the following processing on each of the sub-coding layers: fuse multiple new image features of the sub-coding layer to obtain the fused features of the sub-coding layer.
[0200] In some embodiments, when multi-channel coding is implemented through a single coding layer of an image processing model, step 105 above can be implemented as follows: multiple new image features of the coding layer are fused to obtain fused features of the coding layer.
[0201] In some embodiments, the above-described fusion of multiple new image features to obtain a fused feature can be achieved by traversing i and performing the following process to fuse multiple i-th new image features to obtain the i-th fused feature.
[0202] Thus, when multi-channel coding is implemented through multiple serial sub-coding layers in an image processing model, the fusion of multiple new image features from each sub-coding layer yields the fused features of that sub-coding layer. This effectively integrates the feature information extracted from different sub-coding layers, enhancing the representational power of the image processing model and making the fused features better reflect the complex structure and semantic information of the image samples. This allows the image processing model to not only capture new image features at different scales and levels but also reduce feature redundancy and improve the discriminative power of new image features.
[0203] In step 106, the fusion features are decoded to obtain the decoded image of the image sample.
[0204] In some embodiments, decoding refers to recovering or generating the original image or some form of representation thereof by inversely processing the encoded feature data. In deep learning, decoding involves using a decoder network that transforms the low-dimensional feature representation back into a high-dimensional image space. The decoding process can be used for various tasks, such as image segmentation, image classification, and image generation. In image segmentation tasks, decoding an image refers to converting the encoded feature map into a pixel-level segmentation mask that identifies the boundaries and categories of different objects or regions in the image. In image classification tasks, decoding refers to converting the feature representation into a category probability distribution to determine the image's category. In image generation tasks, decoding refers to generating a new image.
[0205] In some embodiments, see Figure 8 , Figure 8 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 5 , Figure 4 Step 106 shown can be achieved through Figure 8 Steps 1061 to 1062 shown are implemented.
[0206] In step 1061, for the fusion features of each sub-coding layer, feature transformation is performed on the fusion features of the sub-coding layers to obtain the transformed features of the sub-coding layers.
[0207] In some embodiments, feature transformation refers to the process in a deep learning model of changing the representation of features or extracting new feature representations by performing some form of transformation on the features. Feature transformation can include various operations such as linear transformation, nonlinear activation, feature enhancement, and feature dimensionality reduction, with the aim of improving the expressive power of features, reducing feature redundancy, increasing feature discriminative power, or adapting to the input requirements of subsequent model layers.
[0208] In some embodiments, when the sub-coding layer is the first sub-coding layer in the coding layers, the above-mentioned feature transformation of the fusion features of the sub-coding layer to obtain the transformed features of the sub-coding layer can be achieved in the following way: obtaining the random mask features of the fusion features of the sub-coding layer, wherein the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion features of the sub-coding layer; fusing the random mask features of the sub-coding layer into the fusion features of the sub-coding layer to obtain the transformed features of the sub-coding layer.
[0209] In some embodiments, when the sub-coding layer is the first sub-coding layer, the server acquires the fusion features of the sub-coding layer. These fusion features are obtained by fusing multiple new image features, which contain preliminary encoding information of the image. Next, the server generates a random mask feature with the same feature dimension as the fusion feature. The generation of the random mask feature can be achieved by randomly selecting or replacing certain elements in the fusion feature, aiming to simulate missing data or noise, thereby increasing the robustness of the image processing model. After generating the random mask feature, the server fuses it into the original fusion feature. This fusion can be a simple addition operation or a more complex operation, such as element-wise multiplication using a learnable weight matrix. The fused feature is called the transformed feature, which contains information from both the original fusion feature and the random mask feature. The server performs feature transformation on the fusion feature of the first sub-coding layer to obtain the transformed feature.
[0210] In some embodiments, random masking features are feature representations generated by randomly masking or replacing certain elements in the original features. In deep learning, particularly in the first sub-encoding layer within an encoding layer, random masking features are generated to simulate missing or noisy data, thereby enhancing the model's robustness and generalization ability. This masking process forces the model to learn more important features because it must extract useful information from the masked features to complete the task. The feature dimensions of the random masking features are the same as those of the original fused features, ensuring their compatibility during the fusion process. By fusing the random masking features into the original fused features, transformed features are obtained. These transformed features contain information from both the original and masked features, helping image processing models learn more comprehensive and robust feature representations during training.
[0211] As an example, see Figure 13 , Figure 13 This is a schematic diagram illustrating the principle of feature transformation provided in the embodiments of this application. When the sub-coding layer is the first sub-coding layer in the coding layer ( Figure 13 When W1 is shown above, the feature transformation of the fusion features of the sub-coding layer is performed to obtain the transformed features of the sub-coding layer. This can be achieved as follows: obtain the random mask features of the fusion features of the sub-coding layer, where the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion features of the sub-coding layer; fuse the random mask features of the sub-coding layer into the fusion features of the sub-coding layer (as shown in the figure). Figure 13 The figure shown ), to obtain the transformation features of the sub-coding layer ( Figure 13 The figure shown ).
[0212] Thus, when the sub-coding layer is the first sub-coding layer in the coding layer hierarchy, by obtaining random mask features from the fusion features and fusing them into the original fusion features to obtain transformed features, the robustness and generalization ability of the image processing model can be effectively enhanced. The introduction of mask features simulates missing or noisy data, forcing the image processing model to learn more important features during training and reducing dependence on specific features. Through this feature transformation, the image processing model can still extract effective information when faced with incomplete or noisy data, improving the adaptability and accuracy of the image processing model.
[0213] In some embodiments, when the sub-coding layer is not the first sub-coding layer in the coding layers, the above-mentioned feature transformation of the fusion feature of the sub-coding layer to obtain the transformed feature of the sub-coding layer can be implemented in the following way: perform two different linear transformations on the transformed feature of the fusion feature of the first sub-coding layer in the coding layers to obtain key features and value features; obtain the random mask feature of the fusion feature of the sub-coding layer, wherein the feature dimension of the random mask feature of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; fuse the random mask feature of the sub-coding layer into the fusion feature of the sub-coding layer to obtain the candidate transformed feature of the sub-coding layer; perform a linear transformation on the candidate transformed feature to obtain the query feature corresponding to the fusion feature; fuse the query feature corresponding to the fusion feature, the key feature, and the value feature to obtain the transformed feature of the sub-coding layer.
[0214] In some embodiments, when the sub-coding layer is not the first sub-coding layer in the coding layer, the server performs two different linear transformations on the transformed features of the fused features of the first sub-coding layer. These two linear transformations generate key features and value features, respectively. Key features and value features are key components of the attention mechanism, helping the model establish correlations between features at different levels. The server obtains a random mask feature of the fused features of the current sub-coding layer. This mask feature has the same feature dimension as the fused features. By generating the random mask feature, the server simulates missing data or noise, increasing the robustness of the model. The server fuses the random mask feature into the fused features of the current sub-coding layer to obtain candidate transformed features. The server performs a linear transformation on the candidate transformed features to obtain the query features corresponding to the fused features. Then, the server fuses the query features, key features, and value features to obtain the transformed features of the sub-coding layer. By utilizing the attention mechanism, the image processing model can extract the most relevant feature information from the value features based on the correlation between the query features and the key features.
[0215] As an example, see Figure 13 When the sub-coding layer is the second sub-coding layer in the coding layer, the transformation feature of the fusion feature of the first sub-coding layer in the coding layer ( Figure 13 The figure shown Perform two different linear transformations to obtain key features and value features; obtain the fusion features of the sub-coding layer. Figure 13 The figure shown The random mask features of the sub-coding layer are such that the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; the random mask features of the sub-coding layer are fused into the fusion feature of the sub-coding layer. Figure 13 The figure shown The candidate transformation features of the sub-coding layer are obtained; the candidate transformation features are linearly transformed to obtain the query features corresponding to the fusion features; the query features corresponding to the fusion features, the key features, and the value features are fused to obtain the transformation features of the sub-coding layer. Figure 13 As shown in F′2).
[0216] As an example, when the sub-coding layer is the third sub-coding layer in the coding layers, the transformation feature of the fusion feature of the first sub-coding layer in the coding layers ( Figure 13 The figure shown Perform two different linear transformations to obtain key features and value features; obtain the fusion features of the sub-coding layer. Figure 13 The figure shown The random mask features of the sub-coding layer are such that the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; the random mask features of the sub-coding layer are fused into the fusion feature of the sub-coding layer. Figure 13 The figure shown The candidate transformation features of the sub-coding layer are obtained; the candidate transformation features are linearly transformed to obtain the query features corresponding to the fusion features; the query features corresponding to the fusion features, the key features, and the value features are fused to obtain the transformation features of the sub-coding layer. Figure 13 The F′3 shown is given.
[0217] As an example, when the sub-coding layer is the third sub-coding layer in the coding layers, the transformation feature of the fusion feature of the first sub-coding layer in the coding layers ( Figure 13 The figure shown Perform two different linear transformations to obtain key features and value features; obtain the fusion features of the sub-coding layer. Figure 13 The figure shown The random mask features of the sub-coding layer are such that the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; the random mask features of the sub-coding layer are fused into the fusion feature of the sub-coding layer. Figure 13 The figure shown The candidate transformation features of the sub-coding layer are obtained; the candidate transformation features are linearly transformed to obtain the query features corresponding to the fusion features; the query features corresponding to the fusion features, the key features, and the value features are fused to obtain the transformation features of the sub-coding layer. Figure 13 The F′4 shown is given.
[0218] Thus, when the sub-coding layer is not the first sub-coding layer in the coding layer, the transformation features of the first sub-coding layer are subjected to two different linear transformations to generate key features and value features. These two features are the core of the attention mechanism, used to establish correlations between features at different levels. Next, a random mask feature of the fusion feature of the current sub-coding layer is obtained, with the same feature dimension as the fusion feature. The introduction of the random mask feature simulates missing data or noise, enhancing the robustness of the model. Then, the random mask feature is fused into the fusion feature to obtain candidate transformation features. Finally, a linear transformation is performed on the candidate transformation features to generate query features, which are then fused with the key and value features to obtain the final transformation features. By utilizing the attention mechanism and random mask, the image processing model can extract richer and more discriminative information from features at different levels, improving the performance and adaptability of the image processing model in complex tasks.
[0219] In step 1062, the transformation features of each of the sub-coding layers are fused to obtain target fused features, and the target fused features are decoded to obtain the decoded image of the image sample.
[0220] In some embodiments, in image segmentation applications, the transformation features of each of the sub-coding layers are fused to obtain target fusion features, which can be obtained by concatenating the transformation features of the sub-coding layers and using the concatenated features as the target fusion features.
[0221] In some embodiments, in the application scenario of image classification, the transformation features of each of the sub-coding layers are fused to obtain the target fusion feature. This can be achieved by concatenating the transformation features of the sub-coding layers and then performing average pooling on the concatenated features.
[0222] As an example, the expression for the above target fusion feature can be:
[0223] F=W1[F1′,F2′,F3′,F4′]+b1 (7)
[0224] Where F is used to indicate the target fusion feature, F′1, F′2, F′3 and F′4 are used to indicate the transformation features of each sub-coding layer, and b1 is used to indicate the adjustment parameter.
[0225] As an example, see Figure 13 Transformation features of each sub-coding layer ( Figure 13 The F′1, F′2, F′3 and F′4 shown are fused to obtain the target fusion features, and the target fusion features are decoded to obtain the decoded image of the image sample.
[0226] In some embodiments, step 1062 above can be implemented as follows: for each of the i-th fusion features, perform feature transformation on the i-th fusion feature to obtain the transformed feature of the i-th fusion feature; fuse each of the transformed features to obtain the target fusion feature, and decode the target fusion feature to obtain the decoded image of the image sample.
[0227] In some embodiments, when i equals 1, the feature transformation of the i-th fused feature to obtain the transformed feature of the i-th fused feature can be achieved in the following way: obtaining the random mask feature of the first fused feature, wherein the feature dimension of the random mask feature of the first fused feature is the same as the feature dimension of the first fused feature; fusing the random mask feature of the first fused feature into the first fused feature to obtain the transformed feature of the first fused feature.
[0228] In some embodiments, when i is greater than 1, the feature transformation of the i-th fused feature to obtain the transformed feature of the i-th fused feature can be implemented as follows: performing two different linear transformations on the transformed feature of the i-th fused feature to obtain key features and value features; obtaining the random mask feature of the i-th fused feature, wherein the feature dimension of the random mask feature of the i-th fused feature is the same as the feature dimension of the i-th fused feature; fusing the random mask feature of the i-th fused feature to the i-th fused feature to obtain candidate transformed features of the i-th fused feature; performing a linear transformation on the candidate transformed features to obtain the query feature corresponding to the i-th fused feature; fusing the query feature corresponding to the i-th fused feature, the key feature, and the value feature to obtain the transformed feature of the i-th fused feature.
[0229] Thus, by transforming the fusion features of each sub-coding layer, transformed features are obtained. After obtaining the transformed features of each sub-coding layer, these transformed features are fused to further integrate feature information from different levels, forming a more comprehensive and integrated target fusion feature. The generation of the target fusion feature ensures that the image processing model can fully utilize the feature information extracted from each sub-coding layer, improving the overall performance of the image processing model. Decoding the target fusion feature yields the decoded image of the image sample. The decoding process typically involves operations such as deconvolution layers, upsampling layers, and activation functions, converting the low-dimensional feature representation back to a high-resolution image space. The generation of the decoded image can be used for various tasks, such as image segmentation, image classification, and image generation. The image processing model can extract rich information from the features of multiple sub-coding layers, generating high-quality decoded images and improving the performance and effectiveness of the image processing model in image processing tasks.
[0230] In step 107, a second loss value is determined based on the decoded image and the image sample.
[0231] In some embodiments, step 107 above can be implemented as follows: determining a second loss value based on the decoded image and the sample labels carried by the image samples.
[0232] In some embodiments, see Figure 9 , Figure 9 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application. Figure 6 , Figure 4 Step 107 shown can be achieved through Figure 9 Steps 1071 to 1072 shown are implemented.
[0233] In step 1071, the label image of the image sample is obtained, and the label image is subjected to feature extraction to obtain the label image features.
[0234] In some embodiments, a labeled image refers to an image used to label regions of interest or objects in an image sample during image processing and computer vision tasks. These labeled images typically correspond to the original image samples, but they contain segmentation, classification, or other forms of annotation information for specific objects or regions in the image. Uses of labeled images include training data in supervised learning, helping image processing models learn how to identify and classify objects in images, and serving as ground truth values to evaluate model performance in image segmentation tasks.
[0235] In some embodiments, label image features are feature representations obtained by feature extraction from the label image. Feature extraction is the process of converting an image into numerical feature vectors that capture key information in the image, such as edges, textures, and colors. In computer vision tasks, feature extraction is a crucial step because it provides input data for subsequent machine learning algorithms. Label image features can be used to train image processing models, enabling them to learn how to extract features related to the label image from the original image, thereby improving the accuracy and generalization ability of the image processing model.
[0236] In step 1072, feature extraction is performed on the decoded image to obtain decoded image features, and the similarity between the label image features and the decoded image features is determined. The similarity is then used as the second loss value.
[0237] In some embodiments, feature extraction of the decoded image to obtain decoded image features refers to extracting representative feature information from the decoded image using feature extraction techniques (such as convolutional layers in a convolutional neural network). This feature information can be local patterns, textures, color distributions, etc., reflecting the main content and structure of the image. Label image features are obtained by feature extraction from the label image; these features represent the characteristics of regions of interest or objects in the image. Label images are typically used for supervised learning tasks as the target for training image processing models. Determining the similarity between label image features and decoded image features involves calculating the degree of similarity between the two feature representations using a similarity metric (such as mean squared error, cross-entropy, structural similarity index, etc.). This similarity reflects the degree of matching between the decoded image generated by the model and the label image. Using the similarity as a second loss value means incorporating the calculated similarity as a component of the loss function during model training. The loss function measures the difference between the image processing model's prediction and the actual label; the goal of the image processing model is to minimize the loss function through an optimization algorithm, thereby improving prediction accuracy. The second loss value is usually used to supplement the main loss function to guide the image processing model to better match the labeled image at the feature level, thereby improving the generalization ability and robustness of the image processing model.
[0238] As an example, the expression for the second loss value above can be:
[0239] L2 = sim(T1, T2) (8)
[0240] Where T1 is used to indicate the features of the labeled image, T2 is used to indicate the features of the decoded image, and L2 is used to indicate the second loss value.
[0241] Thus, by acquiring the labeled image of an image sample and extracting its features, label image features are obtained. Simultaneously, feature extraction is performed on the decoded image to obtain decoded image features. The similarity between the label image features and the decoded image features is then determined and used as the second loss value. This feature extraction process enables the image processing model to extract representative feature information from the image, reflecting its main content and structure. By calculating the similarity between the label image features and the decoded image features, the image processing model can understand the degree of matching between its generated decoded image and the label image, thereby guiding the optimization process. Using similarity as the second loss value allows the image processing model to consider both pixel-level and feature-level losses during training, thus improving its robustness and accuracy.
[0242] In step 108, the image processing model is trained based on the first loss value and the second loss value.
[0243] In some embodiments, step 108 can be implemented as follows: the first loss value and the second loss value are weighted and summed to obtain the target loss value, and the model parameters of the image processing model are updated based on the target loss value to obtain the target image processing model.
[0244] In some embodiments, the server needs to calculate a first loss value and a second loss value. The first loss value is typically calculated at the pixel level, such as mean squared error (MSE) or cross-entropy loss, to measure the difference between the model output and the true label. The second loss value is calculated at the feature level, such as feature similarity or perceptual loss, to measure the similarity between the model output and the true label in the feature space. The server performs a weighted sum of the first and second loss values to obtain the target loss value. The choice of weights depends on the specific task requirements and is usually determined experimentally. Larger weights mean that the loss value has a greater influence in the optimization process. Based on the target loss value, the server updates the model parameters of the image processing model. This is typically achieved through backpropagation and optimization algorithms (such as stochastic gradient descent (SGD), Adam, etc.). Backpropagation is used to calculate the gradient of the loss value with respect to the model parameters, and optimization algorithms are used to update the model parameters based on the gradient to minimize the target loss value. The server repeats the above process until the model converges or reaches a predetermined number of training epochs. After training, the target image processing model obtained by the server should be able to generate more accurate and realistic decoded images on new image samples.
[0245] As an example, the expression for the target loss value above can be:
[0246] L=α1L1+α2L2 (9)
[0247] Where L is used to indicate the target loss value, α1 and α2 are used to indicate the weights of the first loss value and the second loss value, L1 is used to indicate the first loss value, and L2 is used to indicate the second loss value.
[0248] Thus, by processing image samples, multiple image features corresponding to each channel are extracted. Since image features may contain redundant information, decorrelation operations are performed on the image features to reduce the linear dependence between them, thereby obtaining multiple new image features. This helps improve the independence between the new image features, enabling the image processing model to learn the intrinsic structure of the image samples more effectively. The correlation between any two new image features is calculated. Since the correlation reflects the strength and direction of the linear relationship between the new image features, a first loss value is determined based on the correlation. The first loss value can measure the correlation between features, thereby guiding the image processing model to learn more independent and discriminative features. After obtaining multiple new image features, the new image features are fused to integrate information from different channels, resulting in a comprehensive fused feature. Then, the fused feature is decoded to reconstruct the decoded image of the image sample. By comparing the decoded image with the original image sample, a second loss value is determined. The second loss value can measure the image processing performance of the image processing model, that is, whether the image processing model can accurately decode the decoded image from the fused features. The image processing model is trained based on the first and second loss values to obtain the target image processing model. By training with the first and second loss values, the image processing model can simultaneously improve the independence of image features and the accuracy of image processing, thereby effectively improving the image processing performance of the target image processing model.
[0249] See Figure 10 , Figure 10 This is a flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 , will combine Figure 10 Steps 201 to 204 are described below. The image processing method provided in this application embodiment can be implemented by the server or the terminal alone, or by the server and the terminal working together. The following description will take the implementation by the server alone as an example.
[0250] In step 201, the image to be processed is encoded in multiple channels using the target image processing model to obtain multiple image features corresponding to one channel.
[0251] In some embodiments, multi-channel encoding is implemented through the encoding layer of the target image processing model. The encoding layer includes multiple serial sub-encoding layers. The above-mentioned multi-channel encoding of the image to be processed by the target image processing model to obtain multiple image features corresponding to one channel can be achieved as follows: feature extraction is performed on the image to be processed to obtain the image features of the image to be processed; multi-channel encoding is performed iteratively multiple times through multiple serial sub-encoding layers to obtain multiple image features corresponding to one channel.
[0252] In step 202, decorrelation operation is performed on the image features of multiple channels using the target image processing model to obtain multiple new image features.
[0253] In some embodiments, the target image processing model receives image features from multiple channels as input. These channels may correspond to different color channels of the image (e.g., RGB channels), feature maps of different scales, or different types of feature representations (e.g., semantic features, texture features, etc.). The target image processing model is internally designed with specific decorrelation operations designed to reduce or eliminate statistical correlations between features from different channels. Decorrelation operations can be linear, such as Principal Component Analysis (PCA) or Independent Component Analysis (ICA), or non-linear, such as through specific neural network layer structures. For example, convolutional layers, normalization layers, activation functions, etc., are used to adjust the relationships between features. After decorrelation operations, the model outputs multiple new image features. These new features, while retaining the original feature information, reduce redundancy and correlation between features, enabling the model to more effectively utilize these features for subsequent tasks, such as image classification, object detection, and image segmentation.
[0254] In step 203, the multiple new image features are fused using the target image processing model to obtain fused features.
[0255] In some embodiments, the target image processing model receives multiple new image features as input. These new features may be obtained through decorrelation operations or extracted through other means, each containing information about different aspects of the image. The target image processing model is internally designed with feature fusion mechanisms designed to effectively combine multiple new features into a more comprehensive and discriminative feature representation. Feature fusion can be achieved through simple operations such as element-wise addition or concatenation, or through complex operations such as attention mechanisms, Feature Pyramid Networks (FPNs), or self-attention mechanisms. During feature fusion, the target image processing model learns how to weigh the importance of different features and how to combine them to capture complex patterns and structures in the image. For example, attention mechanisms can make the model focus more on important features and ignore less important ones during the fusion process. After the fusion operation, the target image processing model outputs a fused feature that contains information from multiple new features. This fused information is more synergistic and complementary, contributing to better performance of the target image processing model in subsequent tasks.
[0256] In step 204, the fused features are decoded to obtain the decoded image of the image.
[0257] In some embodiments, the target image processing model is trained using the image processing model training method of the present application embodiment. The fused features are then decoded using the target image processing model to obtain the fused features.
[0258] As an example, in medical image analysis applications, images acquired using different imaging techniques (such as MRI, CT, and PET) may be used as inputs for different channels. By decorrelation operations, noise and redundant information introduced by different imaging techniques can be reduced, and the unique features of each technique can be extracted. The fused features can incorporate the advantages of multiple imaging techniques, providing more comprehensive diagnostic information. The decoded images can then be used to assist doctors in disease diagnosis, such as tumor detection and tissue segmentation.
[0259] As an example, in satellite image processing applications, satellite images may contain multiple spectral channels, such as visible light, near-infrared, and short-wave infrared. Decorrelation operations can help remove the correlation between different spectral channels and extract the unique information of each channel. The fused features can be used for tasks such as land use classification, environmental monitoring, and disaster assessment. The decoded image can provide higher resolution or clearer geographic information.
[0260] As an example, in applications of autonomous driving technology, self-driving cars may use data from multiple sensors (such as cameras, LiDAR, and millimeter-wave radar) as inputs to different channels. Decorrelation operations can reduce redundancy between data from different sensors and extract the unique information from each sensor. The fused features can provide a more comprehensive environmental perception, helping the vehicle make more accurate decisions. The decoded images can be used for real-time road condition analysis, obstacle detection, and more.
[0261] Thus, by using a target image processing model to encode the image to be processed through multiple channels, multiple image features corresponding to each channel are obtained. This process effectively extracts information from different aspects of the image, such as color, texture, and shape. Subsequently, by performing decorrelation operations on the image features of multiple channels, redundancy and correlation between features can be reduced, making each new image feature more independent and discriminative. Next, the target image processing model fuses multiple new image features, combining these independent feature information to form a more comprehensive and discriminative fused feature. Finally, by decoding the fused feature, the decoded image can be obtained, thereby converting the low-dimensional feature representation back to a high-resolution image space. This effectively reduces feature redundancy by eliminating the correlation between image features of different channels, and by integrating image information, improves the image processing capabilities of subsequent image decoding using new image features with eliminated correlation between image features of different channels, thus significantly enhancing the image processing performance of the image processing task.
[0262] The following will describe an exemplary application of the embodiments of this application in a real medical image processing application scenario.
[0263] Medical image processing applies digital image processing techniques to medical images to improve image quality or extract information valuable for diagnosis and treatment. It involves the analysis, enhancement, segmentation, registration, fusion, feature extraction, and interpretation of images generated by various medical imaging techniques (such as X-ray, CT, MRI, ultrasound, and nuclear medicine).
[0264] To address the stylistic differences in images from various hospitals in real-world scenarios, deep neural networks learn similar patterns across multiple channels of the same feature layer. This inevitably introduces feature redundancy. Decoupling the correlations between channels can reduce redundancy and maximize the network's expressive power, thereby improving generalization.
[0265] This application proposes a loss function constraint for domain-generalized medical image representation learning. The main function of this constraint is to reduce the correlation between channels within the same feature layer, thereby reducing the redundancy of feature representations. Furthermore, the reduction in redundancy also encourages the image processing model to learn more diverse feature representations. This application can effectively improve the accuracy of image analysis (including classification, segmentation, and other tasks) in domain-generalized scenarios.
[0266] In some embodiments, see Figure 15 , Figure 15 This is a schematic diagram illustrating the effect of a medical image provided in an embodiment of this application, demonstrating the feature redundancy phenomenon in cross-domain medical image representation learning. Here, D0-D3 represent four different domains. Figure 15 Part (a) shows cross-domain channel alignment, where the same feature channel has different activation locations in images of different domains. Figure 15Part (b) illustrates intra-domain cross-channel redundancy, where multiple channels of the same feature layer capture the same representation. This indicates feature redundancy between network channels, with the network capturing features from different domains through different channels. This approach obviously limits the model's domain generalization ability. Different activation positions of the same feature channel in different domains means that within the same feature channel, the model activates at different locations for images from different domains. For example, D1-D3 represent four different domains; the same feature channel might be activated in one region of an image in domain D1, but in another region of an image in domain D2. This phenomenon shows that the model uses the same feature channel to capture different information in different domains, which may lead to wasted resources in the image processing model during learning, as the same channel could be more focused on capturing a specific type of information. Multiple channels of the same feature layer capturing the same representation means that within the same layer, multiple feature channels may capture the same or very similar feature information. For example, multiple channels may all be capturing edge or texture information in the image. This redundancy reduces the model's efficiency because multiple channels are doing similar work, when only one channel is actually needed to capture this information.
[0267] The embodiments of this application can be used in different medical image analysis scenarios. Common medical image analysis tasks include image classification and segmentation. Figure 1 The document showcases fundus images from different domains (hospitals), enabling simultaneous multi-task (feature channels) performance based on these images, including segmentation (optic cup / disc, lesion segmentation, etc.) and classification (diabetic retinopathy staging, glaucoma identification). Currently, most medical image analysis methods are trained and tested on data from a single center. However, in real-world scenarios, models may be trained using data from one or several hospitals and deployed in hospitals outside the training data range. Data from centers not present in the training may exhibit significantly different styles (brightness, contrast, etc.) compared to the training data, leading to instability in the cross-domain performance of the network model. To address this issue, this application proposes a domain-generalized representation learning algorithm that more effectively learns features with domain invariance.
[0268] In real-world applications of medical image analysis, we often encounter data from multiple centers. These data vary between centers, and some centers' image styles are not present in the training data. Because different centers use different acquisition equipment and have different acquisition standards, cross-center images are prone to style discrepancies. A significant negative impact of this problem on deep neural networks is feature misalignment. Specifically, due to the aforementioned domain offset issue, medical images from different centers may exhibit drastically different activation patterns in the same channel of the deep network model (e.g., ...). Figure 15As shown in section (a) of the diagram. Feature misalignment is particularly pronounced in shallow features, as these features are more sensitive to changes in image color, brightness, and contrast. To capture the necessary information in each domain, the network tends to learn similar patterns across multiple channels, further exacerbating feature redundancy—that is, exhibiting the same activation patterns across multiple channels (e.g., ...). Figure 15 (As shown in part (b) of the document). Feature redundancy helps the model capture the necessary features on training data in different domains even with single-channel misalignment, but this property cannot guarantee the robustness of the model when styles not found in the training set appear in the test set. The primary objective of the embodiments of this application is to mitigate the degree of feature redundancy, which helps image processing models build more expressive cross-domain medical image representations and improve their generalization ability in unseen target domains.
[0269] The design and verification of this application's embodiments utilize three publicly available datasets. Two datasets are used for the segmentation task: one is a fundus segmentation dataset of fundus photographs, containing data from four centers; the other is a target area segmentation dataset of MRI images, consisting of images from six centers. Additionally, there is a classification task dataset, a pathological image dataset, which comprises images from five centers.
[0270] In some embodiments, see Figure 16 , Figure 16 This is a schematic diagram illustrating the principle of the training method for the image processing model provided in this application. The image is first input to a feature encoding part consisting of four Transformer Blocks, and the outputs of the four modules are denoted as F1, F2, F3, and F4, respectively. Then, a channel-to-channel decorrelation operation is performed on the outputs of these four modules, and the decorrelated features are denoted as... and When decoding high-level features, this embodiment employs a mutual attention mechanism between high-level and low-level features. The mutual attention module, similar to the self-attention module, is also based on the query, key, and value for computation. The query is derived from deep features. and Generates, while keys and values are based on shallow features. Generate. Specifically, for and This application embodiment generates a query through a single linear transformation:
[0271]
[0272] Key and Value are based on the decorrelated underlying features. generate:
[0273]
[0274] Decoding high-level features involves a mutual attention process:
[0275] F=W1[F′1,F′2,F′3,F′4]+b1 (12)
[0276] In some embodiments, the probability of each class is output after passing through a Softmax layer. Similarly, in prediction for classification tasks, embodiments of this application perform average pooling on the stacked [F′1,F′2,F′3,F′4], and then use this linear layer to output the probability of the class. Based on... and Before calculating the query, key, and value, this embodiment of the application randomly generates a mask vector M, the dimension of which is the number of channels of its target feature. This mask vector is directly multiplied by its target feature, and... and Some channels in the image are set to zero. This change aims to help the image processing model learn more important features. During feature decorrelation, the image processing model is required to learn more diverse feature representations. However, to achieve higher performance, in addition to increasing diversity, the effectiveness of the newly learned feature representations must also be ensured. Therefore, a random channel mask is applied to the decorrelated features, requiring the image processing model to complete the final task with different channel combinations. This allows the image processing model to learn more effective feature representations.
[0277] In some embodiments, see Figure 16 To address the distribution differences from images with different centers (distribution differences between different tasks, such as task A: WSI classification, task B: grading, tasks C and D: segmentation, etc.), deep neural networks tend to extract similar patterns across multiple channels to improve robustness. However, this inevitably introduces feature redundancy. Decoupling the correlation between channels can reduce redundancy and maximize the expressive power of each channel, thereby improving the generalization of features. Therefore, this application proposes a finite isometric constraint loss function for network channel decoupling (…). Figure 16 The first loss value shown is illustrated.
[0278] For the cross-correlation matrix Σ between channels μ Its finite isometry property can be expressed as:
[0279]
[0280] Where z is For any vector in Σ, C is Σ μ The number of columns; A real number between 0 and 1. The smaller the value, the more Σ μThe closer the columns are to being pairwise orthogonal, the better. When Σ is 0, μ The column vectors are mutually perpendicular. When the left side of formula 11 is used as a constraint in this embodiment of the application, and its value is continuously reduced during training, the column vectors of the cross-correlation matrix become increasingly closer to being mutually perpendicular. In addition, before calculating the cross-correlation matrix, the features are standardized, at which point their diagonal elements are 1. The orthogonal constraint of the above formula (13) makes the cross-correlation matrix increasingly closer to the identity matrix. This embodiment of the application uses... Represents Σ μ The spectral norm, i.e., Σ μ The largest eigenvalue, then the condition in formula 11 is satisfied. The minimum value can be expressed as:
[0281]
[0282] By minimizing Σ μ As the column vectors become more perpendicular, meaning their cross-correlation matrix approaches the identity matrix, the similarity between channels decreases. This embodiment of the application refers to this as a finite isometric constraint (the first loss value described above):
[0283]
[0284] Similarly, in this embodiment, the loss function is calculated for the output of the four Transformer Blocks using the formula (15) above. and Finite equidistant constraints require that all eigenvalues of the cross-correlation matrix be 1, and focus more on rows and columns with high similarity in the cross-correlation matrix, which can achieve better decorrelation results.
[0285] Thus, the algorithm was tested on three tasks. The first task was optic cup-optic disc segmentation on fundus photographs, using a dataset from four centers. During testing, the network was trained on three centers and tested on the remaining centers. The second task was target segmentation on MRI, using a dataset from six centers, similarly trained on five centers and tested on one. For evaluation metrics, this embodiment selected Dice Similarity Coefficient (metric 1) and Average SurfaceDistance (metric 2). Higher values for the former are better, and lower values for the latter are better. The third task was pathological image classification, determining whether cell metastasis was present in the pathological image of the target site. This embodiment trained on the data from the first center and tested the domain generalization ability on the remaining four centers. This embodiment used metrics illuminance (metric 1) and Areaunder Curve (metric 2) as evaluation metrics.
[0286] Table 1 compares the accuracy of the embodiments of this application with related technical solutions in the first segmentation task.
[0287]
[0288] Table 2 Comparison of accuracy between this application and related technical solutions in the second segmentation task
[0289]
[0290] Table 3 Comparison of accuracy between this application and related technical solutions in pathological image classification tasks.
[0291]
[0292] Referring to Tables 1 to 3 above, Table 1 shows the results comparison of the related technologies of this application on the first segmentation task. On all six test centers, the embodiments of this application outperform the related technologies, with an average improvement of 0.63%. Table 2 shows the results comparison on the second segmentation task. In this task, the embodiments of this application outperform the related technologies on all four test centers, with an average improvement of more than 1%. In the image classification task, the embodiments of this application also achieved higher accuracy than the related technologies on all test centers, with an average accuracy improvement of approximately 1.7%.
[0293] In some embodiments, see Figure 17 , Figure 17 This is a schematic diagram illustrating the principle of the covariance matrix provided in the embodiments of this application. To demonstrate the effectiveness of feature decorrelation in this application, the relevant features are visualized. and The covariance matrix. Without any feature decoupling method (related technique A), the covariance values between features are large, indicating high correlation between channels. After adding the original version of the depth whitening transformation (related technique B), the coupling between features is reduced, but it is still very obvious in shallow layers. Related techniques C and D can further reduce the values of the off-diagonal part of the covariance matrix, but compared with them, the embodiments of this application can still significantly improve this effect, which shows that... Figure 17 The grayscale of the last column in the image is noticeably darker.
[0294] In some embodiments, see Figure 18 , Figure 18 This is a visual distribution diagram of the feature distribution provided in the embodiments of this application, comparing the distribution of data in different domains in the feature space after applying various decorrelation constraints. The segmentation dataset using the target region contains a total of 6 centers ( Figure 18The domains shown are used as follows: the first five are used as the training set, and the sixth is used as the test set. In the visualization results of related technologies (related technology A, related technology B, related technology C, and related technology D), the data of the sixth center always shows clustering in the feature space at different encoding stages, i.e., the circular dots in the figure. In contrast, after applying the constraints proposed in the embodiments of this application, the circular dots no longer cluster, indicating that the learned feature representation has better inter-domain generality.
[0295] It is understood that in the embodiments of this application, data such as image samples are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0296] The following description continues to illustrate the exemplary structure of the image processing model training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the training device 455 of the image processing model stored in the memory 450 may include: a decorrelation module, used to encode the image sample in multiple channels to obtain multiple image features corresponding to one channel, and to perform decorrelation operation on the multiple image features through the image processing model to obtain multiple new image features; a first loss module, used to determine the degree of correlation between any two of the new image features, and to determine a first loss value based on the degree of correlation; a decoding module, used to fuse the multiple new image features to obtain fused features, and to decode the fused features to obtain a decoded image of the image sample; and a training module, used to determine a second loss value based on the decoded image and the image sample, so as to train the image processing model based on the first loss value and the second loss value.
[0297] In some embodiments, the multi-channel encoding described above is implemented through the encoding layer of the image processing model, the encoding layer including multiple serial sub-encoding layers; the decorrelation module is further configured to extract features from the image samples to obtain initial image features of the image samples; through the multiple serial sub-encoding layers, the initial image features are iteratively encoded multiple times to obtain multiple image features corresponding to one channel respectively; the decorrelation module is further configured to take the image features output by each sub-encoding layer as layer image features, and perform the following processing on the layer image features output by the multiple sub-encoding layers respectively: perform decorrelation operation on the layer image features of multiple different channels output by the sub-encoding layers to obtain multiple new image features of the sub-encoding layers.
[0298] In some embodiments, the decorrelation module is further configured to, for each sub-coding layer, determine the degree of correlation between any two new image features among the plurality of new image features of the sub-coding layer; the first loss module is further configured to, for each sub-coding layer, determine a first sub-loss value of the sub-coding layer based on the corresponding degree of correlation; and determine a first loss value based on the first sub-loss value of each sub-coding layer.
[0299] In some embodiments, the plurality of new image features in the sub-coding layer include a first new image feature and a second new image feature; the first loss module is further configured to determine the covariance between the first new image feature and the second new image feature, and the standard deviations corresponding to the first new image feature and the second new image feature respectively; obtain the product of the standard deviations corresponding to the first new image feature and the second new image feature respectively, and divide the covariance between the first new image feature and the second new image feature by the product to obtain the correlation degree between the first new image feature and the second new image feature.
[0300] In some embodiments, the first loss module is further configured to construct a cross-correlation matrix of the sub-coding layer based on the correlation degree, the cross-correlation matrix including the correlation degree between any two new image features among the plurality of new image features of the sub-coding layer; obtain the spectral norm of the cross-correlation matrix; and determine the first sub-loss value of the sub-coding layer based on the spectral norm of the cross-correlation matrix.
[0301] In some embodiments, the decoding module is further configured to perform the following processing on each of the sub-coding layers: fusing multiple new image features of the sub-coding layer to obtain fused features of the sub-coding layer; the decoding module is further configured to perform feature transformation on the fused features of the sub-coding layer to obtain transformed features of the sub-coding layer; fusing the transformed features of each of the sub-coding layers to obtain target fused features, and decoding the target fused features to obtain a decoded image of the image sample.
[0302] In some embodiments, when the sub-coding layer is the first sub-coding layer in the coding layers, the decoding module is further configured to obtain the random mask feature of the fusion feature of the sub-coding layer, wherein the feature dimension of the random mask feature of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; and fuse the random mask feature of the sub-coding layer into the fusion feature of the sub-coding layer to obtain the transformation feature of the sub-coding layer.
[0303] In some embodiments, when the sub-coding layer is not the first sub-coding layer in the coding layers, the decoding module is further configured to perform two different linear transformations on the transformation features of the fusion features of the first sub-coding layer in the coding layers to obtain key features and value features; obtain random mask features of the fusion features of the sub-coding layer, wherein the feature dimension of the random mask features of the sub-coding layer is the same as the feature dimension of the fusion features of the sub-coding layer; fuse the random mask features of the sub-coding layer into the fusion features of the sub-coding layer to obtain candidate transformation features of the sub-coding layer; perform a linear transformation on the candidate transformation features to obtain query features corresponding to the fusion features; and fuse the query features corresponding to the fusion features, the key features, and the value features to obtain the transformation features of the sub-coding layer.
[0304] In some embodiments, the decorrelation module is further configured to perform decorrelation operation on the plurality of image features through the image processing model to obtain a plurality of candidate image features; and to perform standardization processing on the plurality of candidate image features with dimensional differences to obtain a plurality of new image features without the dimensional differences.
[0305] In some embodiments, the candidate image features include multiple feature elements. The aforementioned decorrelation module is further configured to perform the following processing on each of the candidate image features: determine the mean of the feature values of the multiple feature elements and the standard deviation of the feature values of the multiple feature elements; subtract the mean from the feature value of each feature element in the candidate image features to obtain the target image features; and divide the feature value of each feature element in the target image features by the standard deviation to obtain the new image features.
[0306] In some embodiments, the first loss module described above is further configured to perform the following processing for any two new image features: determine the covariance between the new image features and the standard deviation of each new image feature; obtain the product value of the standard deviations of each new image feature; divide the covariance between the new image features by the product value to obtain the correlation degree between any two new image features.
[0307] In some embodiments, the first loss module is further configured to construct a cross-correlation matrix among the plurality of new image features based on the correlation degree, the cross-correlation matrix including the correlation degree between any two of the plurality of new image features; obtain the spectral norm of the cross-correlation matrix, and determine a first loss value based on the spectral norm of the cross-correlation matrix.
[0308] In some embodiments, the training module is further configured to acquire the label image of the image sample, extract features from the label image to obtain label image features; extract features from the decoded image to obtain decoded image features, determine the similarity between the label image features and the decoded image features, and determine the similarity as a second loss value.
[0309] The following description continues to illustrate the exemplary structure of the image processing apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 3 As shown, the software modules stored in the image processing device 555 in the memory 550 may include: an encoding module, used to encode the image to be processed through a target image processing model in multiple channels to obtain multiple image features corresponding to one channel, and to perform decorrelation operation on the multiple image features of the multiple channels through the target image processing model to obtain multiple new image features; a decoding module, used to fuse the multiple new image features through the target image processing model to obtain fused features, and to decode the fused features to obtain a decoded image of the image; wherein, the target image processing model is trained using the above-described image processing model training method.
[0310] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image processing model training method and image processing method described in this application embodiment.
[0311] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the training method and image processing method of the image processing model provided in this application. For example, ... Figure 4 The training method for the image processing model is shown.
[0312] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of electronic devices including one or any combination of the above-mentioned memories.
[0313] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.
[0314] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).
[0315] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.
[0316] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0317] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.
[0318] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.
[0319] In summary, the embodiments of this application have the following beneficial effects:
[0320] (1) By processing image samples, multiple image features corresponding to one channel are extracted. Since image features may contain redundant information, the image processing model performs decorrelation operations on the image features to reduce the linear dependence between image features, thereby obtaining multiple new image features. This helps to improve the independence between new image features, enabling the image processing model to learn the internal structure of the image samples more effectively. The correlation degree between any two new image features is calculated. Since the correlation degree reflects the strength and direction of the linear relationship between new image features, a first loss value is determined based on the correlation degree. The first loss value can measure the correlation between features, thereby guiding the image processing model to learn more independent and discriminative features. After obtaining multiple new image features, the new image features are fused to integrate information from different channels and obtain a comprehensive fused feature. Then, the fused feature is decoded to reconstruct the decoded image of the image sample. By comparing the decoded image with the original image sample, a second loss value is determined. The second loss value can measure the image processing performance of the image processing model, that is, whether the image processing model can accurately decode the decoded image from the fused features. The image processing model is trained based on the first and second loss values. By training with the first and second loss values, the image processing model can simultaneously improve the independence of image features and the accuracy of image processing, thereby effectively improving the image processing performance of the target image processing model.
[0321] (2) Through these sub-coding layers, the initial image features are iteratively encoded in multiple channels. In each iteration, the sub-coding layer focuses on extracting specific types of features, such as color, texture, and shape. The iterative process allows multiple image features corresponding to one channel to capture different aspects of information in the image. Multiple serial sub-coding layers are used to iteratively encode the initial image features in multiple channels. In each iteration, different sub-coding layers can analyze and transform features from different angles and scales. Through this multi-channel approach, different types of features in the image can be fully explored, enriching the expressive power of the features. Each sub-coding layer operates at different scales and feature spaces. The first sub-coding layer may extract local detail features of the image through small convolutional kernels. As the iteration progresses, subsequent sub-coding layers will integrate the previous features and further extract more abstract and semantic features. Multi-channel encoding is like observing the image from multiple perspectives, so that the final multiple image features corresponding to one channel can comprehensively and deeply describe the image.
[0322] (3) By using the image features output by each sub-coding layer as layer image features and performing decorrelation operations on the layer image features output by multiple sub-coding layers respectively, the performance of image recognition and classification can be significantly improved. The decorrelation operation reduces or eliminates the correlation between features of different channels, making each new image feature mathematically independent, thereby improving the discriminativeness and uniqueness of the new image features. The orthogonality between the new image features enhances the sensitivity of the image processing model to subtle differences in image samples, reduces feature redundancy, and thus improves the generalization ability of the image processing model.
[0323] (4) By standardizing multiple candidate image features with dimensional differences, the dimensional differences between feature values can be eliminated, making each new image feature numerically identical in scale and distribution. The mean and standard deviation of the feature values of multiple feature elements in each candidate image feature are calculated. Then, the mean is subtracted from the feature value of each feature element to obtain the target image feature. Finally, the feature value of each feature element in the target image feature is divided by the standard deviation to obtain the standardized new image feature. Standardization ensures that the mean of the new image feature is zero and the standard deviation is one, thereby eliminating dimensional differences between features and improving the comparability of features and the training efficiency of the model. Simultaneously, standardization can also reduce the correlation between features, improving the generalization ability and accuracy of the image processing model.
[0324] (5) Performing decorrelation operations on multiple image features through image processing models can effectively reduce redundant information between features, improve feature discriminativeness, and enhance the model's generalization ability. However, since image features typically have different dimensions, this may lead to the model assigning excessive weights to certain features during training, thus affecting model performance. Therefore, standardization of multiple candidate image features with different dimensions is necessary. Standardization transforms all features to the same dimension, typically converting them to a standard normal distribution with a mean of 0 and a standard deviation of 1. This helps the image processing model learn the importance of each feature more fairly, improving the training efficiency and accuracy of the image processing model.
[0325] (6) By determining the covariance between new image features and the standard deviation of each new image feature, and further calculating their product of standard deviations, the correlation between any two new image features can be obtained. The calculation of the correlation not only eliminates the influence of feature dimensions, making the relationship between different features comparable, but also provides an important basis for feature selection and dimensionality reduction. The correlation can help identify redundant features, reduce the complexity of the model, and improve the generalization ability and prediction accuracy of the image processing model.
[0326] (7) By determining the covariance and standard deviation of the first and second new image features in the sub-coding layer, and calculating the product of the standard deviations, the correlation between the two features is obtained by dividing the covariance by this product. This not only eliminates the influence of feature dimensions, making the relationships between different features comparable, but also provides an important basis for feature selection and dimensionality reduction. In image processing and machine learning, the degree of correlation can help identify redundant features, reduce the complexity of image processing models, and improve the generalization ability and prediction accuracy of image processing models.
[0327] (8) By constructing the cross-correlation matrix of the sub-coding layer based on the degree of correlation, the linear relationship between multiple new image features can be comprehensively captured. This not only helps to understand the interaction between features, but also provides key information for feature selection and dimensionality reduction. The spectral norm of the cross-correlation matrix, as the largest singular value of the matrix, reflects the overall size and potential complexity of the matrix. By obtaining the spectral norm of the cross-correlation matrix and using it as part of the first sub-loss value of the sub-coding layer, the complexity of the image processing model can be effectively controlled, preventing overfitting.
[0328] (9) For each sub-coding layer, the first sub-loss value is determined based on the corresponding correlation degree. This allows for an accurate measurement of the linear relationship strength between new image features in that sub-coding layer and the calculation of the spectral norm of the cross-correlation matrix. This controls the complexity of the image processing model and prevents overfitting. By combining the first sub-loss values of each sub-coding layer to determine the first loss value, not only can the overall performance of the image processing model be comprehensively evaluated, but the optimization direction of the image processing model can also be guided.
[0329] (10) When multi-channel encoding of multiple image samples is achieved through a single encoding layer, the method of constructing a cross-correlation matrix based on the degree of correlation can comprehensively capture the linear relationship between multiple new image features. The spectral norm of the cross-correlation matrix, as the largest singular value of the matrix, reflects the overall correlation between new image features and the complexity of the model. By obtaining the spectral norm of the cross-correlation matrix and using it as part of the first loss value, the complexity of the image processing model can be effectively controlled, preventing overfitting. This ensures that the image processing model maintains a good fit to the training data while possessing stronger generalization ability, thus exhibiting higher accuracy and robustness in practical applications.
[0330] (11) When multi-channel coding is implemented through multiple serial sub-coding layers of an image processing model, multiple new image features of each sub-coding layer are fused to obtain the fused features of the sub-coding layer. This effectively integrates the feature information extracted from different sub-coding layers, improves the representational capability of the image processing model, and makes the fused features better reflect the complex structure and semantic information of the image samples. This enables the image processing model to not only capture new image features at different scales and levels, but also to reduce feature redundancy and improve the discriminative power of new image features.
[0331] (12) When the sub-coding layer is the first sub-coding layer in the coding layer, by obtaining the random mask features of the fusion features and fusing them into the original fusion features, the transformed features can be obtained, which can effectively enhance the robustness and generalization ability of the image processing model. The introduction of mask features simulates the missing or noisy data, forcing the image processing model to learn more important features during training and reducing the dependence on specific features. Through this feature transformation, the image processing model can still extract effective information when faced with incomplete or noisy data, improving the adaptability and accuracy of the image processing model.
[0332] (13) When the sub-coding layer is not the first sub-coding layer in the coding layer, the transformation features of the first sub-coding layer are subjected to two different linear transformations to generate key features and value features. These two features are the core of the attention mechanism and are used to establish correlations between features at different levels. Next, the random mask features of the fusion features of the current sub-coding layer are obtained. The feature dimension is the same as that of the fusion features. By introducing random mask features, the missing data or noise is simulated, which enhances the robustness of the model. Then, the random mask features are fused into the fusion features to obtain candidate transformation features. Finally, the candidate transformation features are subjected to linear transformation to generate query features, which are then fused with key features and value features to obtain the final transformation features. By utilizing the attention mechanism and random mask, the image processing model can extract richer and more discriminative information from features at different levels, thereby improving the performance and adaptability of the image processing model in complex tasks.
[0333] (14) By performing feature transformation on the fusion features of each sub-coding layer, transformed features are obtained. After obtaining the transformed features of each sub-coding layer, these transformed features are fused to further integrate feature information from different levels, forming a more comprehensive and integrated target fusion feature. The generation of the target fusion feature ensures that the image processing model can fully utilize the feature information extracted from each sub-coding layer, improving the overall performance of the image processing model. The target fusion feature is decoded to obtain the decoded image of the image sample. The decoding process usually involves operations such as deconvolution layers, upsampling layers, and activation functions to convert the low-dimensional feature representation back to the high-resolution image space. The generation of the decoded image can be used for various tasks, such as image segmentation, image classification, and image generation. The image processing model can extract rich information from the features of multiple sub-coding layers, generate high-quality decoded images, and improve the performance and effect of the image processing model in image processing tasks.
[0334] (15) By acquiring the label images of image samples and performing feature extraction, label image features are obtained. Simultaneously, feature extraction is performed on the decoded images to obtain decoded image features. Then, the similarity between the label image features and the decoded image features is determined, and the similarity is set as the second loss value. The feature extraction process enables the image processing model to extract representative feature information from the image, which reflects the main content and structure of the image. By calculating the similarity between the label image features and the decoded image features, the image processing model can understand the degree of matching between the generated decoded image and the label image, thereby guiding the optimization process of the image processing model. Setting the similarity as the second loss value allows the image processing model to consider both pixel-level loss and feature-level loss during training, thereby improving the robustness and accuracy of the image processing model.
[0335] (16) The target image processing model encodes the image to be processed through multiple channels, obtaining multiple image features corresponding to each channel. This process can effectively extract information from different aspects of the image, such as color, texture, and shape. Subsequently, by performing decorrelation operations on the image features of multiple channels, redundancy and correlation between features can be reduced, making each new image feature more independent and discriminative. Next, the target image processing model fuses multiple new image features, combining these independent feature information to form a more comprehensive and discriminative fused feature. Finally, by decoding the fused feature, the decoded image can be obtained, thereby converting the low-dimensional feature representation back to the high-resolution image space. This effectively reduces feature redundancy by eliminating the correlation between image features of different channels, and by integrating image information, improves the image processing of subsequent image decoding by using new image features that have eliminated the correlation between image features of different channels.
[0336] (17) The algorithm was tested on three tasks. The first task was optic cup-optic disc segmentation on fundus photographs, using a dataset from 4 centers. During testing, the network was trained on 3 centers and tested on the remaining centers each time. The second task was target segmentation on MRI, using a dataset from 6 centers. Similarly, it was trained on five centers and tested on one center. For the evaluation metrics, this embodiment selected Dice Similarity Coefficient (metric 1) and Average SurfaceDistance (metric 2). The higher the former, the better, and the lower the latter, the better. The third task was pathological image classification, determining whether there was metastasis of cancer cells in the pathological image of the target site. This embodiment trained on the data from the first center and tested the domain generalization ability on the remaining four centers. This embodiment used metrics illuminance (metric 1) and Areaunder Curve (metric 2) as evaluation metrics.
[0337] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.
Claims
1. A training method for an image processing model, characterized in that, The method includes: The image samples are encoded in multiple channels to obtain multiple image features corresponding to one channel, and the image processing model is used to perform decorrelation operation on the multiple image features to obtain multiple new image features. Determine the degree of correlation between any two of the new image features, and determine a first loss value based on the degree of correlation; Multiple new image features are fused to obtain fused features, and the fused features are decoded to obtain a decoded image of the image sample; Based on the decoded image and the image sample, a second loss value is determined, and the image processing model is trained based on the first loss value and the second loss value.
2. The method according to claim 1, characterized in that, The multi-channel encoding is implemented through the encoding layer of the image processing model, which includes multiple sequential sub-encoding layers; the multi-channel encoding of image samples to obtain multiple image features corresponding to one channel includes: Feature extraction is performed on the image samples to obtain the initial image features of the image samples; The initial image features are iteratively encoded multiple times through the multiple serial sub-coding layers to obtain multiple image features corresponding to one channel respectively. The decorrelation operation is performed on the multiple image features to obtain multiple new image features, including: The image features output by each of the sub-coding layers are used as layer image features, and the following processing is performed on the layer image features output by the multiple sub-coding layers respectively: Perform decorrelation operation on the layer image features of multiple different channels output by the sub-coding layer to obtain multiple new image features of the sub-coding layer.
3. The method according to claim 2, characterized in that, Determining the correlation between any two of the new image features includes: For each sub-coding layer, determine the degree of correlation between any two new image features among the plurality of new image features in the sub-coding layer; Determining the first loss value based on the degree of correlation includes: For each of the sub-coding layers, a first sub-loss value for the sub-coding layer is determined based on the corresponding degree of correlation. The first loss value is determined based on the first sub-loss value of each of the sub-coding layers.
4. The method according to claim 3, characterized in that, The plurality of new image features in the sub-coding layer include a first new image feature and a second new image feature; determining the correlation degree between any two of the plurality of new image features in the sub-coding layer includes: Determine the covariance between the first new image feature and the second new image feature, and the standard deviations corresponding to the first new image feature and the second new image feature, respectively; Obtain the product of the standard deviations corresponding to the first new image feature and the second new image feature respectively, and divide the covariance between the first new image feature and the second new image feature by the product value to obtain the correlation degree between the first new image feature and the second new image feature.
5. The method according to claim 3, characterized in that, Determining the first sub-loss value of the sub-coding layer based on the corresponding correlation degree includes: Based on the correlation degree, a cross-correlation matrix of the sub-coding layer is constructed, and the cross-correlation matrix includes the correlation degree between any two new image features among the multiple new image features of the sub-coding layer; Obtain the spectral norm of the cross-correlation matrix, and determine the first sub-loss value of the sub-coding layer based on the spectral norm of the cross-correlation matrix.
6. The method according to claim 2, characterized in that, The process of fusing multiple new image features to obtain fused features includes: For each of the sub-coding layers, the following processing is performed: multiple new image features of the sub-coding layer are fused to obtain the fused features of the sub-coding layer; Decoding the fused features to obtain the decoded image of the image sample includes: For the fusion features of each sub-coding layer, feature transformation is performed on the fusion features of the sub-coding layers to obtain the transformed features of the sub-coding layers; The transformation features of each of the sub-coding layers are fused to obtain target fused features, and the target fused features are decoded to obtain the decoded image of the image sample.
7. The method according to claim 6, characterized in that, When the sub-coding layer is the first sub-coding layer in the coding layers, the feature transformation of the fused features of the sub-coding layer to obtain the transformed features of the sub-coding layer includes: Obtain the random mask feature of the fusion feature of the sub-coding layer, wherein the feature dimension of the random mask feature of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; The random mask features of the sub-coding layer are fused to the fusion features of the sub-coding layer to obtain the transformation features of the sub-coding layer.
8. The method according to claim 6, characterized in that, When the sub-coding layer is not the first sub-coding layer in the coding layers, the feature transformation of the fused features of the sub-coding layer to obtain the transformed features of the sub-coding layer includes: The transformed features of the fusion features of the first sub-coding layer in the coding layer are subjected to two different linear transformations to obtain key features and value features; Obtain the random mask feature of the fusion feature of the sub-coding layer, wherein the feature dimension of the random mask feature of the sub-coding layer is the same as the feature dimension of the fusion feature of the sub-coding layer; The random mask features of the sub-coding layer are fused to the fusion features of the sub-coding layer to obtain the candidate transformation features of the sub-coding layer; A linear transformation is performed on the candidate transformation features to obtain the query features corresponding to the fusion features. The query features corresponding to the fusion features, the key features, and the value features are then fused to obtain the transformation features of the sub-coding layer.
9. The method according to any one of claims 1 to 8, characterized in that, The image processing model performs decorrelation operations on the multiple image features to obtain multiple new image features, including: The image processing model is used to perform decorrelation operations on the multiple image features to obtain multiple candidate image features; The multiple candidate image features with dimensional differences are standardized to obtain the multiple new image features without the dimensional differences.
10. The method according to claim 9, characterized in that, The candidate image features include multiple feature elements. The standardization process for the multiple candidate image features with dimensional differences to obtain multiple new image features without the dimensional differences includes: The following processing is performed on each of the candidate image features: Determine the mean of the eigenvalues of the plurality of feature elements, and the standard deviation of the eigenvalues of the plurality of feature elements; Subtract the mean value from the feature value of each feature element in the candidate image features to obtain the target image features; The new image features are obtained by dividing the feature value of each feature element in the target image features by the standard deviation.
11. The method according to any one of claims 1 to 8, characterized in that, Determining the correlation between any two of the new image features includes: Perform the following processing on any two of the new image features: Determine the covariance between the new image features and the standard deviation of each of the new image features; Obtain the product of the standard deviations of each of the new image features, and divide the covariance between the new image features by the product value to obtain the correlation between any two of the new image features.
12. The method according to any one of claims 1 to 8, characterized in that, Determining the first loss value based on the degree of correlation includes: Based on the degree of correlation, a cross-correlation matrix is constructed among the plurality of new image features, wherein the cross-correlation matrix includes the degree of correlation between any two of the plurality of new image features; Obtain the spectral norm of the cross-correlation matrix, and determine the first loss value based on the spectral norm of the cross-correlation matrix.
13. The method according to any one of claims 1 to 8, characterized in that, Determining the second loss value based on the decoded image and the image samples includes: Obtain the label image of the image sample, and extract features from the label image to obtain the label image features; Feature extraction is performed on the decoded image to obtain decoded image features, and the similarity between the label image features and the decoded image features is determined. The similarity is then used as the second loss value.
14. An image processing method, characterized in that, The method further includes: The target image processing model encodes the image to be processed in multiple channels to obtain multiple image features corresponding to one channel. Then, the target image processing model performs decorrelation operation on the multiple image features of the multiple channels to obtain multiple new image features. The target image processing model is used to fuse the multiple new image features to obtain fused features, and the fused features are then decoded to obtain the decoded image of the image. The target image processing model is trained using the method described in any one of claims 1 to 13.
15. A training device for an image processing model, characterized in that, The device includes: The decorrelation module is used to encode image samples in multiple channels to obtain multiple image features corresponding to one channel, and then perform decorrelation operation on the multiple image features through the image processing model to obtain multiple new image features; The first loss module is used to determine the degree of correlation between any two of the new image features, and to determine a first loss value based on the degree of correlation. A decoding module is used to fuse multiple new image features to obtain fused features, and to decode the fused features to obtain a decoded image of the image sample; The training module is used to determine a second loss value based on the decoded image and the image samples, so as to train the image processing model based on the first loss value and the second loss value.
16. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 14.
17. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 14.
18. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the method described in any one of claims 1 to 14.
