Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium. Relates to the technical field of image processing. The method comprises the steps of obtaining a target image; based on an image feature extraction module, performing feature extraction on the target image to obtain target image features; the image feature extraction module comprises a sampling unit, a window holding unit and a grid holding unit which are connected in sequence; a retention self-attention mechanism is applied in a window retention unit and a mesh retention unit. According to the method provided by the invention, when feature extraction is carried out on the full glass slide image, local features can be captured by paying attention to the shape, texture and other information of the cells, the difference between the cells and other cells as well as the difference between the cells and the microenvironment can be learned to obtain global dependency, and finally, a decision is made by combining local and global representations.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the development of computational pathology and artificial intelligence (AI) technology, the number of data processing methods based on whole slide images (WSI) has gradually increased. At present, the methods for processing WSI can be divided into two categories: one is the multiple instance learning (MIL) method that focuses on the overall level classification of WSI, and the other is the supervised learning method that focuses on more subtle patch-level classification. The advantage of the MIL method is that it does not require pathologists to manually annotate different tissue components and regions for all WSIs, and directly predicts WSIs by converting patches into a series of feature vector sequences. However, the performance of the MIL method in multi-classification tasks is not satisfactory, and it is impossible to simultaneously achieve semantic segmentation of cell tissue regions in WSI. In contrast, the supervised learning method based on patch-level classification not only shows higher accuracy in multi-classification tasks of WSI, but also can achieve cell tissue segmentation, showing a broader potential for clinical application.

[0003] In the patch-level classification task, existing models mainly use convolutional neural networks (CNN), Transformer, or a combination of the two. In CNN-based networks, the model lacks understanding of global features, requires extremely deep networks to obtain a large receptive field, and has poor interpretability. Transformer-based networks are extremely dependent on prior knowledge, and their complexity grows with the square of the sequence length, resulting in a sharp increase in computing resource requirements. Summary of the invention

[0004] The present application provides an image processing method, apparatus, device and storage medium to solve the problems of insufficient image feature capture capability and high inference cost in image processing methods.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Get the target image;

[0007] Based on the image feature extraction module, feature extraction is performed on the target image to obtain target image features;

[0008] The image feature extraction module includes a sampling unit, a window holding unit and a grid holding unit connected in sequence; the sampling unit performs sampling in response to an input target image to obtain a first image feature, the window holding unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a self-attention mechanism to pixels in each non-overlapping block and then performs a multi-layer perceptron operation to obtain a second image feature; the grid holding unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, applies a self-attention mechanism to each grid and then performs a multi-layer perceptron operation to obtain a target image feature;

[0009] The calculation process of retaining the self-attention mechanism is:

[0010] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0011] O W ,K W =(Q,K) B,L,C→B,H,W,C

[0012]

[0013] ReSA dec (X) = Attn H (Attn W V)

[0014] Where Q: query matrix, which represents the linear transformation of the input features and is used to calculate the attention score;

[0015] K: key matrix, which is a linear transformation of the input features and is used to calculate similarity with the query matrix;

[0016] O H and O W : The height and width of the output feature matrix, indicating the spatial dimension of the output features after the self-attention mechanism;

[0017] B: Batch size, which indicates the number of samples input at one time;

[0018] L: sequence length, corresponding to the length of the input feature in sequence form;

[0019] C: number of channels, indicating the depth of each input feature;

[0020] K H and K W : corresponds to the representation of the bond matrix in the height and width dimensions, used to calculate spatial correlation;

[0021] DH and D W : represents the distance attenuation function, which is used to reduce the attention weight between distant pixels;

[0022] γ: distance attenuation factor, used to control the influence of distance between different pixels;

[0023] y n and m : Indicates the vertical coordinates of two pixels in the image;

[0024] x n and x m : Indicates the horizontal coordinates of two pixels in the image.

[0025] In a possible design, the sampling unit samples the input target image by the following formula to obtain the first image feature:

[0026]

[0027] In the formula,

[0028] x: represents the input target image or feature map;

[0029] x′: represents the first image feature, which is the feature map generated by this processing;

[0030] Proj: represents the projection layer, which is a 1x1 convolution operation used to map or compress the number of channels of the feature;

[0031] SE: stands for squeeze-incentive mechanism;

[0032] DWConv: stands for Depthwise Separable Convolution, which decomposes the standard convolution into depthwise convolution and pointwise convolution, reducing the computational complexity while retaining good feature extraction capabilities;

[0033] Conv: represents the convolution operation, which refers to the standard 2D convolution operation, used to extract local features from the input image or feature map;

[0034] Norm: represents the normalization operation, which is batch normalization and is used to stabilize the training process and accelerate convergence.

[0035] undownsample: means no downsampling, that is, keeping the original resolution of the image. In this case, the input image or feature map is directly projected after the squeeze-excitation and convolution operations without changing its spatial dimension;

[0036] downsample: represents downsampling, that is, reducing the spatial resolution of an image or feature map through pooling or convolution operations. In this case, the input image is first reduced in resolution through a pooling operation (such as maximum pooling or average pooling), and then processed through the SE module and DWConv convolution;

[0037] Pool2D: represents a 2D pooling operation, which is used to downsample images or feature maps to reduce spatial resolution, usually using maximum pooling or average pooling.

[0038] In one possible design, the method further includes:

[0039] Based on the image feature extraction module, construct an image classification model;

[0040] The image classification model includes a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in sequence. The four feature extraction modules are obtained by stacking the image feature extraction modules with preset proportions and channel numbers on four feature scales; the preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules. After the four image feature extraction modules perform dimensionality reduction processing on the preprocessed image features, the classification results are output through the pooling layer and the fully connected layer.

[0041] In one possible design, after building the image classification model, the method further includes:

[0042] Build a self-supervised learning framework and image dataset;

[0043] The image classification model is trained using the self-supervised learning framework and the image data set, and image classification is achieved using the trained image classification model; when training the image classification model, the fully connected layer used for classification is replaced by a projection head, and the image classification model uses the sample image of the image data set as input, and maps the output sample image features to the self-supervised learning framework through the projection head for weight update, and the image classification model is configured with the updated weights to obtain the trained image classification model.

[0044] In a possible design, the self-supervised learning framework performs two different data augmentation operations in response to input sample image features to generate a first view and a second view, and encodes the first view and the second view through a query encoder and a momentum encoder;

[0045] The momentum encoder is dynamically updated by querying the encoder weights, and the update formula is:

[0046] f k1 =m×fk +(1-m)×f q

[0047] In the formula, f k1 is the updated parameter of the momentum encoder, f k is the parameter of the momentum encoder before updating, f q is to query the current parameters of the encoder, m is the momentum coefficient, and its value ranges from 0 to 1;

[0048] The weights of the query encoder are gradient updated using a contrastive loss function, which is expressed as:

[0049]

[0050] Where, L q is the contrast loss, k + is the positive sample key vector corresponding to q, k - is the negative sample key vector set, τ is the temperature parameter used to control the sensitivity of the similarity calculation, q·k + is the query vector q and its corresponding positive sample key vector k + exp is the exponential function, which is used to convert the dot product into a positive number to amplify the difference.

[0051] In one possible design, when training the image classification model, a multi-classification task loss function and KL divergence are used to guide the gradient descent process of the self-supervised learning framework;

[0052] The multi-classification task loss function is expressed as:

[0053]

[0054] Where, L CE is the multi-classification task loss function value, C is the total number of categories, y i,c is the one-hot encoding representation of whether sample i belongs to category c, p i,c is the probability that the model predicts that sample i belongs to category c, and N is the total number of samples;

[0055] The KL divergence is expressed as:

[0056]

[0057] Where α is a vector parameter of the Dirichlet distribution, Γ represents the gamma function, and S α and They represent the sum of parameters α and α0, ψ is the logarithmic derivative of the gamma function, K is the number of categories, α0 is the parameter of the prior Dirichlet distribution, and represents the initial classification probability distribution; in the Bayesian framework, the prior distribution expresses the degree of belief when no data is seen; θ represents the sample drawn from the Dirichlet distribution, that is, the specific category probability value; α k represents the parameter of the kth category in the Dirichlet distribution, which is used to describe the weight of the kth category; p represents the distribution p(θ|α), that is, the Dirichlet distribution with parameter α, which is used to describe the posterior distribution of a certain category θ.

[0058] In a second aspect, the present application provides an image processing device, comprising:

[0059] An image acquisition module is configured to acquire a target image;

[0060] A feature extraction module is configured to extract features of the target image based on the image feature extraction module to obtain target image features;

[0061] The image feature extraction module includes a sampling unit, a window holding unit and a grid holding unit connected in sequence; the sampling unit performs sampling in response to an input target image to obtain a first image feature, the window holding unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a self-attention mechanism to pixels in each non-overlapping block and then performs a multi-layer perceptron operation to obtain a second image feature; the grid holding unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, applies a self-attention mechanism to each grid and then performs a multi-layer perceptron operation to obtain a target image feature;

[0062] The calculation process of retaining the self-attention mechanism is:

[0063] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0064] O W ,K W =(Q,K) B,L,C→B,H,W,C

[0065]

[0066] ReSA dec (X) = Attn H (Attn W V)

[0067] In the formula, O H Indicates that KH indicates, Q indicates, K indicates, B, L, C indicates, B, W, H, C indicates, O W Indicates that K W Said, Attn H Represents, Softmax represents, represents, ⊙ represents, D H Said, Attn W express, Indicates that D W express, represents, γ represents, y n Indicates that y m express, Indicates that X n Indicates that X m ReSA dec (X) means, X means, Attn W V indicates. Same as above

[0068] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the image processing method described in the first aspect and various possible designs of the first aspect.

[0069] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0070] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the image processing method described in the first aspect and various possible designs of the first aspect.

[0071] The image processing method, apparatus, device and storage medium provided by the present application propose a well-interpretable image feature extraction module, which uses sampling units and a window-based pixel-level retention mechanism to capture local features, and uses a grid-to-grid block retention mechanism to obtain long-distance dependencies. When extracting features from whole slide images, this module can not only focus on information such as cell shape and texture to capture local features, but also learn the differences between cells and other cells and microenvironments to obtain global dependencies, and finally make decisions by combining local and global representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0073] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of a self-attention retention mechanism provided in an embodiment of the present application, wherein (a) is a two-dimensional weight decay mechanism for a window retention unit, and (b) is a two-dimensional weight decay mechanism for a grid retention unit;

[0075] Figure 3 A detailed schematic diagram of the SCRTrans model structure provided in the embodiment of the present application;

[0076] Figure 4 A detailed self-supervised learning flow chart provided for the embodiments of the present application;

[0077] Figure 5 Feature graphs of the SCRTrans model at different scales provided in the embodiment of the present application, where each small square represents each window divided by the model, and the window size doubles with each stage;

[0078] Figure 6 Schematic diagrams of visualization of different models provided in the embodiments of the present application, including: (a) T-SNE visualization of the K-Means-CNN model, (b) T-SNE visualization of the UNI fine-tuning model, (c) T-SNE visualization of the SCRTrans model;

[0079] Figure 7 A schematic diagram of the actual label and the predicted label provided in the embodiment of the present application;

[0080] Figure 8 A confusion matrix for the model recognition performance provided in the embodiments of the present application, wherein: (a) a confusion matrix for the diagnostic performance of the wsi-level SCRTrans model on the Ebrains dataset, (b) a confusion matrix for the diagnostic performance of the wsi-level SCRTrans model on the Southwest dataset;

[0081] Fig. 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0082] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0083] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data, user data or medical image data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0086] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0087] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0088] Currently, in the data processing process of whole slide images, existing data processing methods such as using convolutional neural networks (CNN), Transformer, or a combination of the two are difficult to simultaneously meet the key goals of high accuracy, strong interpretability, and low deployment cost.

[0089] In response to the above problems, the present application proposes an image processing method, apparatus, device and storage medium, and designs an efficient image feature module (MB-RetNet module). First, a sampling unit (MBConv module) is used to capture the spatial relationship and local features in the image, and the spatial information is indirectly processed during the learning process, thereby replacing the explicit position encoding. Then, based on the window holding unit, the output features are divided into several windows, and the local details of the image are captured by using the retentive network mechanism for each pixel in the window. Next, based on the grid holding unit, the output features are gridded again, and the global context dependency is obtained by using the retentive network mechanism between all grids. This design enables the image feature module to not only have a good understanding of the local details such as the shape and texture of pathological cells, but also fully learn the connection between pathological cells and the surrounding microenvironment, so that the model's decision is more in line with human standards. In addition, because RetNet has the advantages of low-cost reasoning, efficient long sequence modeling, performance comparable to Transformer and parallel model training, the MB-RetNet module has excellent feature capture capabilities while greatly reducing the cost of reasoning.

[0090] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0091] An embodiment of the present application provides an image processing method. Figure 1 Flow chart of the image processing method provided in the embodiment of the present application. Figure 1 As shown, the image processing method can be implemented based on an electronic terminal. The image processing method starts with step S100 to obtain a target image. The target image refers to various types of images that meet the set conditions, including but not limited to whole slide images. The set conditions can be the format, resolution, and pixels of the image. If an image is one of the set formats, and the resolution and pixels meet the set standards, then the image can be used as the target image. Of course, the above set conditions are only examples and are not limitations of the present invention. When step S100 is implemented in an electronic terminal, the electronic terminal can obtain the target image by importing or transmitting it from other terminals through communication interaction.

[0092] Step S200: Based on the image feature extraction module, feature extraction is performed on the target image to obtain target image features.

[0093] The image feature extraction module includes a sampling unit, a window holding unit and a grid holding unit connected in sequence; the sampling unit samples in response to an input target image to obtain a first image feature, the window holding unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a self-attention mechanism to the pixels in each non-overlapping block and then performs a multi-layer perceptron operation to obtain a second image feature; the grid holding unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, applies a self-attention mechanism to each grid and then performs a multi-layer perceptron operation to obtain a target image feature.

[0094] In this embodiment, more image features are captured by applying a reserved self-attention mechanism in the window holding unit and the grid holding unit.

[0095] Specifically, the self-attention mechanism is implemented based on the RMT module. RentNet is seen as a powerful successor to CNN and Transformer, and shows significant advantages in large models and long sequence tasks. RetNet consists of a multi-scale retention (MSR) module and a feed-forward network (FFN) module., following a layout similar to Transformer. Specifically, the self-attention mechanism is implemented based on the RMT module (retained multi-task module). RetNet is seen as a powerful successor to CNN and Transformer, and shows significant advantages in large-scale models and long sequence tasks. RetNet consists of a multi-scale retention module (Multi-Scale Retention, MSR) and a feed-forward network module (Feed-Forward Network, FFN), and its layout is similar to Transformer.

[0096] RetNet uses a retention mechanism to replace multi-head attention, and uses a decay matrix multiplied by a weight matrix to control the proportion of each pixel that sees surrounding pixels. RentNet supports three computing paradigms: parallel, recursive, and chunkwise recurrent. The parallel representation allows full utilization of GPU training parallelism. The recursive representation reduces the inference cost to O(1), improves decoding throughput, latency, and GPU memory usage efficiency without sacrificing performance. The chunkwise recurrent representation helps to process long sequences efficiently, with linear complexity, each block is encoded in parallel, and the block information is summarized cyclically. The retention mechanism first considers using a cyclic approach for sequence modeling, which can be written as equation (1):

[0097]

[0098] in, is the conjugate transpose, O n Represents the output in sequence modeling, that is, the output feature of the current moment nnn, combined with the feature information of the previous moment mmm.

[0099] γ represents the position attenuation factor, which is used to control the model's attention to different positions (time steps). As the position gap nm increases, the weight γ^(nm) will gradually decay, thereby paying more attention to inputs closer to the current time step.

[0100] m represents the time step, which is the number of time steps before traversal. θ represents the phase factor, which is used to encode time or position information. K m represents the key vector, which is used to capture the features of the mmth time step in the sequence. m Represents the value vector of the time step in the sequence, which carries the specific information of the time step.

[0101] For the parallel training paradigm, it can be written as Equation (2):

[0102] Q=(XW Q )⊙Θ, V=XW V

[0103]

[0104] Retention(X)=(QK T ⊙D)V

[0105] Where Q: represents the query matrix, which is obtained by linearly transforming the input X Q The vector matrix obtained is used to calculate the attention score with the key vector K.

[0106] X: represents the input feature matrix, which contains the input information of all time steps.

[0107] W Q : Represents the weight matrix of the query vector, which is used to perform a linear transformation on the input X to generate the query vector.

[0108] Θ: represents the position coding matrix, which is used to introduce position information. Usually a phase factor is used to represent the difference in time or position.

[0109] K: represents the key vector (Key matrix), through the weight matrix W KThe input feature X is linearly transformed to represent the correlation of the features.

[0110] W K : Represents the weight matrix of the key vector, which is used to perform a linear transformation on the input X to generate the key vector.

[0111] Similar to Θ, but contains different position information (such as the transposed position information).

[0112] V: represents the value vector (Value matrix), through the weight matrix W V The vector matrix obtained by linearly transforming the input X contains the specific information of each time step in the sequence.

[0113] W V : Represents the weight matrix of the value vector, which is used to perform a linear transformation on the input X to generate a value vector.

[0114] Θ N : represents the phase encoding matrix of the Nth time step, using e inθ It indicates that it is an encoded form of location information.

[0115] D nm : represents the position decay matrix, which is used to represent the distance decay relationship between time steps n and m. For n ≥ m, the decay factor is γ^(nm); for n ≤ m, the decay factor is 0.

[0116] Retention(X): represents the result obtained after the input X is operated on the query, key, and value matrices under the retention mechanism, using the attention mechanism QK T ⊙D calculates the attention score and combines it with the value matrix V to get the final output.

[0117] D: represents the decay matrix, which controls the distribution of attention weights between each time step, ensuring that the model pays more attention to the input features close to the current time step.

[0118] RMT adds bidirectional modeling and two-dimensional expansion based on RetNet, and retains the Softmax operator. At the same time, in order to reduce the overhead of the model, the retention operator is decomposed into two axes of the image. This is the core module of RMT, called Retentive Self-Attention (ReSA), which can be written as equation (3):

[0119] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0120] O W ,K W=(Q,K) B,L,C→B,H,W,V

[0121]

[0122]

[0123] ReSA dex (X) = Attn H (Attn W V)

[0124] Where Q: query matrix, which represents the linear transformation of the input features and is used to calculate the attention score;

[0125] K: key matrix, which is a linear transformation of the input features and is used to calculate similarity with the query matrix;

[0126] O H and O W : The height and width of the output feature matrix, indicating the spatial dimension of the output features after the self-attention mechanism;

[0127] B: Batch size, which indicates the number of samples input at one time;

[0128] L: sequence length, corresponding to the length of the input feature in sequence form;

[0129] C: number of channels, indicating the depth of each input feature;

[0130] K H and K W : corresponds to the representation of the bond matrix in the height and width dimensions, used to calculate spatial correlation;

[0131] D H and D W : Represents the distance attenuation function, which is used to reduce the attention between distant pixels

[0132] Weight;

[0133] γ: distance attenuation factor, used to control the influence of distance between different pixels;

[0134] y n and m : Indicates the vertical coordinates of two pixels in the image;

[0135] x n and x m : Indicates the horizontal coordinates of two pixels in the image.

[0136] The following embodiment will describe in detail the process of image feature extraction by the image feature extraction module. In this embodiment, the image feature extraction module is referred to as the MB-RetNet module, and the sampling unit included in it is referred to as the MBConv module.

[0137] The MBConv module decides whether to downsample by controlling the stride in Depthwise Conv3×3. Specifically, assuming x is the input feature, the operation process of the MBConv module can be written as formula (4):

[0138]

[0139] In the formula,

[0140] x: represents the input target image or feature map;

[0141] x′: represents the first image feature, which is the feature map generated by this processing;

[0142] Proj: represents the projection layer, which is a 1x1 convolution operation used to map or compress the number of channels of the feature;

[0143] SE: stands for squeeze-incentive mechanism;

[0144] DWConv: stands for Depthwise Separable Convolution, which decomposes the standard convolution into depthwise convolution and pointwise convolution, reducing the computational complexity while retaining good feature extraction capabilities;

[0145] Conv: represents the convolution operation, which refers to the standard 2D convolution operation, used to extract local features from the input image or feature map;

[0146] Norm: represents the normalization operation, which is batch normalization and is used to stabilize the training process and accelerate convergence.

[0147] undownsample: means no downsampling, that is, keeping the original resolution of the image. In this case, the input image or feature map is directly projected after the squeeze-excitation and convolution operations without changing its spatial dimension;

[0148] downsample: represents downsampling, that is, reducing the spatial resolution of an image or feature map through pooling or convolution operations. In this case, the input image is first reduced in resolution through a pooling operation (such as maximum pooling or average pooling), and then processed through the SE module and DWConv convolution;

[0149] Pool2D: represents a 2D pooling operation, which is used to downsample images or feature maps to reduce spatial resolution, usually using maximum pooling or average pooling.

[0150] For the first MBConv block in each stage, downsampling is performed by applying stride-2Depthwise Conv3x3, while the shortcut branch should also apply pooling and channel projection.

[0151] The window holding unit takes the first image feature x' as input and performs a window operation, which is defined as the first image feature x' as the input. Divide into multiple non-overlapping blocks of size P×P, where H, W, and C represent the height, bandwidth, and number of channels of the image, respectively. This operation can be expressed as equation (5):

[0152]

[0153] After the window operation, the dimensions of the window are aggregated to the spatial dimension (i.e., the -2 axis). Then, Retained Self-Attention (ReSA) is applied to the pixels in each window, which can be expressed by formula (6):

[0154] x'1←x'+Unblock(RelAttention(Block(LN(x'))))

[0155] x”←x'1+MLP(LN(x'1)) (6)

[0156] Where LN represents layer normalization, MLP represents a multi-layer perceptron containing two linear layers, x'1 represents the image features after applying the reserved self-attention mechanism to the pixels in each non-overlapping block, and x" represents the second image features.

[0157] The grid-preserving unit takes the second image feature x' as input and performs a grid operation, which is defined as dividing the input feature into a uniform grid of G × G, each grid having an adaptive size Unlike the window operation, the grid operation requires an additional transposition operation to place the grid dimension on the assumed spatial axis (i.e., the -2 axis). The axis swap operation involved is swapaxes(axis1=-2,axis2=-3). This operation can be expressed as equation (7):

[0158] Grid:

[0159] Then the preserved self-attention mechanism (ReSA) is applied to each grid, which can be expressed as formula (8):

[0160] x″1←x”+Ungrid(RelAttention(Grid(LN(x”))))

[0161] x”'←x″1+MLP(LN(x″1)) (8)

[0162] Where LN represents layer normalization, MLP represents a multi-layer perceptron containing two linear layers, x'1' represents the image feature after applying the self-attention mechanism to each grid, and x'' represents the third image feature.

[0163] The operation flow here is similar to the window retention mechanism, but the grid operation calculates the retention between grids.

[0164] The MB-RetNet module is composed of the MBConv module, the window retention unit, and the grid retention unit. In the MB-RetNet module, the MBConv module obtains the spatial information and positional relationship between elements while performing deep convolution on the element image or feature. In the subsequent window retention and grid retention mechanisms, the explicit position encoding is cancelled. Experiments have found that the performance of the model remains almost unchanged, and the speed is greatly improved during parallel training. In the self-attention retention mechanism, each marker will perform the same self-attention calculation as all other markers, and the complexity of the calculation is proportional to the square of the sequence length. The retentive network retention network uses a position decay mechanism to focus on the calculation of each marker and its surrounding markers, and combines recursive reasoning to greatly reduce the complexity. Its principle is as follows Figure 2 As shown in the figure, each pixel calculates position attenuation based on its Manhattan distance to the surrounding pixels, making the pixel more sensitive to the surroundings at close range and more sensitive to the distant ones. This position attenuation retention mechanism is particularly suitable for WSI, because each cell or tissue in WSI can have an independent label, instead of having a meaningful label only when it forms a larger pattern or the entire image like in natural images. Specifically, by adjusting the size of the pixel to be equal to the physical size of a cell, calculating the memory retention between grids becomes the cell looking for its relationship with neighboring cells or the surrounding microenvironment.

[0165] In some embodiments, the image processing method further comprises the following steps:

[0166] S300, constructing an image classification model based on the image feature extraction module. The image classification model includes a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in sequence, wherein the four feature extraction modules are obtained by stacking the image feature extraction modules with preset proportions and channel numbers on four feature scales; the preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules, and the four image feature extraction modules perform dimensionality reduction processing on the preprocessed image features and output classification results through the pooling layer and the fully connected layer.

[0167] The image classification model constructed in this embodiment is as follows Figure 3 As shown, the image classification model is referred to as the SCRTrans model in this embodiment, and the preprocessing module is referred to as the Stem module. Exemplarily, by stacking MB-RetNet modules at different proportions and numbers of channels on four feature scales, two versions of Convolution Retentive Network Transformer (SCRTrans), Small and Base, are obtained. The specific parameter configurations are shown in Table 1. The Small version is formed by stacking MB-RetNet modules at a ratio of 1:1:1:1. It is a lightweight model suitable for low-cost and fast reasoning scenarios. The Base version is formed by stacking MB-RetNet modules at a ratio of 2:2:5:2. It has a larger number of parameters and stronger accuracy. It is suitable for scenarios with high accuracy requirements and better deployment equipment.

[0168] Table 1 Parameters of image classification model (SCRTrans model)

[0169]

[0170] It should be noted that the present application can also configure other versions of the SCRTrans model based on other ratios. The above two versions are merely examples and do not constitute a limitation to the present application.

[0171] like Figure 3 As shown in the figure, the SCRTrans model consists of 1 Stem module, 4 MB-RetNet modules, 1 pooling layer and 1 fully connected layer. The features are reduced in dimension in modules S0 to S4, and finally the classification results are output through the pooling layer and the fully connected layer. Among them, the Stem module consists of 2 2D convolutional layers and GELU activation layers. The convolution kernel size is 3×3, and the stride of the first convolutional layer is set to 2 to downsample the feature map scale. Each MB-RetNet module is divided into 3 parts. In the first part, the features first pass through an MBConv module, which is a convolution block based on MobileNetV2. It uses depth-separable convolution for efficient feature extraction and includes downsampling steps. In the second part, the window partitioning function divides the feature map into multiple small windows, and then applies the retention memory mechanism to each window to process local features. In the third part, it is similar to window retention, but grid partitioning is used, which allows the retention memory module to capture the interaction between features in a larger range and process more global information. Finally, by processing image features at different scales, the SCRTrans model not only processes local details but also integrates global information, ultimately achieving a deep understanding of the image content.

[0172] In some embodiments, after constructing the image classification model, the method also includes: constructing a self-supervised learning framework and an image data set; using the self-supervised learning framework and the image data set to train the image classification model, and using the trained image classification model to implement image classification; when training the image classification model, the fully connected layer used for classification is replaced by a projection head, and the image classification model uses the sample image of the image data set as input, and maps the output sample image features to the self-supervised learning framework through the projection head for weight updating, and configures the image classification model with the updated weights to obtain the trained image classification model.

[0173] Considering the inherent characteristics of WSI, that is, the patches cut from the same WSI have very high similarity, direct training is extremely likely to lead to overfitting. Therefore, this embodiment develops a self-supervised learning framework that can learn the inherent characteristics of patches without labels by improving Moco v3. The detailed process is as follows: Figure 4 In this framework, the model learns to distinguish different image representations by comparing positive and negative samples, thereby improving the quality of learned visual representations.

[0174] In some embodiments, the self-supervised learning framework performs two data augmentation operations on each image to generate two different views x1 and x2, respectively, by querying the encoder f q and momentum encoder f k To encode these views. The momentum encoder does not perform training updates directly, but is dynamically updated by querying the encoder weights, which can be specifically expressed as formula (9):

[0175] f k =m×f k +(1-m)×f q (9)

[0176] where f k is the parameter of the momentum encoder before updating, f q is to query the current parameters of the encoder, m is the momentum coefficient, ranging from 0 to 1. The momentum coefficient m determines f k The degree to which the old parameters are maintained and the q The degree of accepting new parameters, choosing a higher value of m can make f k The changes are smoother, which is especially beneficial for processing large-scale datasets because it can provide more stable feature representations to combat sample noise and variation.

[0177] The gradient update of the query encoder directly depends on the contrast loss, which aims to make the feature representations of two views of the same image (positive sample pairs) close, and the feature representations of views of other images (negative sample pairs) far away. The loss function can be expressed as:

[0178]

[0179] Among them, k + is the positive sample key vector corresponding to q, k - is the set of negative sample key vectors, and τ is the temperature parameter used to control the sensitivity of similarity calculation. + is the query vector q and its corresponding positive sample key vector k + The dot product between . The size of the dot product indicates the similarity of the two vectors. The larger the dot product, the higher the similarity. The exponential function exp can convert the dot product into a positive number, thereby amplifying the difference and making the influence of vector pairs with high similarity more significant. The temperature parameter τ can adjust the range of values ​​after the dot product, thereby affecting the "smoothness" of the softmax function. A smaller τ will make the output sharper (i.e., it will highlight vector pairs with high similarity more), while a larger τ will make the output smoother (i.e., reduce the influence of extreme values ​​of similarity).

[0180] In this self-supervised learning framework, the query encoder and the vector encoder are both modified from the SCRTrans model, and the fully connected layer used for classification during training is replaced by the projection head. Since the self-supervised ViT has been found to have instability problems in Moco v3, this embodiment avoids the same problem by freezing the Stem module in the SCRTrans model.

[0181] In some embodiments, in order to improve the prediction accuracy of the model while capturing the uncertainty assessment of its prediction, this embodiment uses CrossEntropyLoss and Kullback-Leibler divergence to jointly guide the gradient descent process. CrossEntropyLoss is a loss function for multi-classification tasks, which focuses on maximizing the probability of correct classification. It measures the cross entropy between the probability distribution of the model output and the probability distribution of the true label. It can be expressed as formula (11):

[0182]

[0183] Where C is the total number of categories, y i,c is the one-hot encoding representation of whether sample i belongs to category c, p i,c is the probability that the model predicts that sample i belongs to category c.

[0184] In machine learning, uncertainty can usually be divided into model uncertainty and data uncertainty. Model uncertainty, also known as apparent uncertainty or epistemic uncertainty, is related to the limitations of the model itself, such as the model's uncertain predictions of unseen patterns in the data. Data uncertainty, also known as intrinsic uncertainty, is related to the noise in the data itself, the randomness that cannot be explained by even the best model. The uncertainty of the model is quantified by modeling the output of the SCRTrans model as the parameters of the Dirichlet distribution and using the Kullback-Leibler divergence to measure the difference between the predicted distribution and the prior distribution. For the Dirichlet distribution, it can be expressed as formula (12):

[0185]

[0186] where α is a vector parameter of the Dirichlet distribution, Γ represents the gamma function, S α and S α0 Represents the sum of α and α0 parameters respectively. ψ is the digamma function, that is, the logarithmic derivative of the gamma function, and K is the number of categories. If the model's prediction (described by the parameter α of the Dirichlet distribution) is close to a uniform distribution (all α are equal), then the uncertainty is high; if some α is significantly larger than other values, it indicates that the model is more confident in the prediction of a specific category, that is, the uncertainty is low. θ represents the sample drawn from the Dirichlet distribution, that is, the specific category probability value; α k represents the parameter of the kth category in the Dirichlet distribution, which is used to describe the weight of the kth category; p represents the distribution p(θ|α), that is, the Dirichlet distribution with parameter α, which is used to describe the posterior distribution of a certain category θ.

[0187] The following example will combine the case of the method for processing medical image data to fully illustrate the feasibility and progress of the present application. It should be noted that the image processing method described in the above example is not limited to the following case. In some embodiments, it can also be applied to target cell recognition in cell images, etc. The application of WSI images described here is only an example and does not constitute a limitation to the present application.

[0188] This example collects two datasets with a total of 1798 patients, of which the Ebrain open source dataset contains brain tumor WSIs of 1368 patients, and the Southwest private dataset contains brain tumor WSIs of 430 patients. For both datasets, the WSIs are divided into a training set and a validation set in a ratio of 7:3.

[0189] In this embodiment, two professional pathologists carried out refined tissue annotation and review on the original brain tumor WSI of two data sets, and the annotation included tumor area, normal area, vascular area, necrotic area, non-diagnostic area, etc. At the same time, in order to ensure the independence of the validation set, the patches used by the training model are all from the WSI of the training set. In the WSI cutting stage, the real physical size represented by each pixel grid of WSI is uniformly adjusted to 1 μm. Then, the ROI area marked by the pathologist is cut into patches of size 256*256, and is divided into 7 types of tumor tissue, normal tissue and non-diagnostic tissue according to the real category of the patch. Next, a double filtering mechanism is used for data cleaning. First, the proportion of white areas is judged according to the brightness and saturation in the HSV color space, and the patches with more blanks are filtered out according to the threshold of 50%. In the second filtering mechanism, CellularityScorer and NucleiScorer in the Histolab library are used to score the cell content of each patch, and then the patches with two scorers less than 0.01 are filtered. In order to reduce the influence of different staining components in WSI on model decision, the color of all patches was normalized by using the Reinhard method in the HistomicsTK library. The Histolab library can be found in the following references: histolab / histolab:Lib raryforDigitalPathologyImageProcessing(github.com) and DigitalSlideArchive / HistomicsTK:APythontoolkitforpathologyimageanalysis algorithms.(github.com) .

[0190] In this embodiment, the model generalization ability is improved by implementing random data enhancement (including 15 operations) and image normalization (based on ImageNet mean and standard deviation). The experiment uses SGD optimizer and batch size of 256, weight decay is 7E-5, initial learning rate is 0.001, and the learning rate is adjusted in combination with cosine annealing scheduler. The loss function used in the model training process, CrossEntropyLoss and Kullback-Leibler divergence jointly guide the gradient descent process of the model. Model training is carried out in an environment equipped with NVIDIA A6000 graphics card, the batch size is set to 256, and 5-fold cross validation is used to ensure the randomness of the training set and the validation set. Finally, a comprehensive set of refined indicators is used to evaluate the pathological grading performance of all models, including: area under the receiver operating characteristic curve (AUC), Top-1 accuracy (Top1-Acc), accuracy (Acc), average class accuracy (MCA), mean average precision (MAP), F1 score, specificity and sensitivity. These indicators together provide a detailed and comprehensive framework for analyzing the performance of the model.

[0191] In order to allow the model to learn richer and more universal feature representations while reducing the risk of overfitting and accelerating convergence, this embodiment uses pre-training plus fine-tuning for transfer learning. At the same time, a variety of pre-training methods such as supervised learning, self-supervised learning, and a combination of the two are introduced, and the effects of different methods are evaluated under the same conditions. In supervised learning, ResNet and ViT are selected as the model backbone, and pre-training on ImageNet is used to introduce prior knowledge of natural images. In self-supervised learning, SimCLR, BYOL and other models are used for comparative learning on the Ebrain training set. In the algorithm that combines the two, the SupCon model is used for supervised comparative learning on the Ebrain training set. The model backbone and pre-training parameters used for self-supervised learning and supervised comparative learning are consistent with the source code of the paper.

[0192] In the fine-tuning stage, the classification head or projection head of the pre-trained model is replaced with the fully connected layer required for brain tumor classification, and the pre-training information is retained by freezing the model weights. Fine-tuning uses the SGD optimizer and a batch size of 256, and dynamically adjusts the learning rate using a cosine annealing scheduler and preheating. The total number of fine-tuning epochs for all models is set to 100, and the classification head is trained on the Ebrain training set using the Early Stop strategy. Finally, all models were evaluated using multiple indicators, including Top-1Accuracy (Top1-Acc), Mean Class Accuracy (MCA), and MeanAverage Precision (MAP), as shown in Table 2. Compared with the ImageNet pre-training used by most studies, the self-supervised method proposed in this application improved Top1-acc by 18 percentage points, showing stronger robustness.

[0193] In the fine-tuning stage, the classification head or projection head of the pre-trained model is replaced with the fully connected layer required for brain tumor classification, and the pre-training information is retained by freezing the model weights. Fine-tuning uses a stochastic gradient descent (SGD) optimizer and a batch size of 256, and dynamically adjusts the learning rate using a cosine annealing scheduler and preheating. The total number of cycles for all model fine-tuning is set to 100, and the classification head is trained on the Ebrain training set using an early stopping strategy. Finally, all models are evaluated on multiple indicators, including Top-1 accuracy (Top1-Acc), mean class accuracy (MCA), and mean average precision (MAP), as shown in Table 2. Compared with the ImageNet pre-training used by most studies, the self-supervised method proposed in this application improves Top1-Acc by 18 percentage points, showing stronger robustness.

[0194] Table 2 Model performance evaluation under different pre-training strategies

[0195]

[0196]

[0197] By freeing up more layers for training during the fine-tuning stage, the accuracy of the SCRTrans model at the patch level is further improved.

[0198] The SCRTrans model is a multi-scale learning model that covers the learning of different cell sizes and tissue characteristics from fine-grained to coarse-grained. At the same time, the construction of the model is combined with tumor pathology knowledge. First, the cell micron size is aligned with the partition window, and then the retention calculation between pixels is performed. Figure 5 As shown in the figure, two original glioma patches were randomly selected for display, and the pathologists used dotted lines to separate different tissue components. In stage 1, the size of each divided window is 16μm, which is just enough to contain 2 red blood cells or 1 other cell. From the feature map of stage 1, it can be seen that the model accurately captures the features of small individual cells. At stage 2, the window size doubles to 32μm, and the model can capture the features of some larger cells and tissues. The window size of Stage 3 is further doubled to 64μm, and the window size at this time can cover almost any size of cells, because the cell size in brain tumor pathological sections is usually less than 40μm. At stage 4, the size of each window is doubled to 128μm, and each window may contain 3-6 normal cells or tumor cells. As shown in the feature map of stage 4, the model can capture larger tissues and deeper features at this time, including some larger cell clusters or tissues. The figure shows that there is a clear brightness contrast between the tumor cell clusters and other tissue area windows.

[0199] This embodiment introduces pathological prior knowledge into the design of the SCRTrans model architecture, and implements a multi-scale learning framework that can simultaneously capture single cells to cell clusters or tissue blocks. This multi-scale learning can effectively capture tissue features in the range of 0 to 128μm in pathological WSI, and perform retention calculations on pixels within the window, thereby enhancing the model's perception of local features. By combining cell size to divide the window into multiple scales, learning starts with smaller red blood cells and gradually expands to cell clusters and tissues, so that the model can have a deep semantic understanding of cells or tissues of various sizes. Finally, the local features extracted independently from the window are combined with grid retention to establish contextual connections, so that the model not only has a good semantic understanding of the content of its own window. It also establishes connections between different windows, and ultimately achieves global perception of the entire image.

[0200] In order to more intuitively show the effect comparison of different models, this example uses T-SNE dimension reduction for feature visualization. Figure 6 The dimensionality reduction of different models on the Ebrains validation set is compared. The data used are all from the Ebrans validation set. By uniformly sampling the patches cut from each WSI, a patch-level visualization validation set consisting of 9 categories and a total of 27,000 images is formed.

[0201] In order to explore the degree of attention paid by the SCRTrans model to the features of different brain tumor categories, three types of brain tumor patches were randomly selected for attention visualization, such as Figure 7 shown.

[0202] After the patch-level deep learning model is trained, the WSI-Level decision results are formed by the majority voting strategy used by most studies. Based on the uncertainty output of the model, a set of highly credible patch-level prediction results are formed, and a set of extremely effective diagnostic decision-making mechanisms are constructed. The first step is the patch filtering stage. First, the patches predicted to be Nondiagnostic are excluded, and then the patches ranked in the bottom 50% are filtered out based on the uncertainty score among the remaining patches. The second step is the benign judgment stage. Among the remaining patches, the proportion of patches predicted to be normal is first calculated. If the proportion is greater than 80%, the WSI type is judged to be normal. If the second step is not established, the tumor type decision is entered in step 3. In step 3, all patches predicted to be normal are filtered out again, and finally the remaining patches are voted by majority.

[0203] Figure 8The confusion matrices of the WSI-Level prediction results of the SCRTrans model on the Ebrains dataset and the Southwest dataset are shown respectively. On the Ebrains dataset, a total of 186 WSIs were predicted in the independently divided validation set. It was found that only 2 predictions were wrong, of which 1 brain tumor of the true type of ependymoma was predicted as glioma, and 1 brain tumor of the true type of ependymoma was predicted as metastases, and all other WSIs were predicted correctly. In the Southwest dataset, a total of 131 WSIs were predicted in the independently divided validation set, and all were found to be correct. The constructed diagnostic decision mechanism is based on the screening mechanism of model uncertainty, which effectively eliminates patches with less confident predictions, thereby greatly enhancing the diagnostic ability of WSI-Level.

[0204] Based on the accurate classification performance of the SCRTrans model at the patch-level, a system for tissue segmentation of WSI was built. The system combines the label and position of each patch in WSI to display the instance segmentation results of tumor area, normal cell area, and non-diagnostic area in WSI.

[0205] Accurate tissue segmentation can assist pathologists in making more detailed reports. This application cleverly incorporates prior knowledge of pathology into the design of the model through cell micron alignment technology, and learns the characteristics of cells and tissues of different sizes in a multi-scale manner.

[0206] Fig. 9 Schematic diagram of the structure of the image processing device provided in the embodiment of the present application. The embodiment of the present application also provides an image processing device. Fig. 9 As shown, the image processing device comprises:

[0207] An image acquisition module 901 is configured to acquire a target image;

[0208] The feature extraction module 902 is configured to extract features of the target image based on the image feature extraction module to obtain target image features;

[0209] The image feature extraction module includes a sampling unit, a window holding unit and a grid holding unit connected in sequence; the sampling unit performs sampling in response to an input target image to obtain a first image feature, the window holding unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a self-attention mechanism to pixels in each non-overlapping block and then performs a multi-layer perceptron operation to obtain a second image feature; the grid holding unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, applies a self-attention mechanism to each grid and then performs a multi-layer perceptron operation to obtain a target image feature;

[0210] The calculation process of retaining the self-attention mechanism is:

[0211] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0212] O W ,K W =(Q,K) B,L,C→B,H,C,C

[0213]

[0214] ReSA dec (X) = Attn H (Attn W V)

[0215] In the formula, O H Represents the feature map in the output height direction, K H represents the height directional feature of the key matrix. Q represents the query matrix, which is used to calculate the attention weight; K represents the key matrix, which is used to calculate the similarity with the query matrix; represents the batch size, sequence length and number of channels respectively; O W Represents the feature map in the output width direction; K W Represents the width-wise characteristics of the key matrix; Attn H Represents the attention weight in the height direction; Softmax represents the operation used for normalization; represents the transpose of the key matrix. ⊙ represents element-wise multiplication; D H Represents the distance attenuation factor in the height dimension; Attn W Represents the attention weight in the width direction; represents the transpose of the key matrix in the width direction; D W Represents the distance attenuation factor in the width dimension; Indicates the distance attenuation weight between two pixels in the height direction. γ indicates the distance attenuation factor, which is used to adjust the influence of far and near pixels; indicates the vertical coordinates of two pixels; Represents the distance attenuation weight between two pixels in the width direction; X m Represents the horizontal coordinates of two pixels; ReSA dec (X) represents the decoded output of the self-attention mechanism; X represents the input image or feature map; Attn W V represents the application of the attention weights in the width direction to the value matrix V.

[0216] In some embodiments, the sampling unit samples the input target image by the following formula to obtain the first image feature:

[0217]

[0218] In the formula,

[0219] x: represents the input target image or feature map;

[0220] x′: represents the first image feature, which is the feature map generated by this processing;

[0221] Proj: represents the projection layer, which is a 1x1 convolution operation used to map or compress the number of channels of the feature;

[0222] SE: stands for squeeze-incentive mechanism;

[0223] DWConv: stands for Depthwise Separable Convolution, which decomposes the standard convolution into depthwise convolution and pointwise convolution, reducing the computational complexity while retaining good feature extraction capabilities;

[0224] Conv: represents the convolution operation, which refers to the standard 2D convolution operation, used to extract local features from the input image or feature map;

[0225] Norm: represents the normalization operation, which is batch normalization and is used to stabilize the training process and accelerate convergence.

[0226] undownsample: means no downsampling, that is, keeping the original resolution of the image. In this case, the input image or feature map is directly projected after the squeeze-excitation and convolution operations without changing its spatial dimension;

[0227] downsample: represents downsampling, that is, reducing the spatial resolution of an image or feature map through pooling or convolution operations. In this case, the input image is first reduced in resolution through a pooling operation (such as maximum pooling or average pooling), and then processed through the SE module and DWConv convolution;

[0228] Pool2D: represents a 2D pooling operation, which is used to downsample images or feature maps to reduce spatial resolution, usually using maximum pooling or average pooling.

[0229] In some embodiments, the image processing apparatus further comprises an image classification module, wherein the image classification module is configured to:

[0230] Based on the image feature extraction module, construct an image classification model;

[0231] The image classification model includes a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in sequence. The four feature extraction modules are obtained by stacking the image feature extraction modules with preset proportions and channel numbers on four feature scales; the preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules. After the four image feature extraction modules perform dimensionality reduction processing on the preprocessed image features, the classification results are output through the pooling layer and the fully connected layer.

[0232] In some embodiments, the image processing apparatus further comprises a self-supervised training module, wherein the self-supervised training module is configured to:

[0233] Build a self-supervised learning framework and image dataset;

[0234] The image classification model is trained using the self-supervised learning framework and the image data set, and image classification is achieved using the trained image classification model; when training the image classification model, the fully connected layer used for classification is replaced by a projection head, and the image classification model uses the sample image of the image data set as input, and maps the output sample image features to the self-supervised learning framework through the projection head for weight update, and the image classification model is configured with the updated weights to obtain the trained image classification model.

[0235] In some embodiments, the self-supervised learning framework performs two different data augmentation operations in response to input sample image features to generate a first view and a second view, and encodes the first view and the second view through a query encoder and a momentum encoder;

[0236] The momentum encoder is dynamically updated by querying the encoder weights, and the update formula is:

[0237] f k1 =m×f k +(1-m)×f q

[0238] In the formula, f k1is the updated parameter of the momentum encoder, f k is the parameter of the momentum encoder before updating, f q is to query the current parameters of the encoder, m is the momentum coefficient, and its value ranges from 0 to 1;

[0239] The weights of the query encoder are gradient updated using a contrastive loss function, which is expressed as:

[0240]

[0241] Where, L q is the contrast loss, k + is the positive sample key vector corresponding to q, k - is the negative sample key vector set, τ is the temperature parameter used to control the sensitivity of the similarity calculation, q·k + is the query vector q and its corresponding positive sample key vector k + exp is the exponential function, which is used to convert the dot product into a positive number to amplify the difference.

[0242] In some embodiments, the self-supervised training module is further configured to:

[0243] When training the image classification model, a multi-classification task loss function and KL divergence are used to guide the gradient descent process of the self-supervised learning framework;

[0244] The multi-classification task loss function is expressed as:

[0245]

[0246] Where, L CE is the multi-classification task loss function value, C is the total number of categories, y i,c is the one-hot encoding representation of whether sample i belongs to category c, p i,c is the probability that the model predicts that sample i belongs to category c, and N is the total number of samples;

[0247] The KL divergence is expressed as:

[0248]

[0249] Where α is a vector parameter of the Dirichlet distribution, Γ represents the gamma function, and S α and S α0They represent the sum of parameters α and α0, ψ is the logarithmic derivative of the gamma function, K is the number of categories, α0 is the parameter of the prior Dirichlet distribution, and represents the initial classification probability distribution; in the Bayesian framework, the prior distribution expresses the degree of belief when no data is seen; θ represents the sample drawn from the Dirichlet distribution, that is, the specific category probability value; α k represents the parameter of the kth category in the Dirichlet distribution, which is used to describe the weight of the kth category; p represents the distribution p(θ|α), that is, the Dirichlet distribution with parameter α, which is used to describe the posterior distribution of a certain category θ.

[0250] The image processing device provided in the embodiment of the present application can be used to execute the technical solution of the image processing method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0251] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device may include: a processor 101 and a memory 102 , wherein the processor 101 and the memory 102 may communicate with each other; illustratively, the processor 101 and the memory 102 communicate with each other via a communication bus 103 .

[0252] The processor 101 executes the computer execution instructions stored in the memory 102, so that the processor 101 executes the scheme in the above embodiment. The processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0253] The communication bus 103 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0254] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0255] The embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the image processing method of the above embodiment.

[0256] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the image processing method in the above embodiment.

[0257] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0258] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0259] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0260] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0261] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0262] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0263] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0264] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0265] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0266] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Get the target image; Based on the image feature extraction module, feature extraction is performed on the target image to obtain target image features; The image feature extraction module includes a sampling unit, a window holding unit and a grid holding unit connected in sequence; the sampling unit performs sampling in response to an input target image to obtain a first image feature, the window holding unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a self-attention mechanism to pixels in each non-overlapping block and then performs a multi-layer perceptron operation to obtain a second image feature; the grid holding unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, applies a self-attention mechanism to each grid and then performs a multi-layer perceptron operation to obtain a target image feature; The calculation process of retaining the self-attention mechanism is: O H ,K H =(Q,K) B,L,C→B,W,H,C O W ,K W =(Q,B) B,L,C→B,H,W , C ReSA dec (X)=Attn H (Attn W V) Where Q: query matrix, which represents the linear transformation of the input features and is used to calculate the attention score; K: key matrix, which is a linear transformation of the input features and is used to calculate similarity with the query matrix; O H and O W : The height and width of the output feature matrix, indicating the spatial dimension of the output features after the self-attention mechanism; B: Batch size, which indicates the number of samples input at one time; L: sequence length, corresponding to the length of the input feature in sequence form; C: number of channels, indicating the depth of each input feature; K H and K W : corresponds to the representation of the bond matrix in the height and width dimensions, used to calculate spatial correlation; D H and D W : represents the distance attenuation function, which is used to reduce the attention weight between distant pixels; γ: distance attenuation factor, used to control the influence of distance between different pixels; y n and m : Indicates the vertical coordinates of two pixels in the image; x n and x m : Indicates the horizontal coordinates of two pixels in the image.

2. The method according to claim 1, characterized in that The sampling unit samples the input target image by the following formula to obtain the first image feature: In the formula, x: represents the input target image or feature map; x′: represents the first image feature, which is the feature map generated by this processing; Proj: represents the projection layer, which is a 1x1 convolution operation used to map or compress the number of channels of the feature; SE: stands for squeeze-incentive mechanism; DWConv: stands for Depthwise Separable Convolution, which decomposes the standard convolution into depthwise convolution and pointwise convolution, reducing the computational complexity while retaining good feature extraction capabilities; Conv: represents the convolution operation, which refers to the standard 2D convolution operation, used to extract local features from the input image or feature map; Norm: represents the normalization operation, which is batch normalization and is used to stabilize the training process and accelerate convergence. undownsample: means no downsampling, that is, keeping the original resolution of the image. In this case, the input image or feature map is directly projected after the squeeze-excitation and convolution operations without changing its spatial dimension; downsample: represents downsampling, that is, reducing the spatial resolution of an image or feature map through pooling or convolution operations. In this case, the input image is first reduced in resolution through a pooling operation (such as maximum pooling or average pooling), and then processed through the SE module and DWConv convolution; Pool2D: represents a 2D pooling operation, which is used to downsample images or feature maps to reduce spatial resolution, usually using maximum pooling or average pooling.

3. The method according to claim 1, characterized in that The method further comprises: Based on the image feature extraction module, construct an image classification model; The image classification model includes a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in sequence. The four feature extraction modules are obtained by stacking the image feature extraction modules with preset proportions and channel numbers on four feature scales; the preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules. After the four image feature extraction modules perform dimensionality reduction processing on the preprocessed image features, the classification results are output through the pooling layer and the fully connected layer.

4. The method according to claim 3, characterized in that: After building the image classification model, the method further includes: Build a self-supervised learning framework and image dataset; The image classification model is trained using the self-supervised learning framework and the image data set, and image classification is achieved using the trained image classification model; when training the image classification model, the fully connected layer used for classification is replaced by a projection head, and the image classification model uses the sample image of the image data set as input, and maps the output sample image features to the self-supervised learning framework through the projection head for weight update, and the image classification model is configured with the updated weights to obtain the trained image classification model.

5. The method according to claim 4, characterized in that The self-supervised learning framework performs two different data augmentation operations in response to input sample image features to generate a first view and a second view, and encodes the first view and the second view through a query encoder and a momentum encoder; The momentum encoder is dynamically updated by querying the encoder weights, and the update formula is: f k1 =m×f k +(1-m)×f q In the formula, f k1 is the updated parameter of the momentum encoder, f k is the parameter of the momentum encoder before updating, f q is to query the current parameters of the encoder, m is the momentum coefficient, and its value ranges from 0 to 1; The weights of the query encoder are gradient updated using a contrastive loss function, which is expressed as: Where, L q is the contrast loss, k + is the positive sample key vector corresponding to q, k - is the negative sample key vector set, τ is the temperature parameter used to control the sensitivity of the similarity calculation, q·k + is the query vector q and its corresponding positive sample key vector k + exp is the exponential function, which is used to convert the dot product into a positive number to amplify the difference.

6. The method according to claim 5, characterized in that When training the image classification model, a multi-classification task loss function and KL divergence are used to guide the gradient descent process of the self-supervised learning framework; The multi-classification task loss function is expressed as: Where, L CE is the multi-classification task loss function value, C is the total number of categories, y i,c is the one-hot encoding representation of whether sample i belongs to category c, p i,c is the probability that the model predicts that sample i belongs to category c, and N is the total number of samples; The KL divergence is expressed as: Where α is a vector parameter of the Dirichlet distribution, Γ represents the gamma function, and S α and They represent the sum of parameters α and α0, ψ is the logarithmic derivative of the gamma function, K is the number of categories, α0 is the parameter of the prior Dirichlet distribution, and represents the initial classification probability distribution; in the Bayesian framework, the prior distribution expresses the degree of belief when no data is seen; θ represents the sample drawn from the Dirichlet distribution, that is, the specific category probability value; α k represents the parameter of the kth category in the Dirichlet distribution, which is used to describe the weight of the kth category; p represents the distribution p(θ|α), that is, the Dirichlet distribution with parameter α, which is used to describe the posterior distribution of a certain category θ.

7. An image processing device, characterized in that: include: An image acquisition module is configured to acquire a target image; A feature extraction module is configured to extract features of the target image based on the image feature extraction module to obtain target image features; The image feature extraction module includes a sampling unit, a window holding unit and a grid holding unit connected in sequence; the sampling unit performs sampling in response to an input target image to obtain a first image feature, the window holding unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a self-attention mechanism to pixels in each non-overlapping block and then performs a multi-layer perceptron operation to obtain a second image feature; the grid holding unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, applies a self-attention mechanism to each grid and then performs a multi-layer perceptron operation to obtain a target image feature; The calculation process of retaining the self-attention mechanism is: O H ,K H =(Q,K) B,L,C→B,W,H,C O W ,K W =(Q,K) B,L,C→B,H,W,C ReSA dec (X)=Attn H (Attn W V) In the formula, O H Represents the feature map in the output height direction, K H represents the height directional feature of the key matrix. Q represents the query matrix, which is used to calculate the attention weight; K represents the key matrix, which is used to calculate the similarity with the query matrix; B, L, C represent the batch size, sequence length and number of channels respectively; O W Represents the feature map in the output width direction; K W Represents the width-wise characteristics of the key matrix; Attn H Represents the attention weight in the height direction; Softmax represents the operation used for normalization; represents the transpose of the key matrix. ⊙ represents element-wise multiplication; D H Represents the distance attenuation factor in the height dimension; Attn W Represents the attention weight in the width direction; represents the transpose of the key matrix in the width direction; D W Represents the distance attenuation factor in the width dimension; Indicates the distance attenuation weight between two pixels in the height direction. γ represents the distance attenuation factor, which is used to adjust the influence of far and near pixels; m Indicates the vertical coordinates of two pixels; Represents the distance attenuation weight between two pixels in the width direction; X m Represents the horizontal coordinates of two pixels; ReSA dec (X) represents the decoded output of the self-attention mechanism; X represents the input image or feature map; Attn W V represents the application of the attention weights in the width direction to the value matrix V.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images