Image processing method, device, apparatus and storage medium

Abstract

The application provides an image processing method, device and equipment and a storage medium. It relates to the technical field of image processing. The method comprises: obtaining a target image; performing feature extraction on the target image based on an image feature extraction module to obtain target image features; the image feature extraction module comprises a sampling unit, a window maintaining unit and a grid maintaining unit connected in sequence; and a self-attention mechanism is applied in the window maintaining unit and the grid maintaining unit. In the method, when performing feature extraction on a whole slide image, local features can be captured by focusing on information such as the shape and texture of cells, and global dependence can be obtained by learning the differences between cells and other cells and microenvironments, so that a decision can be made by combining local and global representations.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of computational pathology and artificial intelligence (AI) technology, the data processing methods based on whole slide image (WSI) are gradually increasing. At present, the methods for processing WSI are mainly divided into two categories: one is the multiple instance learning (MIL) method which focuses on the overall level classification of WSI, and the other is the supervised learning method which focuses on the more subtle patch level classification. The advantage of the MIL method is that it does not need a pathologist to manually label different tissue components and regions for all WSIs, and it directly predicts the WSI by converting the patch into a series of feature vector sequences. However, the MIL method does not perform well in the multi-classification task, and it cannot simultaneously realize the semantic segmentation of the cell tissue region in the WSI. In contrast, the supervised learning method based on patch level classification not only exhibits higher accuracy in the multi-classification task of WSI, but also can realize the segmentation of cell tissue, showing a broader clinical application potential.

[0003] In the classification task of the patch level, the existing models mainly use the method of convolutional neural network (CNN), Transformer, or the combination of the two. In the network based on CNN, the model lacks the understanding of global features, and needs an extremely deep network to obtain a larger receptive field, and the interpretability is poor. While the network based on Transformer relies heavily on prior knowledge, and its complexity grows with the square of the sequence length, resulting in a sharp increase in the demand for computing resources. SUMMARY

[0004] The present application provides an image processing method, device, equipment and storage medium to solve the problem of insufficient image feature capturing ability and high inference cost in the image processing method.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] obtaining a target image;

[0007] performing feature extraction on the target image based on an image feature extraction module to obtain target image features;

[0008] The image feature extraction module includes a sampling unit, a window preservation unit, and a grid preservation unit connected in sequence. The sampling unit samples the input target image to obtain a first image feature. The window preservation unit uses the first image feature as input, divides the first image feature into multiple non-overlapping blocks, and applies a self-attention mechanism to the pixels in each non-overlapping block before performing a multilayer perceptron operation to obtain a second image feature. The grid preservation unit uses the second image feature as input, divides the second image feature into several grids of uniform size, applies a self-attention mechanism to each grid, and performs a multilayer perceptron operation to obtain the target image feature.

[0009] The calculation process for retaining the self-attention mechanism is as follows:

[0010] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0011] O W ,K W =(Q,K) B,L,C→B,H,W,C

[0012]

[0013] ReSA dec (X)=Attn H (Attn W V)

[0014] In the formula, Q: query matrix, representing a linear transformation of the input features, used to calculate the attention score;

[0015] K: Key matrix, a linear transformation of the input features, used to calculate similarity with the query matrix;

[0016] O H and O W The height and width of the output feature matrix represent the spatial dimensions of the output features after the self-attention mechanism.

[0017] B: Batch size, indicating the number of samples input at one time;

[0018] L: Sequence length, corresponding to the length of the input features in sequence form;

[0019] C: Number of channels, representing the depth of each input feature;

[0020] K H and K W : Corresponds to the representation of the key matrix in the height and width dimensions, used to calculate spatial correlation;

[0021] DH and D W : represents a distance decay function, used to reduce the attention weight between distant pixel points;

[0022] γ: distance decay factor, used to control the degree of distance influence between different pixel points;

[0023] y n and y m : represents the coordinates of two pixel points in the image in the vertical direction;

[0024] x n and x m : represents the coordinates of two pixel points in the image in the horizontal direction.

[0025] In one possible design, the sampling unit samples the input target image to obtain a first image feature through the following formula:

[0026]

[0027] In the formula,

[0028] x: represents the input target image or feature map;

[0029] x': represents the first image feature, which is a feature map generated by the processing process;

[0030] Proj: represents the projection layer, which is a 1x1 convolution operation used to map or compress the channel number of the feature;

[0031] SE: represents the squeeze-excitation mechanism;

[0032] DWConv: represents the depthwise separable convolution, which decomposes the standard convolution into depthwise convolution and pointwise convolution, reduces the computational complexity, and retains good feature extraction ability;

[0033] Conv: represents the convolution operation, which refers to the standard 2D convolution operation, used to extract local features from the input image or feature map;

[0034] Norm: represents the normalization operation, which is batch normalization, used to stabilize the training process and accelerate convergence.

[0035] undownsample: represents not down-sampling, i.e. maintaining the original resolution of the image. At this time, the input image or feature map is directly projected after the squeeze-excitation and convolution operation without changing its spatial dimension;

[0036] downsample: represents downsampling, i.e. reducing the spatial resolution of an image or feature map through a pooling operation or a convolution operation. In this case, the input image is first reduced in resolution through a pooling operation (such as max pooling or average pooling), and then processed through the SE module and the DWConv convolution;

[0037] Pool2D: represents a 2D pooling operation, used for down-sampling an image or feature map, reducing the spatial resolution, usually using max pooling or average pooling.

[0038] In one possible design, the method further includes:

[0039] Based on the image feature extraction module, an image classification model is constructed;

[0040] The image classification model includes a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in sequence, the four feature extraction modules being obtained by stacking the image feature extraction module with a preset scale and channel number in four feature scales; the preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules, and the four image feature extraction modules perform dimension reduction processing on the preprocessed image features and output a classification result through the pooling layer and the fully connected layer.

[0041] In one possible design, after constructing the image classification model, the method further includes:

[0042] An auto-encoding learning framework and an image dataset are constructed;

[0043] The image classification model is trained using the auto-encoding learning framework and the image dataset, and the trained image classification model is used to implement image classification; when training the image classification model, the fully connected layer for classification is replaced by a projection head, the image classification model takes a sample image of the image dataset as input, and maps the output sample image features to the auto-encoding learning framework through the projection head for weight updating, and the image classification model is configured with the updated weights to obtain the trained image classification model.

[0044] In one possible design, the auto-encoding learning framework generates a first view and a second view by performing two different data augmentation operations in response to an input sample image feature, and encodes the first view and the second view through a query encoder and a momentum encoder;

[0045] The momentum encoder is dynamically updated through the weights of the query encoder, and the update formula is:

[0046] f k1 =m×fk + (1 - m) x f q

[0047] where f k1 is the updated parameter of the momentum encoder, f k is the previous parameter of the momentum encoder, f q is the current parameter of the query encoder, and m is the momentum coefficient, which is between 0 and 1.

[0048] The weights of the query encoder are updated using the gradient of a contrastive loss function, which is expressed as:

[0049]

[0050] where L q is the contrastive loss, k + is the positive sample key vector corresponding to q, k - is the set of negative sample key vectors, and τ is the temperature parameter used to control the sensitivity of the similarity calculation, q·k + is the dot product between the query vector q and its corresponding positive sample key vector k + , exp is the exponential function used to convert the dot product into a positive number to amplify the difference.

[0051] In one possible design, when training the image classification model, a multi-classification task loss function and KL divergence are used to guide the gradient descent process of the self-supervised learning framework.

[0052] The multi-classification task loss function is expressed as:

[0053]

[0054] where L CE is the multi-classification task loss value, C is the total number of classes, y i,c is the one-hot encoding representation of whether sample i belongs to class c, p i,c is the probability that the model predicts that sample i belongs to class c, and N is the total number of samples.

[0055] The KL divergence is expressed as:

[0056]

[0057] where α is a vector parameter of the Dirichlet distribution, Γ represents the gamma function, S α and denotes the sum of parameters a and a0, y is the logarithmic derivative of the gamma function, K is the number of classes, a0 is the parameter of the prior Dirichlet distribution, and denotes the initial classification probability distribution; in the Bayesian framework, the prior distribution expresses the belief when no data is seen; 0 denotes a sample drawn from the Dirichlet distribution, that is, a specific class probability value; a k denotes the parameter of the kth class in the Dirichlet distribution, which is used to describe the weight of the kth class; p denotes the distribution p(0|a), that is, the Dirichlet distribution with parameter a, which is used to describe the posterior distribution of a certain class 0.

[0058] In a second aspect, the present application provides an image processing device, comprising:

[0059] An image acquisition module configured to acquire a target image;

[0060] A feature extraction module configured to perform feature extraction on the target image based on an image feature extraction module to obtain target image features;

[0061] The image feature extraction module comprises a sampling unit, a window retention unit and a grid retention unit connected in sequence; the sampling unit samples the input target image to obtain first image features; the window retention unit takes the first image features as input, divides the first image features into a plurality of non-overlapping blocks, and applies a reserved self-attention mechanism to the pixel points in each non-overlapping block and then performs a multi-layer perception operation to obtain second image features; the grid retention unit takes the second image features as input, divides the second image features into a plurality of grids with the same size, applies a reserved self-attention mechanism to each grid and then performs a multi-layer perception operation to obtain target image features.

[0062] The calculation process of the reserved self-attention mechanism is as follows:

[0063] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0064] O W ,K W =(Q,K) B,L,C→B,H,W,C

[0065]

[0066] ReSA dec (X)=Attn H (Attn W V)

[0067] In the formula, O H denotes the output height direction feature map, and KH represents the height direction characteristics of the key matrix; Q represents the query matrix, which is used to calculate the attention weight; K represents the key matrix, which is used to calculate the similarity with the query matrix; B, L, and C represent batch size, sequence length, and channel number respectively; O W represents the output width direction feature map; K W represents the width direction characteristics of the key matrix; Attn H represents the attention weight in the height direction; Softmax represents the operation for normalization; represents the transpose of the key matrix. ⊙ represents element multiplication; D H represents the distance decay factor in the height dimension; Attn W represents the attention weight in the width direction; represents the transpose of the key matrix in the width direction; DW represents the distance decay factor in the width dimension; represents the distance decay weight between two pixel points in the height direction. γ represents the distance decay factor, which is used to adjust the influence of the pixel points far and near; y m represents the coordinates of two pixel points in the vertical direction; represents the distance decay weight between two pixel points in the width direction; X m represents the coordinates of two pixel points in the horizontal direction; ReSA dec (X) represents the decoding output of the reserved self-attention mechanism; X represents an input image or a feature map; Attn W V represents applying the attention weight in the width direction to the value matrix V.

[0068] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the image processing method as described in the above first aspect and various possible designs of the first aspect.

[0069] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when a processor executes the computer execution instructions, the image processing method as described in the above first aspect and various possible designs of the first aspect is implemented.

[0070] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the image processing method as described in the above first aspect and various possible designs of the first aspect is implemented.

[0071] The image processing method, device, equipment and storage medium provided by the application provide an image feature extraction module with good interpretability, local features are captured by a sampling unit and a window-based pixel-level reservation mechanism, and long-distance dependence is obtained by an inter-block reservation mechanism between grids. When the module extracts features of a whole slide image, it can not only focus on the shape, texture and other information of cells to capture local features, but also learn the differences between cells and other cells and microenvironments to obtain global dependence, and finally combine local and global representations to make decisions together. BRIEF DESCRIPTION OF DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application together with the specification.

[0073] Figure 1 The flowchart of the image processing method provided by the embodiment of the application;

[0074] Figure 2 The principle diagram of the reservation self-attention mechanism provided by the embodiment of the application, wherein (a) is a two-dimensional weight decay mechanism of the window reservation unit, and (b) is a two-dimensional weight decay mechanism of the grid reservation unit;

[0075] Figure 3 The detailed schematic diagram of the SCRTrans model structure provided by the embodiment of the application;

[0076] Figure 4 The detailed self-supervised learning flowchart provided by the embodiment of the application;

[0077] Figure 5 The feature maps of the SCRTrans model at different scales provided by the embodiment of the application, wherein each small square represents each window divided by the model, and the window size doubles with the stage;

[0078] Figure 6 The visualization schematic diagram of different models provided by the embodiment of the application, wherein (a) is the T-SNE visualization of the K-Means-CNN model, (b) is the T-SNE visualization of the UNI fine-tuning model, and (c) is the T-SNE visualization of the SCRTrans model;

[0079] Figure 7 The schematic diagram of real labels and predicted labels provided by the embodiment of the application;

[0080] Figure 8The confusion matrix of model identification performance provided in the embodiments of the present application, wherein: (a) the confusion matrix of the wsi-level SCRTrans model diagnosis performance on the Ebrains dataset, (b) the confusion matrix of the wsi-level SCRTrans model diagnosis performance on the Southwest dataset;

[0081] Figure 9 The structural schematic diagram of the image processing device provided in the embodiments of the present application is shown in the following figure.

[0082] Figure 10 The structural schematic diagram of the electronic device provided in the embodiments of the present application is shown in the following figure.

[0083] Through the above figures, the specific embodiments of the present application have been shown, and will be described in more detail hereinafter. These figures and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0084] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data or medical image data, etc. comply with relevant laws and regulations and do not violate public order and good customs.

[0086] It should be noted that in the embodiments of the present application, some existing industry solutions may be mentioned, such as software, components, models, etc. They should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0087] It should be noted that in the embodiments of the present application, some existing industry solutions may be mentioned, such as software, components, models, etc. They should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0088] Nowadays, in the data processing process of the whole slide image, the existing data processing methods such as using convolutional neural network (CNN), Transformer, or the combination of the two, are difficult to meet the key goals of high accuracy, strong interpretability and low deployment cost at the same time.

[0089] To solve the above problems, the present application proposes an image processing method, device and equipment, and a storage medium, and designs an efficient image feature module (MB-RetNet module). First, a sampling unit (MBConv module) is used to capture the spatial relationship and local features in the image, and the spatial information is indirectly processed during the learning process, thereby replacing the explicit position encoding. Then, based on the window retention unit, the output features are divided into several windows, and the retentive network mechanism is used for each pixel point in the window to capture the local details of the image. Next, based on the grid retention unit, the output features are further divided into grids, and the retentive network mechanism is used between all grids to obtain the dependence of the global context. This design makes the image feature module not only have a good understanding of the local details such as the shape and texture of the pathological cells, but also fully learn the relationship between the pathological cells and the surrounding microenvironment, so that the model's decision-making is more in line with human standards. In addition, due to the advantages of RetNet such as low-cost reasoning, efficient long sequence modeling, comparable performance with Transformer, and parallel model training, the MB-RetNet module not only has excellent feature capturing ability, but also greatly reduces the reasoning cost.

[0090] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0091] The present application provides an image processing method. Figure 1 The flowchart of the image processing method provided by the embodiments of the present application is shown in FIG. 1. Figure 1As shown, the image processing method can be implemented based on an electronic terminal, which starts from step S100, obtaining a target image. The target image refers to various images meeting the set conditions, including but not limited to whole slide images. The set conditions can be the format, resolution, and pixels of the image. For example, if an image is within the set format range, and the resolution and pixels both meet the set standards, the image can be used as the target image. Of course, the above set conditions are only examples and are not a limitation of the present application. When step S100 is implemented in an electronic terminal, the electronic terminal can obtain the target image by importing or transmitting through communication interaction with other terminals.

[0092] Step S200, based on the image feature extraction module, performing feature extraction on the target image to obtain target image features.

[0093] The image feature extraction module includes a sampling unit, a window retention unit, and a grid retention unit connected in sequence; the sampling unit samples the input target image to obtain first image features; the window retention unit takes the first image features as input, divides the first image features into a plurality of non-overlapping blocks, and applies a reserved self-attention mechanism to the pixels in each non-overlapping block before performing a multi-layer perception operation to obtain second image features; the grid retention unit takes the second image features as input, divides the second image features into a plurality of grids of the same size, applies a reserved self-attention mechanism to each grid, and then performs a multi-layer perception operation to obtain target image features.

[0094] In this embodiment, the reserved self-attention mechanism is applied in the window retention unit and the grid retention unit to capture more image features.

[0095] In particular, the retention self-attention mechanism is implemented based on the RMT module. RentNet is considered as a powerful successor of CNN and Transformer, showing significant advantages in large-scale models and long sequence tasks. RetNet is composed of multi-scale retention (MSR) module and feed-forward network (FFN) module, following a similar layout as Transformer. In particular, the retention self-attention mechanism is implemented based on the RMT module (Retention Multi-Task Module). RetNet is considered as a powerful successor of Convolutional Neural Network (CNN) and Transformer, showing significant advantages in large-scale models and long sequence tasks. RetNet is composed of multi-scale retention module (Multi-Scale Retention, MSR) and feed-forward network module (Feed-Forward Network, FFN), which has a layout similar to Transformer.

[0096] RetNet uses a retention mechanism to replace multi-head attention, and uses a decay matrix to multiply a weight matrix to control the proportion of each pixel to see the surrounding pixels. RentNet supports three computing paradigms: parallel, recurrent and chunkwise recurrent. Parallel representation allows full use of GPU training parallelism. Recurrent representation reduces the inference cost to O(1), improves decoding throughput, latency and GPU memory usage efficiency, while not sacrificing performance. Chunkwise recurrent representation helps to efficiently process long sequences with linear complexity, each block parallel encoding, while the loop summarizes the block information. The retention mechanism first considers using a cyclic way for sequence modeling, which can be written as equation (1):

[0097]

[0098] wherein, is the conjugate transpose, O n represents the output in sequence modeling, that is, the output feature of the current time n, which combines the feature information of the previous time m.

[0099] γ represents a position decay factor, which is used to control the attention between different positions (time steps). As the position difference n-m increases, the weight γ^(n-m) gradually decays, so as to pay more attention to the input closer to the current time step.

[0100] m represents the time step, which traverses the previous time steps. θ is a phase factor, which is used to encode time or position information. K mKey vector, used to capture the features at the m-th time step in the sequence. m Value vector at the m-th time step in the sequence, which carries specific information at this time step.

[0101] For the parallel training paradigm, it can be written as equation (2):

[0102]

[0103] Retention(X) = (QK T ⊙D)V

[0104] In the formula, Q: represents the query vector (Query matrix), which is a vector matrix obtained by linearly transforming the input X Q After obtaining, it is used to calculate the attention score with the key vector K.

[0105] X: represents the input feature matrix, which contains the input information of all time steps.

[0106] W Q : represents the weight matrix of the query vector, which is used to linearly transform the input X to generate the query vector.

[0107] Θ: represents the position encoding matrix, which is used to introduce position information, usually using phase factors to represent the difference in time or position.

[0108] K: represents the key vector (Key matrix), which is generated by linearly transforming the input feature X through the weight matrix W K , represents the relevance of the features.

[0109] W K : represents the weight matrix of the key vector, which is used to linearly transform the input X to generate the key vector.

[0110] Similar to Θ, but contains different position information (such as transposed position information).

[0111] V: represents the value vector (Value matrix), which is a vector matrix obtained by linearly transforming the input X through the weight matrix W V , contains specific information of each time step in the sequence. V : represents the weight matrix of the value vector, which is used to linearly transform the input X to generate the value vector.

[0112] Θ N : represents the phase encoding matrix at the N-th time step, represented by e inθ , which is an encoded form of position information.

[0113] D nm : represents the position attenuation matrix, which is used to represent the distance attenuation relationship between time steps n and m. For the case of n≥m, the attenuation factor is γ^(n-m); for the case of n≤m, the attenuation factor is 0.

[0114] Retention(X): represents the result of input X after query, key, value matrix operation under the retention mechanism, which uses attention mechanism QK T ⊙D calculates the attention score and combines the value matrix V to get the final output.

[0115] D: represents the attenuation matrix, which controls the attention weight distribution between each time step, ensuring that the model pays more attention to the input features close to the current time step.

[0116] RMT adds bidirectional modeling and two-dimensional expansion based on RetNet, and retains the Softmax operator. At the same time, in order to reduce the overhead of the model, the retention operator is decomposed into two axes of the image. This is the core module of RMT, called Retentive Self-Attention (ReSA), which can be written as equation (3):

[0117] Q H ,K H =(Q,K) B,L,C→B,W,H,C

[0118] O W ,K W =(Q,K) B,L,C→B,H,W,C

[0119]

[0120] ReSA dec (X)=Attn H (Attn W V)

[0121] In the formula, Q: query matrix, represents the linear transformation of input features, used to calculate the attention score;

[0122] K: key matrix, is a linear transformation of input features, used to calculate similarity with query matrix;

[0123] O H and O W : the height and width of the output feature matrix, representing the spatial dimension of the output feature after self-attention mechanism;

[0124] B: batch size, representing the number of samples input at a time;

[0125] L: sequence length, corresponding to the length of the input feature in the sequence form;

[0126] C: channel number, representing the depth of each input feature;

[0127] K H and K W : corresponding to the representation of the key matrix in the height and width dimensions, used to calculate the spatial correlation;

[0128] D H and D W : representing the distance decay function, used to reduce the attention weight between distant pixel points;

[0129] γ: distance decay factor, used to control the degree of distance influence between different pixel points;

[0130] y n and y m : representing the coordinates of two pixel points in the image in the vertical direction;

[0131] x n and x m : representing the coordinates of two pixel points in the image in the horizontal direction.

[0132] The following embodiment will describe in detail the process of image feature extraction by the image feature extraction module. In this embodiment, the image feature extraction module is referred to as the MB-RetNet module, and the sampling unit included therein is referred to as the MBConv module.

[0133] The MBConv module determines whether to down-sample by controlling the stride in the Depthwise Conv3x3. Specifically, assuming that x is the input feature, the operation process of the MBConv module can be written as formula (4):

[0134]

[0135] In the formula,

[0136] x: representing the input target image or feature map;

[0137] x': representing the first image feature, which is a feature map generated by the processing process;

[0138] Proj: representing the projection layer, which is a 1x1 convolution operation, used to map or compress the channel number of the feature;

[0139] SE: representing the squeeze-excitation mechanism;

[0140] DWConv: represents a depthwise separable convolution, which decomposes a standard convolution into a depthwise convolution and a pointwise convolution, reducing computational complexity while preserving good feature extraction ability;

[0141] Conv: represents a convolution operation, which refers to a standard 2D convolution operation used to extract local features from input images or feature maps;

[0142] Norm: represents a normalization operation, which is batch normalization, used to stabilize the training process and accelerate convergence.

[0143] undownsample: represents no downsampling, i.e., maintaining the original resolution of the image. At this time, the input image or feature map is directly projected after squeeze-and-excitation and convolution operations without changing its spatial dimension;

[0144] downsample: represents downsampling, i.e., reducing the spatial resolution of the image or feature map through pooling operations or convolution operations. In this case, the input image is first reduced in resolution through a pooling operation (such as max pooling or average pooling), and then processed through the SE module and DWConv convolution;

[0145] Pool2D: represents a 2D pooling operation, which is used to downsample images or feature maps, reducing spatial resolution, usually using max pooling or average pooling.

[0146] For the first MBConv block of each stage, downsampling is performed by applying stride-2Depthwise Conv3x3, while the shortcut branch should also apply pooling and channel projection.

[0147] The window maintaining unit takes the first image feature x' as input, performs window operation, which is defined as dividing the input first image feature into a plurality of non-overlapping blocks with size P x P, where H, W, C represent the height, width and channel number of the image respectively. This operation can be represented as equation (5):

[0148]

[0149] After the window operation, the dimensions of the window are aggregated into the spatial dimension (i.e., the -2 axis). Then, the Retention Self-Attention (ReSA) is applied to the pixel points in each window, which can be represented by equation (6):

[0150] x'1←x'+Unblock(RelAttention(Block(LN(x'))))

[0151] x”←x'1+MLP(LN(x'1)) (6)

[0152] where LN denotes layer normalization, MLP denotes a multi-layer perceptron comprising two linear layers, x'1 denotes the image feature after applying the windowed self-attention mechanism to the pixels in each non-overlapping block, and x" denotes the second image feature.

[0153] The grid holding unit takes the second image feature x" as input, performs a grid operation, which is defined as dividing the input feature into GxG uniform grids, each cell has adaptive size Unlike the window operation, the grid operation requires an additional transpose operation to place the grid dimension on the assumed spatial axis (i.e., -2 axis), where the axis swapping operation involved is swapaxes(axis1=-2, axis2=-3). This operation can be represented as equation (7):

[0154]

[0155] Then apply the Retentive self-attention mechanism (ReSA) to each grid, which can be represented by equation (8):

[0156]

[0157] where LN denotes layer normalization, MLP denotes a multi-layer perceptron comprising two linear layers, x'1' denotes the image feature after applying the Retentive self-attention mechanism to each grid, and x'" denotes the third image feature.

[0158] The operation process here is similar to the windowed Retentive mechanism, but the grid operation calculates the Retentive between grids.

[0159] Through the combination of the MBConv module, the window holding unit and the grid holding unit, the MB-RetNet module is formed. In the MB-RetNet module, the MBConv module obtains the spatial information and positional relationship between elements while performing deep convolution on the element image or feature. In the subsequent windowed Retentive and grid Retentive mechanisms, explicit positional encoding is cancelled. Experiments show that the model performance is almost unchanged, and the speed is greatly improved when training in parallel. In the Retentive self-attention mechanism, each token will perform equal self-attention calculation with all other tokens, and the complexity of the calculation is proportional to the square of the sequence length. The Retentive network uses the position decay mechanism to focus on the calculation of each token and its surrounding tokens, and combines recursive reasoning to greatly reduce the complexity. The principle is as follows Figure 2As shown, each pixel calculates position decay based on its Manhattan distance to surrounding pixels, making the pixel more sensitive to its nearby periphery and less sensitive to distant objects. This position decay preservation mechanism is particularly suitable for WSI because each cell or tissue in WSI can have an independent label, unlike in natural images where only the constituent patterns or the entire image have a meaningful label. Specifically, by adjusting the pixel size to be equal to the physical size of a cell, calculating inter-mesh memory preservation becomes about the cell finding its relationship with neighboring cells or its surrounding microenvironment.

[0160] In some embodiments, the image processing method further includes the following steps:

[0161] S300. Based on the image feature extraction module, an image classification model is constructed. The image classification model includes a preprocessing module, four image feature extraction modules, a pooling layer, and a fully connected layer connected in sequence. The four image feature extraction modules are obtained by stacking the image feature extraction modules at four feature scales with preset ratios and channel numbers. The preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules. After the four image feature extraction modules perform dimensionality reduction processing on the preprocessed image features, they output the classification result through the pooling layer and the fully connected layer.

[0162] The image classification model constructed in this embodiment is as follows: Figure 3 As shown, this image classification model is referred to as the SCRTrans model in this embodiment, and its preprocessing module is referred to as the Stem module. For example, by stacking MB-RetNet modules at four feature scales according to different ratios and channel numbers, two versions of the ConvolutionRetentive Network Transformer (SCRTrans) are obtained: Small and Base. Specific parameter configurations are shown in Table 1. The Small version is constructed by stacking MB-RetNet modules in a 1:1:1:1 ratio, making it a lightweight model suitable for low-cost, fast inference scenarios. The Base version, on the other hand, is constructed by stacking MB-RetNet modules in a 2:2:5:2 ratio, offering a larger number of parameters and higher accuracy, making it suitable for scenarios requiring high accuracy and requiring well-equipped deployment devices.

[0163] Table 1. Parameters of the Image Classification Model (SCRTrans Model)

[0164]

[0165] It should be noted that other versions of the SCRTrans model can also be configured based on other ratios. The above two versions are only examples and do not constitute a limitation on the present application.

[0166] As shown in Figure 3 The SCRTrans model is composed of 1 Stem module, 4 MB-RetNet modules, 1 pooling layer and 1 fully connected layer. The features are reduced in the S0 to S4 modules, and finally the classification results are output through the pooling layer and the fully connected layer. The Stem module is composed of 2 two-dimensional convolution layers and GELU activation layers. The convolution kernel size is 3x3, and the feature map scale is down-sampled by setting the stride of the first convolution layer to 2. Each MB-RetNet module is divided into 3 parts. In the first part, the feature is first passed through an MBConv module, which is a MobileNetV2-based convolution block that uses depth separable convolution for efficient feature extraction and incorporates the down-sampling step. In the second part, the window partition function divides the feature map into multiple small windows, and then applies the retention memory mechanism to each window to process local features. In the third part, similar to window retention, but using grid partitioning, which allows the retention memory module to capture interactions between features in a larger range and process more global information. Finally, by processing image features at different scales, the SCRTrans model can both process local details and integrate global information, ultimately achieving a deep understanding of image content.

[0167] In some embodiments, after building the image classification model, the method further comprises: building a self-supervised learning framework and an image dataset; training the image classification model using the self-supervised learning framework and the image dataset, so that the trained image classification model realizes image classification; when training the image classification model, the fully connected layer for classification is replaced by a projection head, the image classification model takes a sample image of the image dataset as input, and maps the output sample image feature to the self-supervised learning framework through the projection head for weight update, and configures the image classification model with the updated weights to obtain the trained image classification model.

[0168] Considering the inherent characteristics of WSI, i.e. the similarity of patches obtained from the same WSI is very high, direct training is extremely prone to overfitting. Therefore, the present embodiment improves Moco v3 to develop a self-supervised learning framework that can learn the inherent features of patches without labels, and the detailed process is as follows Figure 4 In this framework, the model learns to distinguish different image representations by comparing positive and negative samples, thereby improving the learning quality of visual representations.

[0169] In some embodiments, the self-supervised learning framework performs two data augmentation operations for each image, generating two different views x1 and x2, which are encoded by the query encoder f q and the momentum encoder f k respectively. The momentum encoder does not directly perform training updates, but instead dynamically updates by the weights of the query encoder, which can be expressed as formula (9):

[0170] f k = m x f k + (1 - m) x f q (9)

[0171] where f k is the parameter of the momentum encoder before updating, f q is the current parameter of the query encoder, and m is the momentum coefficient, which takes a value between 0 and 1. The momentum coefficient m determines the degree to which f k maintains the old parameter and the degree to which f q receives the new parameter from f k . Selecting a higher m value can make f + change more smoothly, which is particularly beneficial for handling large-scale datasets, as it can provide more stable feature representations to combat sample noise and variation.

[0172] The gradient update of the query encoder, on the other hand, directly depends on the contrastive loss, which aims to make the feature representations of two views of the same image (positive sample pair) similar, while making the feature representations of views from other images (negative sample pairs) dissimilar. The loss function can be expressed as formula (10):

[0173]

[0174] where k + is the positive sample key vector corresponding to q, k - is the set of negative sample key vectors, and τ is the temperature parameter, which controls the sensitivity of the similarity calculation. q·k + is the dot product between the query vector q and its corresponding positive sample key vector k + . The size of the dot product represents the similarity between the two vectors, and the larger the dot product, the higher the similarity. The exponential function exp can convert the dot product into a positive number, thereby amplifying the difference and making the influence of high-similarity vector pairs more significant. The temperature parameter τ can adjust the range of values after the dot product, thereby affecting the "smoothness" of the softmax function. A smaller τ will make the output more sharp (i.e., more highlighting high-similarity vector pairs), while a larger τ will make the output more smooth (i.e., reducing the influence of extreme similarity values).

[0175] In this self-supervised learning framework, both query encoder and vector encoder are modified from the SCRTrans model, and the fully connected layer for classification is replaced by a projection head. Since it has been found in Moco v3 that self-supervised ViT can be unstable, the Stem module in the SCRTrans model is frozen to avoid the same problem.

[0176] In some embodiments, in order to capture the uncertainty assessment of the model's prediction while improving the prediction accuracy, CrossEntropyLoss and Kullback-Leibler divergence are used to guide the gradient descent process together. CrossEntropyLoss is a loss function for multi-classification tasks, mainly focusing on maximizing the probability of correct classification. It measures the cross-entropy between the probability distribution of the model output and the probability distribution of the true label. It can be expressed as formula (11):

[0177]

[0178] where C is the total number of classes, y i,c is the one-hot encoding representation of whether sample i belongs to class c, p i,c is the probability of the model predicting that sample i belongs to class c.

[0179] In machine learning, uncertainty can be generally divided into model uncertainty and data uncertainty. Model uncertainty, also known as apparent uncertainty or cognitive uncertainty, is related to the limitations of the model itself, such as the uncertain prediction of the model for patterns not seen in the data. Data uncertainty, also known as intrinsic uncertainty, is related to the noise of the data itself, and even the best model cannot explain the randomness. By modeling the output of the SCRTrans model as the parameters of the Dirichlet distribution and using Kullback-Leibler divergence to measure the difference between the predicted distribution and the prior distribution, the uncertainty of the model is quantified. For Dirichlet distribution, it can be expressed as formula (12):

[0180]

[0181] where α is a vector parameter of Dirichlet distribution, Γ represents the gamma function, S α and denotes the sum of the a and a0 parameters, respectively. ψ is the digamma function, the logarithmic derivative of the gamma function, and K is the number of classes. If the model's predictions (described by the parameters a of the Dirichlet distribution) are close to a uniform distribution (all a are equal), then the uncertainty is high; if some a are significantly larger than others, it indicates that the model is more confident in its predictions for certain classes, i.e. the uncertainty is low. θ denotes a sample drawn from the Dirichlet distribution, i.e. the concrete class probability values; a k denotes the parameter of the k-th class in the Dirichlet distribution, which is used to describe the weight of the k-th class; p denotes the distribution p(θ|a), i.e. the Dirichlet distribution with parameters a, which is used to describe the posterior distribution of a certain class θ.

[0182] The following embodiment will fully illustrate the feasibility and progressiveness of the present application by combining the method with the case of processing medical image data. It should be noted that the image processing method described in the above embodiment is not limited by the following case, and in some embodiments, it can also be applied to target cell recognition in cell images, etc. The application of the WSI image described herein is only an example and does not constitute a limitation on the present application.

[0183] This embodiment collects two data sets of 1798 patients, of which the Ebrain open source data set contains 1368 patients of brain tumor WSI, and the Southwest private data set contains 430 patients of brain tumor WSI. For the two data sets, the WSI is divided into a training set and a validation set according to a ratio of 7:3.

[0184] In this embodiment, two professional pathologists fine-tuned and reviewed the original brain tumor WSI of two datasets, and the annotations included tumor regions, normal regions, blood vessel regions, necrotic regions, and non-diagnostic regions. At the same time, in order to ensure the independence of the validation set, the patches used for training the model were all from the WSI of the training set. In the WSI cutting stage, the real physical size represented by each pixel grid of the WSI was adjusted to 1 μm. Then, the ROI region annotated by the pathologist was cut into patches with a size of 256*256, and according to the real category of the patch, it was divided into 7 categories of tumor tissue, normal tissue and non-diagnostic tissue. Next, a double filtering mechanism was used for data cleaning. First, according to the brightness and saturation in the HSV color space, the proportion of white areas was judged, and according to the threshold of 50%, the patches with more white areas were filtered out. In the second filtering mechanism, the CellularityScorer and NucleiScorer in the Histolab library were used to score the cell content of each patch, and then the patches with scores less than 0.01 from both scorers were filtered out. In order to reduce the influence of different staining components in the WSI on the model decision, the Reinhard method in the HistomicsTK library was also used to normalize the colors of all patches. The Histolab library can be referred to in the following literature: histolab / histolab:Lib raryforDigitalPathologyImageProcessing(github.com) and DigitalSlideArchive / HistomicsTK:APythontoolkitforpathologyimageanalysis algorithms.(github.com) .

[0185] In this embodiment, random data augmentation (including 15 operations) and image normalization (based on ImageNet mean and standard deviation) were implemented to improve the generalization ability of the model. The experiment used the SGD optimizer and a batch size of 256, the weight decay was 7E-5, the initial learning rate was 0.001, and the cosine annealing scheduler was used to adjust the learning rate. The loss function used in the model training process is CrossEntropyLoss and Kullback-Leibler divergence to guide the gradient descent process of the model. The model training was carried out in an environment equipped with NVIDIA A6000 graphics card, the batch size was set to 256, and 5-fold cross-validation was used to ensure the randomness of the training set and the validation set. Finally, a comprehensive set of refined indicators were used to evaluate the pathological grading performance of all models, including: area under the receiver operating characteristic curve (AUC), Top-1 accuracy (Top1-Acc), accuracy (Acc), mean class accuracy (MCA), mean precision mean (MAP), F1 score, specificity and sensitivity. These indicators together provide a detailed and comprehensive framework for analyzing the performance of the model.

[0186] To make the model learn more rich and general feature representation, while reducing the risk of overfitting and accelerating convergence, the embodiment uses pre-training and fine-tuning for transfer learning. At the same time, supervised learning, self-supervised learning, and their combination are introduced, and the effects of different methods are evaluated under the same conditions. In supervised learning, ResNet and ViT are selected as the model backbone, and pre-trained on ImageNet to introduce prior knowledge of natural images. In self-supervised learning, SimCLR and BYOL models are used for contrastive learning on the Ebrain training set. In the combination algorithm, the SupCon model is used for supervised contrastive learning on the Ebrain training set. The model backbone and pre-training parameters used in self-supervised learning and supervised contrastive learning are consistent with the paper source code.

[0187] In the fine-tuning stage, the classification head or projection head of the pre-trained model is replaced by the fully connected layer required for brain tumor classification, and the pre-training information is preserved by freezing the model weights. Fine-tuning uses the stochastic gradient descent (SGD) optimizer and a batch size of 256, and uses a cosine annealing scheduler and preheating to dynamically adjust the learning rate. The total number of epochs for all model fine-tuning is set to 100, and the Early Stop strategy is used to train the classification head on the Ebrain training set. Finally, all models are evaluated for multiple indicators, including Top-1 Accuracy (Top1-Acc), Mean Class Accuracy (MCA), and Mean Average Precision (MAP), as shown in Table 2. Compared with ImageNet pre-training used in most researches, the self-supervised method proposed in this application improves Top1-Acc by 18 percentage points, showing stronger robustness.

[0188] In the fine-tuning stage, the classification head or projection head of the pre-trained model is replaced by the fully connected layer required for brain tumor classification, and the pre-training information is preserved by freezing the model weights. Fine-tuning uses the stochastic gradient descent (SGD) optimizer and a batch size of 256, and uses a cosine annealing scheduler and preheating to dynamically adjust the learning rate. The total number of epochs for all model fine-tuning is set to 100, and the Early Stop strategy is used to train the classification head on the Ebrain training set. Finally, all models are evaluated for multiple indicators, including Top-1 Accuracy (Top1-Acc), Mean Class Accuracy (MCA), and Mean Average Precision (MAP), as shown in Table 2. Compared with ImageNet pre-training used in most researches, the self-supervised method proposed in this application improves Top1-Acc by 18 percentage points, showing stronger robustness.

[0189] Table 2 Model performance evaluation under different pre-training strategies

[0190]

[0191] By releasing more layers for training in the fine-tuning stage, the SCRTrans model further improves the accuracy at the patch level.

[0192] The SCRTrans model is a multi-scale learning model that covers the learning of different cell sizes and tissue features from fine-grained to coarse-grained. At the same time, the construction of the model is combined with the knowledge of tumor pathology. First, the cell micron size is aligned with the division window, and then the retention calculation between pixel points is performed. As shown in Figure 5 , 2 original glioma patches are randomly selected for display, and different tissue components are separated by a dashed line by a pathologist. In stage 1, each window size is divided into 16 μm, which can contain 2 red blood cells or 1 other cell. From the feature map of stage 1, it can be found that the model accurately captures the features of single cells with smaller sizes. In stage 2, the window size is doubled to 32 μm, and the model can capture the features of some larger cells and tissues. The window size of stage 3 continues to double to 64 μm, and the window size can almost cover any size of cell because the cell size in brain tumor pathology sections is usually less than 40 μm. In stage 4, the size of each window is doubled to 128 μm, and each window may contain 3-6 normal cells or tumor cells. As shown in the stage 4 feature map, the model can capture larger tissues and deeper features at this time, including some larger cell groups or tissues, and the model can see the obvious brightness contrast between the tumor cell group and other tissue area windows.

[0193] This embodiment realizes a multi-scale learning framework that can capture single cells to cell groups or tissue blocks by introducing prior knowledge of pathology into the design of the SCRTrans model architecture. This multi-scale learning can effectively capture the tissue features in the 0-128 μm range in the pathology WSI and perform retention calculation of pixel points within the window, thereby enhancing the model's ability to perceive local features. By combining cell size with multi-scale division windows, the model learns from small red blood cells and gradually expands to cell groups and tissues, allowing the model to have a deep semantic understanding of cells or tissues of various sizes. Finally, the local features extracted by the window are combined with the grid retention to establish a context relationship, allowing the model not only to have a good semantic understanding of its own window content, but also to establish a relationship between different windows, and ultimately achieve global perception of the entire image.

[0194] To visually demonstrate the effectiveness of different models, this embodiment uses T-SNE dimensionality reduction for feature visualization, as shown in Figure 6 The data used for dimensionality reduction by different models on the Ebrains validation set all come from the Ebrans validation set. By uniformly sampling the patches cut from each WSI, a patch-level visualization validation set consisting of 27000 images in 9 categories is formed.

[0195] To explore the attention of the SCRTrans model to the features of different brain tumor categories, 3 types of brain tumor patches were randomly selected for attention visualization, as shown in Figure 7

[0196] After training the patch-level deep learning model, unlike the majority voting strategy used by most studies to form the WSI-level decision result. Based on the uncertainty output of the model, a set of highly reliable patch-level prediction results is formed, and an effective diagnostic decision mechanism is constructed. The first step is the patch filtering stage, which first excludes patches with a predicted category of Nondiagnostic, and then filters out the last 50% of patches based on the uncertainty score in the remaining patches. The second step is the benign judgment stage, in which the proportion of patches predicted as normal is first calculated, and if the proportion is greater than 80%, the WSI type is judged to be normal. If the second step is not true, the third step of tumor type decision is entered. In the third step, all patches predicted as normal are filtered out again, and finally a majority vote is taken on the remaining patches.

[0197] Figure 8 The confusion matrices of the WSI-Level prediction results of the SCRTrans model on the Ebrains dataset and the Southwest dataset are shown respectively. On the Ebrains dataset, a total of 186 WSI in the independently divided validation set were predicted. It was found that only 2 were predicted incorrectly, of which 1 was a brain tumor of the type ependymoma predicted as glioma, and 1 was a brain tumor of the type ependymoma predicted as metastases, and all the remaining WSI were predicted correctly. In the Southwest dataset, a total of 131 WSI in the independently divided validation set were predicted, and it was found that all were predicted correctly. The diagnostic decision mechanism constructed based on the screening mechanism of model uncertainty effectively eliminates patches with less confident predictions, thereby greatly enhancing the WSI-Level diagnostic ability.

[0198] ​Based on the accurate classification performance of the SCRTrans model at the patch-level, a system for segmenting the WSI is constructed. The system can display the instance segmentation results of the tumor area, normal cell area and non-diagnostic area in the WSI by combining the label and position of each patch in the WSI.

[0199] The accurate tissue segmentation can assist the pathologist to make a more detailed report. The present application ingeniously integrates the prior knowledge of pathology into the design of the model through the cell micron alignment technology, and learns the characteristics of cells and tissues of different sizes in a multi-scale manner.

[0200] Figure 9 The structure schematic diagram of the image processing device provided by the embodiment of the present application is provided. The embodiment of the present application also provides an image processing device. As shown in the figure, Figure 9 The image processing device comprises:

[0201] The image acquisition module 901 is configured to acquire a target image;

[0202] The feature extraction module 902 is configured to perform feature extraction on the target image based on an image feature extraction module to obtain target image features;

[0203] The image feature extraction module comprises a sampling unit, a window retention unit and a grid retention unit connected in sequence; the sampling unit samples in response to the input target image to obtain first image features; the window retention unit takes the first image features as input, divides the first image features into a plurality of non-overlapping blocks, and applies a reserved self-attention mechanism to the pixel points in each non-overlapping block and then performs a multi-layer perception operation to obtain second image features; the grid retention unit takes the second image features as input, divides the second image features into a plurality of grids with the same size, applies a reserved self-attention mechanism to each grid and then performs a multi-layer perception operation to obtain target image features;

[0204] The calculation process of the reserved self-attention mechanism is as follows:

[0205] O H ,K H =(Q,K) B,L,C→B,W,H,C

[0206] O W ,K W =(Q,K) B,L,C→B,H,W,C

[0207]

[0208] ReSA dec (X)=Attn H (AttnW V)

[0209] In the formula, O H represents the output height direction feature map, K H represents the height direction feature of the key matrix. Q represents the query matrix, which is used to calculate the attention weight; K represents the key matrix, which is used to calculate the similarity with the query matrix; respectively represent the batch size, sequence length and channel number; O W represents the output width direction feature map; K W represents the width direction feature of the key matrix; Attn H represents the attention weight in the height direction; Softmax represents the operation for normalization; represents the transpose of the key matrix. ⊙ represents element multiplication; D H represents the distance decay factor in the height dimension; Attn W represents the attention weight in the width direction; represents the transpose of the key matrix in the width direction; D W represents the distance decay factor in the width dimension; represents the distance decay weight between two pixel points in the height direction. γ represents the distance decay factor, which is used to adjust the influence of the pixel points far and near; represents the coordinates of the two pixel points in the vertical direction; represents the distance decay weight between two pixel points in the width direction; X m represents the coordinates of the two pixel points in the horizontal direction; ReSA dec (X) represents the decoding output of the residual self-attention mechanism; X represents the input image or feature map; Attn W V represents applying the attention weight in the width direction to the value matrix V.

[0210] In some embodiments, the sampling unit samples to obtain the first image feature by the following formula in response to the input target image:

[0211]

[0212] In the formula,

[0213] x: represents the input target image or feature map;

[0214] x': represents the first image feature, which is a feature map generated by the processing process;

[0215] Proj: represents the projection layer, which is a 1x1 convolution operation, used to map or compress the channel number of the feature;

[0216] SE: represents the squeeze-excitation mechanism;

[0217] DWConv: represents a depthwise separable convolution, which decomposes a standard convolution into a depthwise convolution and a pointwise convolution, reducing computational complexity while preserving good feature extraction ability;

[0218] Conv: represents a convolution operation, which refers to a standard 2D convolution operation used to extract local features from input images or feature maps;

[0219] Norm: represents a normalization operation, which is batch normalization, used to stabilize the training process and accelerate convergence.

[0220] undownsample: represents no downsampling, i.e., maintaining the original resolution of the image. At this time, the input image or feature map is directly projected after the squeeze-and-excitation and convolution operations without changing its spatial dimension;

[0221] downsample: represents downsampling, i.e., reducing the spatial resolution of the image or feature map through a pooling operation or a convolution operation. In this case, the input image is first reduced in resolution by a pooling operation (such as max pooling or average pooling), and then processed by the SE module and DWConv convolution;

[0222] Pool2D: represents a 2D pooling operation, which is used to downsample images or feature maps, reducing spatial resolution, usually using max pooling or average pooling.

[0223] In some embodiments, the image processing device further comprises an image classification module configured to:

[0224] construct an image classification model based on the image feature extraction module;

[0225] The image classification model comprises a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in turn, the four feature extraction modules are obtained by stacking the image feature extraction module with a preset scale and channel number in four feature scales; the preprocessing module is used for preprocessing the input target image and feeding the preprocessed image features to the four image feature extraction modules, the four image feature extraction modules perform dimension reduction processing on the preprocessed image features, and output classification results through the pooling layer and the fully connected layer.

[0226] In some embodiments, the image processing device further comprises a self-supervised training module configured to:

[0227] construct a self-supervised learning framework and an image dataset;

[0228] The image classification model is trained by using the self-supervised learning framework and an image dataset, and the trained image classification model is used to realize image classification. When training the image classification model, a full connection layer for classification is replaced by a projection head. The image classification model takes a sample image of the image dataset as input, and maps the output sample image feature to the self-supervised learning framework through the projection head for weight updating. The image classification model is configured with the updated weights to obtain the trained image classification model.

[0229] In some embodiments, the self-supervised learning framework generates a first view and a second view by performing two different data augmentation operations in response to an input sample image feature, and encodes the first view and the second view through a query encoder and a momentum encoder.

[0230] The momentum encoder is dynamically updated through the weights of the query encoder, and the update formula is:

[0231] f k1 = m x f k + (1-m) x f q

[0232] In the formula, f k1 is the updated parameter of the momentum encoder, f k is the parameter of the momentum encoder before updating, f q is the current parameter of the query encoder, and m is a momentum coefficient, which is 0 to 1.

[0233] The weights of the query encoder are updated by using a contrastive loss function, and the contrastive loss function is expressed as:

[0234]

[0235] In the formula, L q is the contrastive loss, k + is a positive sample key vector corresponding to q, k - is a negative sample key vector set, τ is a temperature parameter for controlling the sensitivity of similarity calculation, q·k + is the dot product between the query vector q and its corresponding positive sample key vector k + , exp is an exponential function for converting the dot product into a positive number to amplify the difference.

[0236] In some embodiments, the self-supervised training module is further configured to:

[0237] When training the image classification model, a multi-classification task loss function and a KL divergence are used to guide the gradient descent process of the self-supervised learning framework.

[0238] The multi-classification task loss function is represented as:

[0239]

[0240] In the formula, L CE is a multi-classification task loss function value, C is the total number of categories, y i,c is a one-hot encoding representation of whether sample i belongs to category c, p i,c is the probability that the model predicts that sample i belongs to category c, and N is the total number of samples.

[0241] The KL divergence is represented as:

[0242]

[0243] In the formula, a is a vector parameter of the Dirichlet distribution, Γ represents the gamma function, S α and respectively represent the sum of parameters a and a0, ψ is the logarithmic derivative of the gamma function, K is the number of categories, a0 is the parameter of the prior Dirichlet distribution, and represents the initial classification probability distribution; in the Bayesian framework, the prior distribution expresses the belief when no data is seen; θ represents a sample drawn from the Dirichlet distribution, that is, a specific category probability value; a k represents the parameter of the kth category in the Dirichlet distribution, which is used to describe the weight of the kth category; p represents the distribution p(θ|a), that is, the Dirichlet distribution with parameter a, which is used to describe the posterior distribution of a certain category θ.

[0244] The image processing apparatus provided by the embodiments of the present application can be used to execute the technical solutions of the image processing method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0245] Figure 10 The structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 1. Figure 10 As shown in the figure, the electronic device can include a processor 101 and a memory 102, wherein the processor 101 and the memory 102 can communicate; for example, the processor 101 and the memory 102 communicate through a communication bus 103.

[0246] The processor 101 executes computer-executed instructions stored in the memory 102, so that the processor 101 executes the solutions in the above-described embodiments. The processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; and can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0247] The communication bus 103 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The transceiver is used to realize the communication between the database access device and other computers (for example, a client, a read-write library and a read-only library). The memory can include a random access memory (RAM), and can also include a non-volatile memory.

[0248] The electronic device provided by the embodiments of the present application can be the terminal device of the above-described embodiments.

[0249] The embodiments of the present application further provide a computer-readable storage medium, which stores computer instructions, and when the computer instructions run on a computer, the computer executes the technical solutions of the image processing method of the above-described embodiments.

[0250] The embodiments of the present application further provide a computer program product, which includes a computer program stored in a computer-readable storage medium, and at least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, the technical solutions of the image processing method in the above-described embodiments can be implemented.

[0251] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. For example, the embodiments of the device described above are merely schematic. For example, the division of the modules is merely logical function division. There can be another division manner for the actual implementation. For example, a plurality of modules or a function can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0252] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to implement the embodiments of the present application.

[0253] In addition, each function module in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present separately, or two or more modules can be integrated in one unit. The unit formed by the above modules can be realized in the form of hardware, or in the form of hardware plus software function unit.

[0254] The integrated module realized in the form of software function module can be stored in a computer readable storage medium. The software function module stored in the storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method of each embodiment of the present application.

[0255] It should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor. The steps of the method disclosed in the present application can be directly embodied as a hardware processor to execute, or be executed by a combination of hardware and software modules in the processor.

[0256] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0257] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0258] The storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0259] An exemplary storage medium is coupled to the processor so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be part of the processor. The processor and the storage medium can be located in an application specific integrated circuits (ASIC). Of course, the processor and the storage medium can exist as discrete components in an electronic control unit or a host device.

[0260] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by relevant hardware instructed by programs. The foregoing programs can be stored in a computer readable storage medium. When the programs are executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes ROM, RAM, magnetic or optical disk, and various media that can store program codes.

[0261] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a target image; performing feature extraction on the target image based on an image feature extraction module to obtain target image features; the image feature extraction module comprises a sampling unit, a window maintaining unit and a grid maintaining unit connected in sequence; the sampling unit performs sampling on the input target image to obtain first image features; the window maintaining unit takes the first image features as input, divides the first image features into a plurality of non-overlapping blocks, and applies a preserved self-attention mechanism to the pixels in each non-overlapping block before performing a multi-layer perception operation to obtain second image features; the grid maintaining unit takes the second image features as input, divides the second image features into a plurality of grids of the same size, and applies a preserved self-attention mechanism to each grid before performing a multi-layer perception operation to obtain target image features; the calculation process of the preserved self-attention mechanism is as follows: O H ,K H =(Q,K) B,L,C→B,W,H,C O W ,K W = (Q, K) B,L,C→B,H,W,C ReSA dec (X) = Attn H (Attn W V) where O H represents the output height direction feature map, K H represents the height direction feature of the key matrix; Q represents the query matrix, which is used to calculate the attention weight; K represents the key matrix, which is used to calculate the similarity with the query matrix; B, L, and C represent batch size, sequence length, and channel number, respectively; O W represents the output width direction feature map; K W represents the width direction feature of the key matrix; Attn H represents the attention weight in the height direction; Softmax represents the operation for normalization; represents the transpose of the key matrix; ⊙ represents element multiplication; D H represents the distance decay factor in the height dimension; Attn W represents the attention weight in the width direction; represents the transpose of the key matrix in the width direction; D W represents the distance decay factor in the width dimension; represents the distance decay weight between two pixel points in the height direction; γ represents the distance decay factor, which is used to adjust the influence of the pixel points far and near; y m represents the coordinates of two pixel points in the vertical direction; represents the distance decay weight between two pixel points in the width direction; X m represents the coordinates of two pixel points in the horizontal direction; ReSA dec (X) represents the decoding output of the residual self-attention mechanism; X represents the input image or feature map; Attn W V represents applying the attention weight in the width direction to the value matrix V.

2. The method of claim 1, wherein, the sampling unit performs sampling on the input target image by the following formula to obtain first image features: in the formula, x represents the input target image or feature map; x' represents the first image features, which are generated by the processing process; Proj represents a projection layer, which is a 1x1 convolution operation for mapping or compressing the number of channels of the features; SE represents a squeeze-excitation mechanism; DWConv represents a depthwise separable convolution, which decomposes a standard convolution into a depthwise convolution and a pointwise convolution, reduces the computational complexity, and retains good feature extraction capability; Conv represents a convolution operation, which refers to a standard 2D convolution operation for extracting local features from an input image or feature map; Norm represents a normalization operation, which is batch normalization for stabilizing the training process and accelerating convergence; undownsample represents no downsampling, i.e., maintaining the original resolution of the image, in which case the input image or feature map is directly projected after the squeeze-excitation and convolution operations without changing its spatial dimension; downsample represents downsampling, i.e., reducing the spatial resolution of the image or feature map by a pooling operation or a convolution operation, in which case the input image is first reduced in resolution by a pooling operation, and then processed by the SE module and the DWConv convolution; the pooling operation includes maximum pooling or average pooling; Pool2D represents a 2D pooling operation for downsampling the image or feature map to reduce the spatial resolution, usually using maximum pooling or average pooling.

3. The method of claim 1, wherein, The method further comprises: constructing an image classification model based on the image feature extraction module; the image classification model comprises a preprocessing module, four image feature extraction modules, a pooling layer and a fully connected layer connected in sequence; the four image feature extraction modules are obtained by stacking the image feature extraction module at four feature scales with a preset scale and channel number; the preprocessing module is used to preprocess the input target image and feed the preprocessed image features to the four image feature extraction modules; the four image feature extraction modules perform dimension reduction processing on the preprocessed image features, and output classification results through the pooling layer and the fully connected layer.

4. The method of claim 3, wherein, After constructing the image classification model, the method further comprises: constructing a self-supervised learning framework and an image dataset; training the image classification model using the self-supervised learning framework and the image dataset, so that the image classification model is trained to realize image classification; when training the image classification model, a fully connected layer for classification is replaced by a projection head, the image classification model takes a sample image of the image dataset as input, and maps the output sample image feature to the self-supervised learning framework through the projection head for weight updating, and the image classification model is configured with the updated weights to obtain the trained image classification model.

5. The method of claim 4, wherein, The self-supervised learning framework generates a first view and a second view by performing two different data augmentation operations in response to an input sample image feature, and encodes the first view and the second view through a query encoder and a momentum encoder; The momentum encoder is dynamically updated through the weights of the query encoder, and the update formula is: f k1 = m x f k + (1 - m) x f q where f k1 is the updated parameter of the momentum encoder, f k is the parameter of the momentum encoder before updating, f q is the current parameter of the query encoder, and m is the momentum coefficient, which is valued between 0 and 1. The weights of the query encoder are updated by gradient using a contrastive loss function, and the contrastive loss function is expressed as: where L q is the contrastive loss, k + is the positive sample key vector corresponding to q, k - is the set of negative sample key vectors, τ is a temperature parameter to control the sensitivity of the similarity calculation, q·k + is the dot product between the query vector q and its corresponding positive sample key vector k + is the dot product between the query vector q and its corresponding positive sample key vector k, exp is the exponential function to convert the dot product into a positive number to amplify the difference.

6. The method of claim 5, wherein, When training the image classification model, a multi-classification task loss function and a KL divergence are used to guide the gradient descent process of the self-supervised learning framework; The multi-classification task loss function is expressed as: where L CE is the multi-class task loss function value, C is the total number of classes, y i,c is the one-hot encoded representation of whether sample i belongs to class c, p i,c is the probability that the model predicts that sample i belongs to class c, and N is the total number of samples. The KL divergence is expressed as: where a is a vector parameter of Dirichlet distribution, Γ represents gamma function, S α and S α0 respectively represent the sum of parameters a and a0, ψ is the logarithmic derivative of gamma function, K is the number of categories, a0 is the parameter of prior Dirichlet distribution, representing the initial classification probability distribution; in the Bayesian framework, the prior distribution expresses the belief when no data is seen; θ represents a sample drawn from Dirichlet distribution, i.e. a specific category probability value; a k represents the parameter of the kth category in Dirichlet distribution, used to describe the weight of the kth category; p represents the distribution p(θ|a), i.e. Dirichlet distribution with parameter a, used to describe the posterior distribution of a certain category θ.

7. An image processing apparatus characterized by comprising: comprising: an image acquisition module configured to acquire a target image; a feature extraction module configured to perform feature extraction on the target image based on an image feature extraction module to obtain a target image feature; The image feature extraction module comprises a sampling unit, a window retention unit and a grid retention unit connected in sequence; the sampling unit samples an input target image to obtain a first image feature, the window retention unit takes the first image feature as input, divides the first image feature into a plurality of non-overlapping blocks, and applies a reserved self-attention mechanism to the pixel points in each non-overlapping block to obtain a second image feature after multi-layer perceptron operation; the grid retention unit takes the second image feature as input, divides the second image feature into a plurality of grids of the same size, and applies a reserved self-attention mechanism to each grid to obtain a target image feature after multi-layer perceptron operation; The calculation process of the reserved self-attention mechanism is: O H ,K H =(Q,K) B,L,C→B,W,H,C O W ,K W =(Q,K) B,L,C→B,H,W,C ReSA dec (X) = Attn H (Attn W V) where O H represents the output height direction feature map, K H represents the height direction feature of the key matrix; Q represents the query matrix, which is used to calculate the attention weight; K represents the key matrix, which is used to calculate the similarity with the query matrix; B, L, and C represent batch size, sequence length, and channel number, respectively; O W represents the output width direction feature map; K W represents the width direction feature of the key matrix; Attn H represents the attention weight in the height direction; Softmax represents the operation for normalization; represents the transpose of the key matrix; ⊙ represents element multiplication; D H represents the distance decay factor in the height dimension; Attn W represents the attention weight in the width direction; represents the transpose of the key matrix in the width direction; D W represents the distance decay factor in the width dimension; represents the distance decay weight between two pixel points in the height direction; γ represents the distance decay factor, which is used to adjust the influence of the pixel points far and near; y m represents the coordinates of two pixel points in the vertical direction; represents the distance decay weight between two pixel points in the width direction; X m represents the coordinates of two pixel points in the horizontal direction; ReSA dec (X) represents the decoding output of the residual self-attention mechanism; X represents the input image or feature map; Attn W V represents applying the attention weight in the width direction to the value matrix V.

8. An electronic device, comprising: comprising: a processor, and a memory in communication connection with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to realize the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize the method of any one of claims 1-6.

10. A computer program product, characterised in that, The computer program is executed by the processor to realize the method of any one of claims 1-6.

Images