Image block feature extraction method and device and storage medium

Abstract

This application provides an image block feature extraction method, apparatus, and storage medium. The method includes: inputting an image block to be processed into a graph attention network model to obtain feature vectors for the image block; wherein the graph attention network model is trained based on the following steps: concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain a concatenated image block; determining the topological features of the sample image block using a graph attention mechanism based on the concatenated image block; and training the graph attention network model based on the topological features. The image block feature extraction method, apparatus, and storage medium provided in this application, through the trained graph attention network model, can acquire local and global feature information of the image block to be processed, improving the accuracy of feature information extraction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus and storage medium for extracting image block features. Background Technology

[0002] With the rapid development of artificial intelligence, deep learning algorithms are frequently applied to image processing methods. Represented by convolutional neural networks, these algorithms train models using large amounts of labeled data to extract deep feature information from images in an end-to-end manner, and their applications in image classification, semantic segmentation, and other fields are already mature.

[0003] Processing images with ultra-high resolution and massive amounts of data is even more complex. For example, processing whole-slide images (WSI) involves converting pathological slides into high-magnification, large-scale digital images suitable for computer display, transmission, and processing using a specialized scanning imaging system. This requires first dividing the image into appropriately sized blocks, and then using deep learning for feature extraction. However, dividing these images into multiple blocks increases the difficulty of extracting useful information, and this feature extraction method can lead to inaccurate feature information. Summary of the Invention

[0004] This application provides an image block feature extraction method, apparatus, and storage medium to solve the technical problem of inaccurate feature information extraction of image blocks in the prior art.

[0005] In a first aspect, embodiments of this application provide an image block feature extraction method, including:

[0006] Input the image blocks to be processed into the graph attention network model to obtain the feature vectors of the image blocks to be processed;

[0007] The graph attention network model is obtained by training based on the following steps:

[0008] The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block;

[0009] The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks;

[0010] A graph attention network model is trained based on the aforementioned topological features.

[0011] In some embodiments, before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the method further includes:

[0012] The sample image is divided into blocks based on a semantic segmentation model to obtain an initial segmentation result;

[0013] Based on the initial segmentation results, a channel attention mechanism is used to determine the semantic segmentation map of the sample image blocks.

[0014] In some embodiments, the loss function expression of the semantic segmentation model is as follows:

[0015]

[0016] Where x and y represent the coordinates of the pixel; c represents the category of the pixel; Y i Represents the actual value of a pixel; Represents the predicted value of a pixel; |D x,y | represents the distance of a pixel from the boundary; ρ represents a constant value used to prevent the denominator from being 0; L represents the loss function value.

[0017] In some embodiments, determining the topological features of the sample image blocks using a graph attention network based on the stitched image blocks includes:

[0018] The stitched image is divided into blocks and input into a residual neural network for compression to obtain node features;

[0019] The weights of the connections between nodes are determined through an attention mechanism;

[0020] The topological features of the sample image blocks are determined based on the node features and the weights of the connections between the nodes.

[0021] In some embodiments, training a graph attention network model based on the topological features includes:

[0022] The feature vector of the sample image block is determined based on the topological features of the sample image block;

[0023] A graph attention network model is trained based on the feature vectors of the sample image blocks.

[0024] In some embodiments, determining the feature vector of a sample image block based on the topological features of the image block includes:

[0025] Assign weights to adjacent nodes for each node; the adjacent nodes refer to the nodes that are next to the node.

[0026] The feature vector of the sample image block is updated based on the topological features and the weights of the adjacent nodes.

[0027] In some embodiments, before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the method further includes:

[0028] The thumbnail of the full slice image is obtained using the Otsu method (OTSU) to obtain the mask of the thumbnail;

[0029] The initial image blocks of the full slice image are determined based on the compressed full slice image;

[0030] Data augmentation is performed on the initial image blocks based on the mask of the thumbnail to obtain the sample image blocks.

[0031] Secondly, embodiments of this application provide an image block feature extraction apparatus, comprising:

[0032] The feature extraction module is used to input the image blocks to be processed into the graph attention network model to obtain the feature vectors of the image blocks to be processed;

[0033] The graph attention network model is obtained by training based on the following steps:

[0034] The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block;

[0035] The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks;

[0036] A graph attention network model is trained based on the aforementioned topological features.

[0037] Thirdly, embodiments of this application provide an electronic device, including a memory, a transceiver, and a processor;

[0038] The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the processor is used to read the computer programs in the memory and execute the image block feature extraction method as described in the first aspect above.

[0039] Fourthly, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to execute the image block feature extraction method described in the first aspect above.

[0040] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program for causing a computer to perform the image block feature extraction method described in the first aspect above.

[0041] In a sixth aspect, embodiments of this application also provide a communication device readable storage medium storing a computer program for causing the communication device to perform the image block feature extraction method described in the first aspect above.

[0042] In a seventh aspect, embodiments of this application also provide a chip product readable storage medium storing a computer program for causing the chip product to perform the image block feature extraction method described in the first aspect above.

[0043] The image block feature extraction method, apparatus, and storage medium provided in this application embodiment stitch together the original image of the sample image block and the semantic segmentation map of the sample image block, construct the stitched image block, use a graph attention network to determine the topological features of the sample image block, and train a graph attention network model based on the topological features. This achieves effective learning of local features in the sample image block and global features between sample image blocks. The feature information of the image block to be processed is obtained through this model, which improves the accuracy of image block feature information extraction. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the image block feature extraction method provided in an embodiment of this application;

[0046] Figure 2 This is a semantic segmentation model structure diagram of an example scenario of an image block feature extraction method provided in this application embodiment;

[0047] Figure 3 This is a schematic diagram of the structure of an image block feature extraction method apparatus provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Figure 1 This is a flowchart illustrating the image block feature extraction method provided in this application embodiment, as shown below. Figure 1 As shown in the figure, this application provides an image block feature extraction method. The method includes:

[0051] Step 101: Input the image blocks to be processed into the graph attention network model to obtain the feature vectors of the image blocks to be processed;

[0052] The graph attention network model is obtained by training based on the following steps:

[0053] The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block;

[0054] The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks;

[0055] A graph attention network model is trained based on the aforementioned topological features.

[0056] Specifically, a graph attention network model is first constructed to obtain the feature vectors of image blocks. The graph attention network model is trained using sample image blocks. Then, the image block to be processed is input into the graph attention network model, and the feature vectors of the image block to be processed are output.

[0057] The training of a graph attention network model includes the following steps:

[0058] First, the selected full-slice image is divided into image blocks as sample image blocks. The semantic segmentation map of the sample image block is then concatenated or fused with the original image of that sample image block. The full-slice image can be obtained using the medical processing library OpenSlide. The semantic segmentation map of the sample image block can be obtained using the Segformer segmentation model.

[0059] For example, a graph neural network is used to construct a graph of sample image blocks, where each sample image block is a node and the connections between sample image blocks are edges. The semantic segmentation graph of a sample image block is then concatenated with the original graph of that sample image block to obtain a concatenated image block. Node features can be determined based on this concatenated image block.

[0060] Then, based on a graph neural network, a graph is constructed using sample image blocks as nodes and the connections between sample image blocks as edges. After obtaining the stitched image blocks, node features are generated based on these blocks. An attention mechanism is then used to determine the weights of the connections between nodes. Based on these weights and the node features generated from the stitched image blocks, the topological relationships between nodes or image blocks are determined, thus obtaining global topological features. Finally, a graph attention network model is trained based on the obtained topological features.

[0061] For example, the stitched image is divided into blocks and input into a Residual Neural Network (ResNet) for compression, compressing each block into a 128-dimensional vector, which is then used as node features. A graph attention network is then used to assign weights to the connections between nodes, and these weights are used as the features of the corresponding edges. Next, the topological relationships between nodes are learned to obtain global topological features. Finally, the constructed graph attention network model is trained based on the obtained topological features of the sample image blocks.

[0062] The image block feature extraction method provided in this application determines node features based on the semantic segmentation map of the sample image block and the stitched image block obtained by stitching the original image of the sample image block. It then determines edge features based on the connection relationship between the sample image blocks using an attention mechanism. Based on the node features and edge features, it learns the topological relationship between the sample image blocks / nodes to obtain global topological features. Finally, it trains a graph attention network model based on these topological features, enabling the model to obtain local and global feature information of the image block to be processed, thereby improving the accuracy of feature information extraction.

[0063] In some embodiments, before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the method further includes:

[0064] The sample image is divided into blocks based on a semantic segmentation model to obtain an initial segmentation result;

[0065] Based on the initial segmentation results, a channel attention mechanism is used to determine the semantic segmentation map of the sample image blocks.

[0066] The image block feature extraction method provided in this application embodiment...

[0067] Specifically, the Segformer segmentation model can be used to perform semantic segmentation on the sample image blocks to obtain initial segmentation results. A channel attention mechanism is then applied to the initial segmentation results, assigning different weights to different channels. Based on these weights, low-level detail features are fused with high-level semantic features to output the final semantic segmentation map of the sample image blocks.

[0068] For example, sample images are divided into blocks and input into the Segformer segmentation model. These blocks are then processed by an encoder to obtain multi-scale features, followed by a decoder that performs upsampling based on a Multilayer Perceptron (MLP). The outputs are then concatenated to obtain the initial segmentation result. A channel attention module is added to the Segformer model. The initial segmentation result is input into this module, and different weights are assigned to different channels. This allows the segmentation model to autonomously learn how to fuse low-level detail information and high-level semantic information based on these weights. Channel compression is then performed to obtain the final semantic segmentation map of the sample image blocks.

[0069] The image block feature extraction method provided in this application effectively integrates low-level detail information and high-level semantic information by using a channel attention mechanism during semantic segmentation, resulting in more accurate semantic segmentation results for image blocks and thus improving the accuracy of image block feature extraction.

[0070] In some embodiments, the loss function expression of the semantic segmentation model is as follows:

[0071]

[0072] Where x and y represent the coordinates of the pixel; c represents the category of the pixel; Y i Represents the actual value of a pixel; Represents the predicted value of a pixel; |D x,y | represents the distance of a pixel from the boundary; ρ represents a constant value used to prevent the denominator from being 0; L represents the loss function value.

[0073] Specifically, due to the inaccuracy in segmenting organizational boundaries, a boundary-aware loss function needs to be designed for the segmentation model. For example, a boundary-aware loss function can be designed based on cross-entropy loss.

[0074] For example, based on the ground truth, if a pixel near the boundary is misclassified, the distance between that pixel and the boundary is calculated. The closer the distance, the greater the penalty for loss.

[0075] The image block feature extraction method provided in this application embodiment, by designing a boundary-aware loss function to penalize erroneously segmented pixels near the boundary, achieves boundary awareness, effectively improves the problem of inaccurate boundary segmentation, obtains a more accurate semantic segmentation map, and thus improves the accuracy of image block feature extraction.

[0076] In some embodiments, determining the topological features of the sample image blocks using a graph attention network based on the stitched image blocks includes:

[0077] The stitched image is divided into blocks and input into a residual neural network for compression to obtain node features;

[0078] The weights of the connections between nodes are determined through an attention mechanism;

[0079] The topological features of the sample image blocks are determined based on the node features and the weights of the connections between the nodes.

[0080] Specifically, the stitched image blocks are input into a residual neural network (ResNet) for compression to generate node features. Then, the weights of the connection relationships between sample image blocks / nodes are determined based on the attention mechanism. Based on the node features and the determined weights, the topological features of the sample image blocks are determined.

[0081] For example, ResNet is used to compress the stitched image into 128-dimensional vectors and use them as node features; an attention mechanism is used to automatically assign weights to the connection relationships between nodes to generate edge features; based on the node features and edge weights, the topological relationships between nodes are learned and global topological features are extracted.

[0082] The image block feature extraction method provided in this application embodiment obtains node features by compressing and stitching image blocks through a residual neural network, determines edge weights using an attention mechanism, and determines global topological features based on node features and edge weights. This method simultaneously considers the local and global features of the sample image blocks and trains a graph attention network model based on these features, resulting in higher accuracy in image block feature extraction.

[0083] In some embodiments, training a graph attention network model based on the topological features includes:

[0084] The feature vector of the sample image block is determined based on the topological features of the sample image block;

[0085] A graph attention network model is trained based on the feature vectors of the sample image blocks.

[0086] Specifically, after obtaining the topological features of the sample image blocks, the feature vectors of the nodes are determined based on the topological features, and the graph attention network model is trained using the obtained feature vectors of the sample image blocks.

[0087] For example, node features are determined based on the topological features of sample image blocks / nodes, and the feature vector of the node is obtained based on the node features. The constructed graph attention network model is trained based on the obtained node feature vector, and then the image block to be processed is input into the trained graph attention network model, and the feature vector of the image block to be processed is output.

[0088] The image block feature extraction method provided in this application uses a graph attention network model trained based on the topological features of image blocks, which takes into account the global features of image blocks, making the image block feature extraction results more accurate.

[0089] In some embodiments, determining the feature vector of a sample image block based on the topological features of the image block includes:

[0090] Assign weights to adjacent nodes for each node; the adjacent nodes refer to the nodes that are next to the node.

[0091] The feature vector of the sample image block is updated based on the topological features and the weights of the adjacent nodes.

[0092] Specifically, after determining the topological features of the sample image blocks, feature combination or fusion is performed to update the node features. Each node also obtains the weights of its neighboring nodes, and the node features are updated based on the topological features and the weights of its neighboring nodes, thereby obtaining the updated feature vector of the node.

[0093] For example, a node obtains the weights of its neighboring nodes, and generates new features based on these weights, combined with topological features and its own node features, as new node features.

[0094] For example, a multi-head attention coefficient can be set or calculated to fuse the weight information of neighboring nodes, topological features, and its own node features to generate new node features and update the node feature vector.

[0095] The image block feature extraction method provided in this application update the node feature vector based on the weights of adjacent nodes, the topological features of sample image blocks, and node features. This constructs a more accurate and information-rich graph attention network model, thereby improving the accuracy of image block feature extraction based on this graph attention network model.

[0096] In some embodiments, before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the method further includes:

[0097] The thumbnail of the full slice image is obtained using the Otsu method (OTSU) to obtain the mask of the thumbnail;

[0098] The initial image blocks of the full slice image are determined based on the compressed full slice image;

[0099] Data augmentation is performed on the initial image blocks based on the mask of the thumbnail to obtain the sample image blocks.

[0100] Specifically, some preprocessing is required on the sample image blocks to obtain the final required sample image blocks.

[0101] The whole-slice pathological image can be obtained by reading the medical processing library OpenSlide. This whole-slice pathological image is stored in a multi-resolution pyramid structure, and the thumbnail of the whole-slice pathological image is the image in the pyramid structure. For example, reading the image of the 7th layer yields the thumbnail of the whole-slice pathological image.

[0102] Specifically, the thumbnail is filtered for background using the inter-class variance method (OTSU) to obtain a thumbnail mask.

[0103] For example, by iterating through the grayscale values ​​of the thumbnail using the OTSU algorithm, selecting the grayscale value corresponding to the largest inter-class variance as the segmentation threshold, the thumbnail is segmented into foreground and background to obtain a thumbnail mask.

[0104] Specifically, the whole-section pathological image is compressed to a size consistent with the mask size, and then the compressed whole-section pathological image is cut to obtain the initial image blocks of the whole-section pathological image.

[0105] For example, OpenSlide reads the 7th layer image to obtain a thumbnail of the whole-section pathological image, reads the 0th layer image to obtain the original whole-section pathological image, compresses the image to one-sixtieth of its original size, and then cuts the compressed image into non-overlapping image blocks of size 512×512.

[0106] Optionally, after cutting the compressed whole-slice pathological image to obtain multiple image blocks, it is also necessary to remove image blocks that do not contain the foreground, and the remaining image blocks are the initial image blocks of the whole-slice pathological image.

[0107] Specifically, after obtaining the thumbnail mask and initial image blocks, data augmentation is performed based on the thumbnail mask and initial image blocks, and the resulting image blocks are used as sample image blocks. The data augmentation methods include inversion, rotation, cropping, scaling, translation transformation, random brightness enhancement or reduction, etc.

[0108] For example, data augmentation can be achieved by simultaneously inverting, rotating, cropping, scaling, translating, and randomly enhancing or reducing the brightness of the thumbnail mask and the initial image blocks, thereby further expanding the data and obtaining the final sample image blocks needed to train the graph attention network model.

[0109] The image block feature extraction method provided in this application preprocesses the image blocks by filtering and data augmentation to obtain the final sample image blocks, which strengthens the important image features of the image blocks, reduces the influence of irrelevant information on the training of the graph attention network model, and improves the accuracy of image block feature extraction.

[0110] The image block feature extraction methods provided in the above embodiments are further illustrated below with specific examples:

[0111] Step 1: Use the medical processing library OpenSlide to read the full scan pathological images of the liver tissue removed from the patient.

[0112] Specifically, because the whole-slice pathology images are too large and stored in a multi-resolution pyramid structure, a thumbnail of the whole-slice pathology image is obtained by reading the 7th layer image when using OpenSlide.

[0113] Step 2: Filter the blank background of the thumbnail using the OTSU method.

[0114] Specifically, the OTSU algorithm iterates through the gray values ​​of the entire image, selects the gray value corresponding to the largest inter-class variance as the threshold, and performs foreground and background segmentation on the thumbnail to obtain the thumbnail mask.

[0115] Step 3: Scale the whole-section pathology image to one-sixtieth of the original image and perform tile processing, cutting the scaled whole-section pathology image into non-overlapping image tiles of size 512×512.

[0116] Specifically, OpenSlide is used to read the image at layer 0, which is the original image of the whole-section pathology image, and the image is compressed to one-sixtieth of the original image. The get_tile function is used to cut it into non-overlapping tiles of size 512×512, and tiles that do not contain the foreground are removed to obtain the initial image blocks of the whole-section pathology image.

[0117] Step 4: Perform data augmentation on the initial image blocks.

[0118] Specifically, by simultaneously applying data augmentation methods such as inversion, rotation, cropping, scaling, translation, and random brightness enhancement or reduction to the initial image blocks and thumbnail masks, the data is further expanded to obtain the final sample image blocks required for training the model.

[0119] Step 5: Figure 2 This is a semantic segmentation model structure diagram of an example scenario of an image block feature extraction method provided in this application embodiment, such as... Figure 2 As shown, the sample images obtained after data augmentation are divided into blocks and input into the improved Segformer segmentation model for semantic segmentation to obtain tile-level segmentation results. The specific steps include the following:

[0120] Step 5.1: Select the lightweight hybrid transformer encoder MiT-B0, which has fewer parameters and higher performance. Input the tile obtained in Step 4 into the Segformer encoder to obtain multi-scale features.

[0121] Segformer encoding first undergoes an Overlap Patch Embedding module, followed by downsampling through four Transformer modules to obtain feature maps at different scales. Each Transformer module includes an efficient self-attention module, a Mix-FFN (Mix-FFN) forward pass module, and an Overlap Patch Embedding module. The feature map size obtained from each downsampling stage is [size missing].

[0122]

[0123] Where H represents the length of the feature map, W represents the width of the feature map, and C i+1 Represents the feature map channel.

[0124] Step 5.2: In the decoder section, modify the MLP upsampling module. For example... Figure 2 As shown, the upsampled input features at each stage include downsampled features of the same scale from the encoder and downsampled feature maps of a smaller scale. The smaller-sized features are upsampled and then concatenated with the features of the same scale, and then upsampled together to the original image size by the MLP module.

[0125] The feature maps obtained from each stage, which are the same size as the original image, are concatenated and then input into a channel attention module. Different weights are assigned to different channels, so that the model can learn how to fuse low-level detail information and high-level semantic information autonomously during the process of fusing low-level detail information and high-level semantic information. Then, channel compression is performed.

[0126] Step 5.3: To address the issue of inaccurate organizational boundary segmentation, a boundary-aware loss function is designed based on cross-entropy loss. Based on the ground truth, if a pixel near the boundary is misclassified, its distance to the boundary is calculated. The closer the pixel, the greater the penalty. Therefore, the loss function formula is designed as follows:

[0127]

[0128] Where x and y represent the coordinates of the pixel; c represents the category of the pixel; Y i Represents the actual value of a pixel; Represents the predicted value of a pixel; |D x,y | represents the distance of a pixel from the boundary; ρ represents a constant value used to prevent the denominator from being 0; L represents the loss function value.

[0129] Step 6: Concatenate the semantic segmentation map of each sample image block with the original image of that sample image block, and generate node features using a Residual Neural Network (ResNet). Use a graph attention network to automatically assign weights to neighboring nodes for each node, and train the graph attention network model by extracting the topological features between sample image blocks. Specifically, this includes the following steps:

[0130] Step 6.1: Graph Attention Network Model Construction: The semantic segmentation map of the sample image block is concatenated with the original image to obtain the concatenated image block. Then, it is compressed into a 128-dimensional vector through a ResNet network as the node feature. The 8 sample image blocks adjacent to this sample image block are taken as neighboring nodes.

[0131] The node feature set is represented as N represents the number of nodes, and F represents the number of features per node. Let i represent the node characteristics of the i-th node (referred to as node i), where i ranges from 1 to N.

[0132] The updated node feature set is represented as F′ represents the number of features for each node. The value of F′ may differ from F. This represents the updated node characteristics of node i, where i ranges from 1 to N.

[0133] Step 6.2: Edge weight matrix construction: For the generation of weights for connections between different nodes, the graph convolutional network uses an attention mechanism to automatically assign weights between nodes, learn the topological relationships between nodes, and obtain global features.

[0134] Calculate the similarity coefficient between node i and its neighboring node j. This is used to represent the importance of the features of node j to node i. A linear mapping with shared parameters W increases the dimensionality of the node features, and function a is used to map the concatenated high-dimensional features to a real number.

[0135] The selection of all nodes j is standardized using the normalized exponential function (softmax function), and the standardized result is expressed as:

[0136]

[0137] Among them, e ij N is the similarity coefficient between node i and its neighboring node j; i represents a node; LeakyReLu is the activation function; exp is the exponential function.

[0138] Step 6.3: Node Update: For the feature combination part, the node obtains the weight of each neighboring node, and calculates the high-level feature based on the weight of the neighboring nodes, the topological features of the node, and its own node features. The node features are then updated using the high-level feature.

[0139] Based on the calculated multi-head attention coefficient σ, the node features are updated to obtain the final feature vector of the sample image block. The updated node features are represented as follows: Where K represents the number of heads.

[0140] Step 6.4: Model Training: Train the graph attention network model based on the feature vectors of the obtained sample image blocks.

[0141] Step 7: Feature extraction: Input the image to be processed into blocks into the trained graph attention network model, and output the feature vector corresponding to the image block to be processed.

[0142] The image block feature extraction method provided in this application embodiment concatenates the original image of a sample image block with the semantic segmentation map of the sample image block, constructs a graph of the concatenated image block, uses a graph attention network to determine the topological features of the sample image block, and trains a graph attention network model based on the topological features. This achieves effective learning of local features in the sample image block and global features between sample image blocks. By obtaining feature information of the image block to be processed through this model, the accuracy of image block feature information extraction is improved.

[0143] Figure 3 This is a schematic diagram of the structure of an image block feature extraction device provided in an embodiment of this application, as shown below. Figure 3 As shown in the figure, this application embodiment provides an image block feature extraction device, including a feature extraction module 301.

[0144] The feature extraction module 301 is used to input the image block to be processed into the graph attention network model to obtain the feature vector of the image block to be processed.

[0145] The graph attention network model is obtained by training based on the following steps:

[0146] The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block;

[0147] The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks;

[0148] A graph attention network model is trained based on the aforementioned topological features.

[0149] In some embodiments, it also includes:

[0150] The semantic segmentation module is used to perform semantic segmentation on the sample image blocks based on the semantic segmentation model to obtain the initial segmentation result;

[0151] The first determining module is used to determine the semantic segmentation map of the sample image block based on the initial segmentation result using a channel attention mechanism.

[0152] In some embodiments, the loss function expression of the semantic segmentation model is as follows:

[0153]

[0154] Where x and y represent the coordinates of the pixel; c represents the category of the pixel; Y i Represents the actual value of a pixel; Represents the predicted value of a pixel; |D x,y| represents the distance of a pixel from the boundary; ρ represents a constant value used to prevent the denominator from being 0; L represents the loss function value.

[0155] In some embodiments, it also includes:

[0156] The compression module is used to input the stitched image into a residual neural network in blocks for compression to obtain node features;

[0157] The second determination module is used to determine the weights of the connection relationships between nodes through an attention mechanism;

[0158] The third determining module is used to determine the topological features of the sample image blocks based on the node features and the weights of the connection relationships between the nodes.

[0159] In some embodiments, the graph attention network model is also trained based on the following sub-modules:

[0160] The first determining submodule is used to determine the feature vector of the sample image block based on the topological features of the sample image block;

[0161] The training submodule is used to train a graph attention network model based on the feature vectors of the sample image blocks.

[0162] In some embodiments, the first determining submodule includes:

[0163] The allocation unit is used to assign weights to adjacent nodes to each node; the adjacent nodes refer to the nodes that are adjacent to the node.

[0164] The update unit is used to update the feature vector of the sample image block based on the topological features and the weights of the adjacent nodes.

[0165] In some embodiments, the graph attention network model is also trained based on the following sub-modules:

[0166] The acquisition submodule is used to obtain the mask of the thumbnail based on the thumbnail of the full slice image using the Otsu method (maximum inter-class variance).

[0167] The second determining submodule is used to determine the initial image blocks of the compressed full-slice image based on the compressed full-slice image;

[0168] The data augmentation submodule is used to perform data augmentation on the initial image blocks based on the mask of the thumbnail to obtain the sample image blocks.

[0169] Specifically, the image block feature extraction device provided in this application embodiment can implement all the method steps implemented in the above image block feature extraction method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0170] It should be noted that the division of units / modules in the above embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0171] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 4 As shown, the electronic device includes a memory 420, a transceiver 400, and a processor 410, wherein:

[0172] Memory 420 is used to store computer programs; transceiver 400 is used to send and receive data under the control of processor 410; processor 410 is used to read the computer program in memory 420 and execute the above-described image block feature extraction method, the method comprising:

[0173] Input the image blocks to be processed into the graph attention network model to obtain the feature vectors of the image blocks to be processed;

[0174] The graph attention network model is obtained by training based on the following steps:

[0175] The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block;

[0176] The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks;

[0177] A graph attention network model is trained based on the aforementioned topological features.

[0178] In some embodiments, before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the method further includes:

[0179] The sample image is divided into blocks based on a semantic segmentation model to obtain an initial segmentation result;

[0180] Based on the initial segmentation results, a channel attention mechanism is used to determine the semantic segmentation map of the sample image blocks.

[0181] In some embodiments, the loss function expression of the semantic segmentation model is as follows:

[0182]

[0183] Where x, y represent the coordinates of the pixel; c represents the category of the pixel; Y i Represents the actual value of a pixel; Represents the predicted value of a pixel; |D x,y | represents the distance of a pixel from the boundary; ρ represents a constant value used to prevent the denominator from being 0; L represents the loss function value.

[0184] In some embodiments, determining the topological features of the sample image blocks using a graph attention network based on the stitched image blocks includes:

[0185] The stitched image is divided into blocks and input into a residual neural network for compression to obtain node features;

[0186] The weights of the connections between nodes are determined through an attention mechanism;

[0187] The topological features of the sample image blocks are determined based on the node features and the weights of the connections between the nodes.

[0188] In some embodiments, training a graph attention network model based on the topological features includes:

[0189] The feature vector of the sample image block is determined based on the topological features of the sample image block;

[0190] A graph attention network model is trained based on the feature vectors of the sample image blocks.

[0191] In some embodiments, determining the feature vector of a sample image block based on the topological features of the image block includes:

[0192] Assign weights to adjacent nodes for each node; the adjacent nodes refer to the nodes that are next to the node.

[0193] The feature vector of the sample image block is updated based on the topological features and the weights of the adjacent nodes.

[0194] In some embodiments, before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the method further includes:

[0195] The thumbnail of the full slice image is obtained using the Otsu method (OTSU) to obtain the mask of the thumbnail;

[0196] The initial image blocks of the full slice image are determined based on the compressed full slice image;

[0197] Data augmentation is performed on the initial image blocks based on the mask of the thumbnail to obtain the sample image blocks.

[0198] Specifically, transceiver 400 is used to receive and send data under the control of processor 410.

[0199] Among them, Figure 4 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 410) and memory (memory 420). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 400 can be multiple elements, including transmitters and receivers, providing units for communicating with various other devices over transmission media, including wireless channels, wired channels, optical fibers, etc. The processor 410 is responsible for managing the bus architecture and general processing, and the memory 420 can store data used by the processor 410 during operation.

[0200] The processor 410 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] In some embodiments, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program for causing a computer to perform the image block feature extraction method provided in the above method embodiments.

[0203] Specifically, the computer-readable storage medium provided in the embodiments of this application can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0204] It should be noted that the computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0205] It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited. For example, the first object can be one or more.

[0206] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0207] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0208] In this application, "determining B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," "determining C based on A, and further determining B based on C," etc. It can also include using A as a condition for determining B, for example, "when A satisfies the first condition, B is determined using the first method"; or "when A satisfies the second condition, B is determined," or "when A satisfies the third condition, B is determined based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A satisfies the first condition, C is determined using the first method, and B is further determined based on C," etc.

[0209] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0210] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0211] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0212] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0213] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for extracting features from image blocks, characterized in that, include: Input the image blocks to be processed into the graph attention network model to obtain the feature vectors of the image blocks to be processed; The graph attention network model is obtained by training based on the following steps: The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block; The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks; A graph attention network model is trained based on the aforementioned topological features; The step of concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block includes: The sample image blocks are constructed based on graph neural networks to obtain the semantic segmentation graph of the sample image blocks, wherein the sample image blocks are nodes and the connection relationships between the sample image blocks are edges. The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block.

2. The image block feature extraction method according to claim 1, characterized in that, Before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the process also includes: The sample image is divided into blocks based on a semantic segmentation model to obtain an initial segmentation result; Based on the initial segmentation results, a channel attention mechanism is used to determine the semantic segmentation map of the sample image blocks.

3. The image block feature extraction method according to claim 2, characterized in that, The loss function expression of the semantic segmentation model is as follows: ； in, Represents the coordinates of a pixel; This represents the total number of pixel categories; Represents the actual value of a pixel; This represents the predicted value of a pixel. This represents the distance of a pixel from the boundary. This represents a constant value, used to prevent the denominator from being 0; This represents the value of the loss function.

4. The image block feature extraction method according to claim 1, characterized in that, Based on image patching, a graph attention network is used to determine the topological features of the sample image patches, including: The stitched image is divided into blocks and input into a residual neural network for compression to obtain node features; The weights of the connections between nodes are determined through an attention mechanism; The topological features of the sample image blocks are determined based on the node features and the weights of the connections between the nodes.

5. The image block feature extraction method according to claim 1, characterized in that, The graph attention network model trained based on the aforementioned topological features includes: The feature vector of the sample image block is determined based on the topological features of the sample image block; A graph attention network model is trained based on the feature vectors of the sample image blocks.

6. The image block feature extraction method according to claim 5, characterized in that, Determining the feature vector of a sample image block based on the topological features of the image block includes: Assign weights to adjacent nodes for each node; the adjacent nodes refer to the nodes that are next to the node. The feature vector of the sample image block is updated based on the topological features and the weights of the adjacent nodes.

7. The image block feature extraction method according to claim 1, characterized in that, Before concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block, the process also includes: The thumbnail of the full slice image is obtained using the Otsu method (OTSU) to obtain the mask of the thumbnail; The initial image blocks of the full slice image are determined based on the compressed full slice image; Data augmentation is performed on the initial image blocks based on the mask of the thumbnail to obtain the sample image blocks.

8. An image block feature extraction device, characterized in that, include: The feature extraction module is used to input the image blocks to be processed into the graph attention network model to obtain the feature vectors of the image blocks to be processed; The graph attention network model is obtained by training based on the following steps: The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block; The topological features of the sample image blocks are determined by using a graph attention mechanism based on the stitched image blocks; A graph attention network model is trained based on the aforementioned topological features; The step of concatenating the semantic segmentation map of the sample image block with the original image of the sample image block to obtain the concatenated image block includes: The sample image blocks are constructed based on graph neural networks to obtain the semantic segmentation graph of the sample image blocks, wherein the sample image blocks are nodes and the connection relationships between the sample image blocks are edges. The semantic segmentation map of the sample image block is stitched together with the original image of the sample image block to obtain the stitched image block.

9. An electronic device, characterized in that, Includes memory, transceiver, and processor; The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the processor is used to read the computer programs in the memory and execute the image block feature extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that causes a computer to perform the image block feature extraction method according to any one of claims 1 to 7.

Images