Image processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses an image processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a target image which comprises a target object and is divided into a plurality of image blocks; performing feature extraction processing on the target image to obtain image features of the target image; the image features comprise block features of each image block in a hidden space; performing classification prediction processing on the corresponding image block in the hidden space according to the block feature of each image block to obtain a block classification result of each image block; performing aggregation processing on the block classification result of each image block in the hidden space, and generating an image classification result of the target image according to an aggregation processing result; the aggregation processing in the hidden space is used for adjusting the contribution degree of the block classification results of different image blocks when the image classification result is generated, and the accuracy of the result obtained after classification prediction processing is carried out on the image can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the continuous development of computer technology, its application in production and daily life has become a current focus, such as image classification and prediction. Currently, image classification and prediction typically involves extracting image features and then performing classification and prediction based on those features. However, practical experience shows that current methods for image classification and prediction suffer from low accuracy. Summary of the Invention

[0003] This invention provides an image processing method, apparatus, computer device, and storage medium that can improve the accuracy of the results obtained after image classification and prediction processing.

[0004] On one hand, embodiments of the present invention provide an image processing method, including:

[0005] Acquire a target image, the target image containing a target object, and the target image is divided into multiple image blocks;

[0006] The target image is subjected to feature extraction processing to obtain the image features of the target image; the image features include the block features of each image patch in the latent space;

[0007] In the latent space, the corresponding image blocks are classified and predicted according to the block features of each image block to obtain the block classification result of each image block;

[0008] The block classification results of each image patch are aggregated in the latent space, and the image classification result of the target image is generated based on the aggregation result. The aggregation process in the latent space is used to adjust the contribution of the block classification results of different image patches in the generation of the image classification result. The image classification result is used to indicate the probability that the target object in the target image is an anomalous object.

[0009] In another aspect, embodiments of the present invention provide an image processing apparatus, comprising:

[0010] An acquisition unit is used to acquire a target image, the target image containing a target object, and the target image being divided into multiple image blocks;

[0011] The processing unit is used to perform feature extraction processing on the target image to obtain image features of the target image; the image features include block features of each image patch in the latent space;

[0012] The processing unit is further configured to perform classification prediction processing on the corresponding image block in the latent space according to the block features of each image block, so as to obtain the block classification result of each image block;

[0013] The processing unit is further configured to perform aggregation processing on the block classification results of each image block in the latent space, and generate an image classification result of the target image based on the aggregation processing result; wherein, the aggregation processing in the latent space is used to adjust the contribution of the block classification results of different image blocks in the generation of the image classification result, and the image classification result is used to indicate the probability that the target object in the target image is an abnormal object.

[0014] In one embodiment, when the processing unit performs feature extraction processing on the target image to obtain the image features of the target image, it is specifically used for:

[0015] Multiple image blocks obtained after dividing the target image are acquired, and the positional relationship between any two image blocks is determined based on the position of each image block in the target image.

[0016] Extract the initial block features of each image block and determine the relationship features corresponding to the positional relationship;

[0017] Based on the relational features, the initial block features of each image block are perceptually fused to obtain perceptual block features, and the obtained perceptual block features are used as the image features of the target image.

[0018] In one embodiment, the relational features corresponding to the positional relationships include an adjacency matrix, and the perceptual fusion processing is performed by calling a context-aware block feature extraction module, which includes a graph convolutional network and a rectified network. When the processing unit performs perceptual fusion processing on the initial block features of each image block based on the relational features to obtain perceptual block features, it is specifically used for:

[0019] The initial block features and the adjacency matrix of each image block are input into the first graph convolutional network in the context-aware block feature extraction module to obtain the first intermediate block features;

[0020] The first rectified network in the context-aware block feature extraction module is invoked to perform linear rectification on the first intermediate block features to obtain the second intermediate block features of each image block.

[0021] The second graph convolutional network and the second rectified network in the context-aware block feature extraction module are invoked to identify the second intermediate block features and the adjacency matrix of each image block, and the identified block features are used as the perceptual block features.

[0022] In one embodiment, the input features of the graph convolutional network include initial block features and intermediate block features of each image block. When the processing unit uses the graph convolutional network to perform recognition processing on the input features and the adjacency matrix, the graph convolutional network is specifically used for:

[0023] Based on the adjacency matrix and the input features, determine the sum of the input features corresponding to each image patch in the target image;

[0024] The degree matrix corresponding to the positional relationship is determined based on the adjacency matrix;

[0025] The adjacency matrix is ​​scaled by row and column using the degree matrix to obtain the feature mean corresponding to the feature sum; the feature mean is used to generate the recognition processing result of the graph convolutional network.

[0026] In one embodiment, the processing unit is further configured to perform foreground segmentation processing on the target image to extract the corresponding foreground image region from the target image;

[0027] The processing unit is further configured to enlarge the foreground image region and crop the enlarged foreground image region according to the target pixel size to obtain multiple cropped image regions; wherein, the cropped image region is an image block of the target image.

[0028] In one embodiment, the relational feature corresponding to the positional relationship includes an adjacency matrix, which contains multiple elements, any one of which describes the positional relationship between any two image patches of the target image; when determining the relational feature corresponding to the positional relationship, the processing unit is specifically used to:

[0029] Obtain the first position of any image patch in the target image, and based on the first position, obtain the associated image patches that have an eight-adjacent relationship with any image patch;

[0030] The elements in the adjacency matrix that describe the positional relationship between any image block and the associated image block are set to 1, while the elements that describe the positional relationship between any image block and other image blocks are set to 0.

[0031] In one embodiment, when the processing unit aggregates the block classification results of each image patch in the latent space and generates the image classification result of the target image based on the aggregation result, it is specifically used for:

[0032] The block classification results of each image patch in the latent space are aggregated to obtain the classification prediction result of the target image in the latent space.

[0033] The classification prediction result of the target image in the latent space is transformed to obtain the image classification result of the target image in the probability space.

[0034] In one embodiment, when the processing unit aggregates the block classification results of each image patch in the latent space to obtain the classification prediction result of the target image in the latent space, it is specifically used for:

[0035] In the latent space, an exponential operation is performed based on the block classification result of each image block, and the exponential operation results corresponding to each image block are summed to obtain the parameter exponential sum;

[0036] Logarithmic operations are performed on the exponents of the parameters in the latent space, and the result of the logarithmic operation is used as the classification prediction result of the target image in the latent space.

[0037] In one embodiment, the image classification result includes the anomaly type of the target object in the target image and the probability of being the anomaly type, wherein there are one or more anomaly types, and one anomaly type corresponds to one probability;

[0038] The processing unit is further configured to select the maximum probability from one or more probabilities contained in the image classification result, and determine the anomaly type corresponding to the maximum probability.

[0039] The processing unit is further configured to take the anomaly type corresponding to the maximum probability as the anomaly type of the target object in the target image, and output the anomaly type.

[0040] In one embodiment, the processing unit is further configured to obtain the logical value of the block classification result of each image block during the process of aggregating the block classification results of each image block in the latent space;

[0041] The processing unit is further configured to generate a classification heatmap of the target image based on the logical value corresponding to each image block.

[0042] In one embodiment, when the processing unit generates a classification heatmap of the target image based on the logical value corresponding to each image patch, it is specifically used for:

[0043] Based on the value of the logical value corresponding to each image block, the corresponding image block in the target image is colored; wherein, the value of the logical value is positively correlated with the color intensity of the coloring process.

[0044] The colored target image is used as the classification heatmap of the target image.

[0045] In one embodiment, the image classification result is obtained by calling a trained target classification model to perform recognition processing on the target image; the processing unit is further configured to obtain a sample image and a category label for the sample image; the category label is used to indicate the reference probability that a sample object in the sample image is an anomalous object;

[0046] The processing unit is further configured to call the target classification model to perform recognition processing on the sample image, and obtain the image classification result of the sample image, wherein the image classification result of the sample image includes the predicted probability that the sample object in the sample image is the object category;

[0047] The processing unit is further configured to train the target classification model based on the difference between the predicted probability and the reference probability, so as to obtain a trained target classification model.

[0048] In another aspect, embodiments of the present invention provide a computer device, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program supporting the computer device in executing the above-described method, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the following steps:

[0049] Acquire a target image, the target image containing a target object, and the target image is divided into multiple image blocks;

[0050] The target image is subjected to feature extraction processing to obtain the image features of the target image; the image features include the block features of each image patch in the latent space;

[0051] In the latent space, the corresponding image blocks are classified and predicted according to the block features of each image block to obtain the block classification result of each image block;

[0052] The block classification results of each image patch are aggregated in the latent space, and the image classification result of the target image is generated based on the aggregation result. The aggregation process in the latent space is used to adjust the contribution of the block classification results of different image patches in the generation of the image classification result. The image classification result is used to indicate the probability that the target object in the target image is an anomalous object.

[0053] In another aspect, embodiments of the present invention provide a computer-readable storage medium storing program instructions, which, when executed by a processor, are used to perform the image processing method as described in the first aspect.

[0054] In another aspect, embodiments of the present invention provide a computer program product, characterized in that the computer program product includes a computer program or computer instructions, wherein the computer program or computer instructions are executed by a processor as the image processing method described in the first aspect.

[0055] In this embodiment, after acquiring a target image, the computer device first divides the target image into multiple image patches. Then, a trained neural network encoder converts these image patches into low-dimensional features. The image patch features calculated by the pre-trained neural network encoder are then used as input to perform classification prediction on each image patch. Furthermore, the computer device can aggregate these features to obtain the image classification prediction result. To address the issue of normal image patches contributing excessively to image prediction, this embodiment employs a logit-based Log-Sum-Exp Aggregator (L-LSE). This L-LSE suppresses the contribution of normal image patches by performing log-sum-exp aggregation in the logit space, thereby achieving accurate image classification. Additionally, to effectively utilize the contextual information of image patches, this embodiment further introduces a Context-aware Feature Extraction Module (CAFE). This module extracts deep features containing contextual information by aggregating information from adjacent image patches, successfully modeling the micro-environmental relationships of neighboring image patches, thereby effectively improving the accuracy of the classification prediction result obtained by the computer device. In addition, the classification prediction results obtained from computer equipment can be combined with the classification label of the target image to conduct supervised model training and obtain a classification model with good performance. This allows the image to be classified and predicted in subsequent processes based on the trained classification model. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1aThis is a schematic diagram of an image processing system provided in an embodiment of the present invention;

[0058] Figure 1b This is a schematic diagram of another image processing system provided in an embodiment of the present invention;

[0059] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of the present invention;

[0060] Figure 3a This is a schematic diagram of a foreground extraction process provided in an embodiment of the present invention;

[0061] Figure 3b This is a schematic diagram illustrating the extraction of block features from an image block according to an embodiment of the present invention;

[0062] Figure 3c This is a schematic diagram illustrating another method for extracting block features from an image block according to an embodiment of the present invention;

[0063] Figure 4 This is a schematic flowchart of another image processing method provided in an embodiment of the present invention;

[0064] Figure 5a This is a schematic diagram illustrating a 4-neighborhood relationship between image blocks according to an embodiment of the present invention;

[0065] Figure 5b This is a schematic diagram illustrating the D-neighborhood relationship between image blocks provided in an embodiment of the present invention;

[0066] Figure 5c This is a schematic diagram illustrating an 8-neighborhood relationship between image blocks provided in an embodiment of the present invention;

[0067] Figure 5d This is a schematic diagram of obtaining an adjacency matrix based on 8-neighborhood relationships provided by an embodiment of the present invention;

[0068] Figure 5e This is a schematic diagram of obtaining block features of an image block based on a graph convolutional network, provided by an embodiment of the present invention.

[0069] Figure 6a This is a schematic diagram of a classification prediction process based on an aggregator that maximizes latent space smoothness, provided by an embodiment of the present invention.

[0070] Figure 6b This is a schematic diagram of a probability prediction in the latent space provided by an embodiment of the present invention;

[0071] Figure 6c This is a schematic diagram illustrating the comparison of the contribution of different image patches when performing probability prediction in the probability space and the latent space, as provided in an embodiment of the present invention.

[0072] Figure 6d This is a schematic diagram of a target image classification and prediction process provided by an embodiment of the present invention;

[0073] Figure 6e This is a schematic diagram of a pathological image prediction based on a trained target classification model provided in an embodiment of the present invention;

[0074] Figure 6f This is a schematic diagram of a method for classifying and predicting pathological images and outputting prediction results, provided by an embodiment of the present invention.

[0075] Figure 6g This is a schematic diagram illustrating the performance comparison of various methods on different datasets provided by an embodiment of the present invention;

[0076] Figure 6h This is a schematic diagram of outputting lesion images from pathological images according to an embodiment of the present invention;

[0077] Figure 7 This is a schematic block diagram of an image processing device provided in an embodiment of the present invention;

[0078] Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0079] This application proposes an image processing method that enables a computer device to perform classification and prediction processing on a target image containing a target object, and to obtain the probability that the target object contained in the target image is an abnormal object. In one embodiment, the target image is a pathological image, where a pathological image refers to an image obtained after pathological collection of an object. The target object contained in this image is a tissue or structural object of that object. That is, by performing pathological collection on the tissue or structure of an object, a pathological image containing that tissue (or structure) can be obtained. This pathological image is then used as the target image for classification and prediction processing to determine whether the tissue (or structure) contained in the pathological image is an abnormal tissue (or abnormal structure). In another implementation, the target image can also be a game image, and the target object contained in the target image can be a scene element in the game image. Therefore, by performing classification and prediction processing on the target image, the scene elements in the game image can be analyzed to determine whether the game image is a cheating image containing cheat software. In another implementation, the target image can be a scene image obtained by image acquisition of an examination scenario. In this case, the target image can represent the various characters participating in the examination. The computer device can then determine, through classification and prediction processing of the target image, whether any of the characters participating in the examination exhibit cheating behavior. In yet another implementation, the target image is a game image, and the target objects contained within it are game objects within the game image. The computer device can analyze the game actions of the game objects in the game image through classification and prediction processing of the target image. By analyzing these actions, the computer device can determine the action category of the game object. This action category can indicate the type of game action performed by the game object in the game, and the type can be one or more of the following: offensive, defensive, and idle, etc.

[0080] It is understood that a pathological image containing tissue structure is a type of histopathological image, which can be used for pathological diagnosis in histopathology. Classifying histopathological images can effectively assist pathologists in diagnostic procedures. In one embodiment, the current histopathological image is an automated whole-slide image (WSI). Because pathological images are used for pathological diagnosis, the acquired WSI images are generally ultra-high resolution images. To reduce the computational burden on computer equipment when processing ultra-high resolution WSIs, the computer equipment can first divide the WSI into multiple image blocks, and then obtain classification prediction results for the WSI through the recognition processing of these image blocks.

[0081] When a computer device divides a WSI into image patches and performs classification and prediction processing on the WSI through the recognition of these image patches, the following two methods can be used:

[0082] ① Feature-based methods.

[0083] Feature-based methods first obtain the block features corresponding to each image patch in WSI, then aggregate all block features in the probability prediction space to obtain the corresponding WSI features, and generate the WSI classification result based on the aggregated WSI features.

[0084] ② Instance-based approach.

[0085] The instance-based method can obtain a classification result for each image patch after obtaining multiple image patches of WSI by performing recognition processing on each image patch. Furthermore, the classification results of each image patch can be integrated to obtain the classification result corresponding to WSI.

[0086] The two aforementioned methods for classifying and predicting WSI images by recognizing and processing image patches suffer from several drawbacks. The first, feature-based method, suffers from high computational complexity and poor lesion localization capabilities. The second, instance-based method, suffers from a lack of contextual information, difficulty in optimization, and an excessive contribution of normal image patches to the prediction results. Consequently, both methods achieve low accuracy in classifying and predicting WSI images. Therefore, this application proposes an image processing method that, after acquiring a target image containing the target object, first divides the target image into multiple image patches, then performs feature extraction processing on each image patch to obtain the block features of each image patch in the latent space. Based on these latent space block features, the block classification result of the corresponding image patch can be determined. After the computer device obtains the block classification results of each image block, it can further aggregate the block classification results of each image block in the latent space. Based on the result of the aggregation process, the image classification result of the target image in the probability space can be generated. By determining the image classification result of the target image in the probability space, the computer device can determine the probability that the target object contained in the target image is an abnormal object.

[0087] In one embodiment, after the computer device performs segmentation and feature extraction on the target image and obtains the block features of each image patch in the latent space, it further obtains the block classification results of each image patch in the latent space. By aggregating these block classification results in the latent space, the computer device can adjust the contribution of different image patches to the image classification results of the target image. In other words, by aggregating the block classification results in the latent space, the computer device can suppress the contribution of normal image patches in the classification prediction of the target image, thereby improving the accuracy of the computer device's classification prediction of the target image. Furthermore, when extracting the block features corresponding to each image patch, the computer device can also obtain the positional relationship between each image patch and other image patches based on the position of each image patch in the target image. This allows the corresponding positional relationship to be incorporated into the block features when extracting the block features of each image patch, making the extracted block features deep features containing contextual information. Based on the extraction of deep features, the block features extracted by the computer device can more accurately describe the target image.

[0088] This image processing method can be applied to, for example... Figure 1a In the image processing system shown, the system includes an image acquisition device 10 and a computer device 11. The image acquisition device 10 acquires a target image, and after acquiring the target image, it sends the image to the computer device 11. The computer device 11 then performs classification and prediction processing on the target image and obtains the corresponding classification and prediction results. After obtaining the classification and prediction results, the computer device 11 can either return the results to the image acquisition device 10 for display or send them to a doctor's terminal for display. In one embodiment, the process of the computer device 11 sending the classification and prediction results to the doctor's terminal after completing the classification and prediction processing can be as follows: Figure 1b As shown, the doctor's terminal can be as follows: Figure 1b Terminal devices marked with 12.

[0089] In one embodiment, both the image acquisition device 10 and the doctor's terminal 12 are terminal devices. Terminal devices include, but are not limited to, smartphones, tablets, smart wearable devices, smart voice interaction devices, smart home appliances, personal computers, in-vehicle terminals, etc. This application does not impose any restrictions on these. The computer device 11 can be a terminal device or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but is not limited to these. The number of servers is also not limited in this application. In another implementation, the image processing system may only include the computer device 11, meaning that the computer device 11 itself acquires the target image and subsequently performs classification and prediction processing on the acquired target image.

[0090] Please see Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application, such as... Figure 2 As shown, the method may include:

[0091] S201, Obtain the target image, which contains the target object and is divided into multiple image blocks.

[0092] S202, perform feature extraction processing on the target image to obtain the image features of the target image; the image features include the block features of each image patch in the latent space.

[0093] In steps S201 and S202, the computer device can first acquire the target image and then perform image block division processing on the target image to obtain multiple image blocks of the target image. In one implementation, the computer device can directly divide the target image according to a certain pixel size, such as directly dividing the target image into square image blocks of 256 pixels each. The resulting square image blocks are the image blocks of the target image. In another implementation, the computer device can also first extract a foreground image from the target image. The foreground image refers to an image containing only the target object. That is, since the target image contains not only the target object but also other image elements, when extracting the foreground image from the target image, the computer device removes the other image elements from the target image, retaining only the image of the target object itself, thus obtaining the corresponding foreground image.

[0094] After the computer device extracts the foreground image, it can perform image block segmentation on the foreground image, and use the resulting image blocks as image blocks of the target image. The computer device can use a foreground extraction algorithm to determine the foreground image from the target image. The foreground image extracted by the foreground extraction algorithm is the target to be identified by the computer device. That is, since the target to be identified in the target image is the target object contained in the target image, the computer device uses the foreground extraction algorithm to extract the foreground image containing only the target object from the target image. Figure 3a As shown, if the target object contained in the target image is Figure 3a The foreground image obtained by the computer device after performing foreground extraction on the target image is an image containing only the target object. It can be seen that by extracting the foreground image from the target image by the computer device, image content that does not need to be recognized in the target image can be effectively deleted, so that the computer device does not need to recognize the image content, thus achieving effective saving of the computer device's processing resources.

[0095] After the computer device extracts the foreground image from the target image using a foreground extraction algorithm, it can also divide the foreground image into square image blocks of 256 pixels each when performing block partitioning based on the foreground image. It should be noted that since the foreground extraction algorithm performed on the target image may affect the image size, the computer device can first enlarge the foreground image to the target size before performing block partitioning. In one embodiment, the image blocks obtained by the computer device after block partitioning the target image are non-overlapping. Therefore, the block features extracted from each image block can effectively reflect the image features of the target image.

[0096] After a computer device obtains a target image divided into multiple image blocks, it can perform feature extraction processing on each image block to obtain block features for each block. These block features can then be used to reflect the image characteristics of the target image. When extracting block features for each image block, the computer device can input each image block into a pre-trained encoder to obtain the corresponding block features. The process by which the computer device calls the pre-trained encoder to obtain the block features of the corresponding image block can be described as follows: Figure 3bAs shown. In one embodiment, since the target image is divided into multiple image blocks, and each image block corresponds to an image region in the target image, the block features of the image block obtained by the computer device are used to represent the regional image features of the image region within that image block. In another implementation, to incorporate contextual information of surrounding image blocks into the block features corresponding to each image block, the computer device can determine the corresponding adjacency matrix based on the positional relationship of each image block in the target image after calling the pre-trained encoder to obtain the initial block features of the corresponding image block, such as... Figure 3c As shown in the diagram. The values ​​in this adjacency matrix describe whether a given image patch is adjacent to any other image patch in the target image.

[0097] After the computer device obtains the initial block features of the target image patch and determines the corresponding adjacency matrix based on the positional relationship between each image patch, the computer device can further integrate the adjacency matrix into the initial block features of the corresponding image patch to obtain the final block features of the corresponding image patch. By integrating the adjacency matrix into the block features of the corresponding image patch, the computer device can incorporate the contextual image information in the target image into the block features of the corresponding image patch. In other words, through the process of determining the adjacency matrix and integrating it into the block features of the corresponding image patch, the neighborhood relationship of the target image can be integrated into the block features of each image patch. This means that the block features obtained by the computer device can not only describe the image features but also contain the positional information between different image patches, thus improving the information richness of the block features obtained by the computer device based on the image patch. Consequently, the accuracy of the computer device in classifying and predicting the target image based on the block features can be improved.

[0098] The block features obtained by the computer device are features in the latent space. After the computer device obtains the block features of each image block in the latent space, it can perform classification and prediction processing on the corresponding image block in the latent space based on the block features, and obtain the block classification result of the corresponding image block, and then execute step S203.

[0099] S203, in the latent space, classify and predict the corresponding image blocks according to the block features of each image block to obtain the block classification result of each image block.

[0100] S204, perform aggregation processing on the block classification results of each image patch in the latent space, and generate the image classification result of the target image based on the aggregation processing result; wherein, the aggregation processing in the latent space is used to adjust the contribution of the block classification results of different image patches in the generation of the image classification result, and the image classification result is used to indicate the probability that the target object in the target image is an abnormal object.

[0101] In steps S203 and S204, when the computer device performs classification prediction processing based on the block features of each image block in the latent space and aggregates the block classification results of each image block, it can use a logit-based Log-Sum-Exp Aggregator (L-LSE). That is, the block features of each image block are input into the aggregator L-LSE to obtain the block classification result of the image block in the latent space. Then, the aggregator L-LSE can aggregate the block classification result to generate the image classification result of the target image. In one embodiment, the computer device performs aggregation processing on the block classification results of each image patch in the latent space using a smooth-maximizing aggregator. Since the aggregation processing involves exponential operations, image patches containing normal image regions will generate asymptotic zero values ​​when performing exponential operations in the latent space. Based on these asymptotic zero values, when the computer device aggregates the block classification results of the corresponding image patches and generates image predictions for the target image, it reduces the reference to the block classification results of the image patches that have generated asymptotic zero values. Therefore, it effectively offsets the contribution of image patches containing normal image regions in the image prediction process, thereby effectively strengthening the role of image patches containing abnormal image regions in the image prediction process. This can effectively improve the accuracy of the computer device in subsequently predicting the image type of the target image based on the block classification results of each image patch.

[0102] When a computer device performs classification prediction processing on a target image using the block classification results of each image patch in the target image, it can first aggregate the block classification results of each image patch in the latent space to obtain the classification prediction result of the target image in the latent space. Then, the classification prediction result of the target image in the latent space can be transformed to the probability space to obtain the image classification result of the target image. When transforming the classification prediction result of the target image in the latent space to the probability space, the computer device can use a sigmoid (a smoothing function) operation to transform the classification prediction result of the target image in the latent space and obtain the final image classification result of the target image in the probability space. In one embodiment, the image classification result obtained by the computer device for the target image can be used to indicate the probability that the target object contained in the target image is an anomalous object. The image classification result obtained by the computer device can be a binary classification result. When the image classification result is a binary classification result, the image classification result is 0 or 1. When the image classification result is 0, it indicates that the target object contained in the target image is an anomalous object, and when the image classification result is 1, it indicates that the target object contained in the target image is a normal object. In another implementation, the image classification result obtained by the computer device can also be a multi-class prediction result. When the image classification result is a multi-class prediction result, the image classification result includes the corresponding category result and the probability that the target object belongs to the corresponding category result. After obtaining the multi-class prediction result, since the multi-class prediction result contains multiple prediction probabilities, the computer device can determine the maximum value of the prediction probability from the multiple prediction probabilities obtained. Then, the computer device can use the category corresponding to the maximum value prediction probability as the predicted classification category of the target object.

[0103] In this embodiment, after acquiring a target image containing a target object, the computer device can segment the target image to obtain multiple image blocks. After obtaining these multiple image blocks, feature extraction is performed on each image block to obtain block features in the latent space that describe the image features of the target image. After obtaining the block features of each image block in the latent space, the computer device can further perform classification prediction processing on the corresponding image blocks in the latent space based on these block features to obtain the block classification results of the corresponding image blocks. Since the aggregation processing of the block prediction results of each image block in the latent space can effectively suppress the contribution of normal image blocks in the aggregation process, it effectively adjusts the weight given to the block classification results of normal image blocks in the multiple image blocks when performing classification prediction on the target image. Therefore, after obtaining the block classification results of each image block in the latent space, the computer device can further aggregate the block classification results of each image block in the latent space to obtain the classification prediction result of the target image in the latent space. Based on the determination of the classification prediction result of the target image in the latent space, the computer device can transform the classification prediction result of the target image in the latent space into the probability space, thereby obtaining the image classification result of the target image. Since the computer device can effectively adjust the contribution of normal image blocks in the aggregated image block image classification result when aggregating the block classification results of each image block in the latent space, the classification prediction result of the target image obtained by the computer device effectively amplifies the proportion of the block classification results of abnormal image blocks in the multiple image blocks. Therefore, when the computer device uses the classification prediction result of the target image in the latent space to obtain the final image classification result, it can effectively rely on the abnormal image regions in the target image blocks for prediction processing, thereby improving the accuracy and reliability of the computer device in classifying and predicting the target image.

[0104] Please see Figure 4 This is a schematic flowchart of an image processing method proposed in an embodiment of this application. Please refer to [link / reference]. Figure 4 The method may include:

[0105] S401, Obtain the target image, which contains the target object and is divided into multiple image blocks.

[0106] S402, perform feature extraction processing on the target image to obtain the image features of the target image; the image features include the block features of each image patch in the latent space.

[0107] In steps S401 and S402, after acquiring the target image, the computer device can first perform segmentation processing on the target image. Since the target image is a pathological image (i.e., WSI), and WSI is a large-sized and high-resolution image, coupled with the display memory of the computer device's graphics processing unit (GPU), in this embodiment, the computer device first extracts the foreground image from the target image and then performs block segmentation processing on the foreground image. In specific implementation, the computer device can first perform foreground segmentation processing on the target image to extract the corresponding foreground image region from the target image; the foreground image region is the image region containing the target image in the target image. After obtaining the foreground image region, the computer device can enlarge the foreground image region and crop the enlarged foreground image region according to the target pixel size to obtain multiple cropped image regions; wherein, the cropped image region is the image block of the target image.

[0108] In other words, the computer device can divide the target image (i.e., the pathological image) into small, non-overlapping image patches. Then, the computer device can use a pre-trained encoder to extract low-dimensional features. In a specific implementation, the computer device can use the foreground segmentation algorithm in CLAM (a pathological image classification algorithm) to segment the corresponding foreground image region from the target image. Then, the computer device can crop the foreground image region at a magnification of 20 times to obtain a square image patch of 256 pixels (i.e., the cropped image region mentioned above). After obtaining these image patches, the computer device can use a ResNet18 model (a classification model) pre-trained on ImageNet-1K (a dataset) as an encoder to convert each image patch into 512-dimensional low-dimensional features, thereby extracting the block features of each image patch in the latent space.

[0109] After dividing an image patch into multiple image patches, the computer device can perform feature extraction processing on these multiple image patches and obtain the block features of each image patch in the latent space. During feature extraction, the computer device can first acquire the multiple image patches obtained after dividing the target image. Then, based on the position of each image patch in the target image, it can determine the positional relationship between any two image patches. Furthermore, the computer device can extract the initial block features of each image patch and determine the relational features corresponding to the positional relationships. In one embodiment, when extracting the initial block features of each image patch, the computer device can use a pre-trained encoder (such as a pre-trained ResNet18 model) to encode each image patch, thereby obtaining the initial block features of the corresponding image patch.

[0110] When determining the positional relationship between any two image blocks in a plurality of image blocks, this computer device determines the 8-neighborhood relationship between any two image blocks. In one embodiment, an image block in the target image can be denoted as P, and its position in the target image can be denoted as (x, y). Then, the positions of the image blocks adjacent to image block P (x, y) in the target image are (x+1, y), (x-1, y), (x, y+1), and (x, y-1), respectively, as detailed below. Figure 5a As shown, and with image block P, there exists... Figure 5a The image patches with the shown adjacency relationship can be referred to as the 4-neighborhood of image patch P. In addition, image patch P also has the following relationships: Figure 5b The diagonal adjacency relationship shown is illustrated in the diagram. The image blocks adjacent to image block P in the target image are located at positions (x+1, y+1), (x-1, y+1), (x-1, y-1), and (x+1, y-1), respectively. This positional relationship can be referred to as the D-neighborhood of image block P. Therefore, the 8-neighborhood of image block P includes the aforementioned 4-neighborhood and D-neighborhood. In other words, the 8-neighborhood image blocks of image block P include the aforementioned 4-neighborhood and D-neighborhood image blocks. Thus, the 8-neighborhood image blocks of image block P can be... Figure 5c As shown, Figure 5c As shown, the image blocks that have an 8-neighbor relationship with image block P include all image blocks that have a 4-neighbor relationship with image block P and a D-neighbor relationship with image block P.

[0111] After determining the 8-neighborhood relationships between image patches in the target image, the computer device can determine the relational features corresponding to these positional relationships. These features include an adjacency matrix, which contains multiple elements, each describing the positional relationship between any two image patches in the target image. Therefore, when determining the relational features, the computer device can first obtain the first position of any image patch in the target image (e.g., the position of image patch P in the target image mentioned above). Then, based on this first position, the computer device can obtain the relationship between any two image patches. Image blocks with an 8-neighbor relationship are associated image blocks. The associated image blocks with an 8-neighbor relationship to any image block, as acquired by the computer device, include all image blocks with a 4-neighbor relationship to that image block and a D-neighbor relationship to image block P. After the computer device acquires the associated image blocks with an 8-neighbor relationship, the elements in the adjacency matrix describing the positional relationship between any image block and associated image blocks are set to 1, while the elements describing the positional relationship between any image block and other image blocks are set to 0. If the target image contains multiple image blocks as follows... Figure 5dGiven the four image patches a, b, c, and d shown, the computer device can determine the corresponding adjacency matrix based on the 8-neighborhood positional relationships as follows: Figure 5d As shown.

[0112] After extracting the initial block features of each image patch and the relational features of each image patch in the target image, the computer device can perform perceptual fusion processing on the initial block features of each image patch according to the relational features to obtain perceptual block features, and use the obtained perceptual block features as the image features of the target image. In one embodiment, the relational features corresponding to the positional relationships include an adjacency matrix. The perceptual fusion processing is performed by the computer device by calling the Context Aware Feature Extraction Module (CAFE), which includes a Graph Convolutional Network (GCN) and a Rectified Luminaire (RELU). Therefore, when the computer device performs perceptual fusion processing on the initial block features of each image patch according to the relational features to obtain perceptual block features, it can first input the initial block features and adjacency matrix of each image patch into the first Graph Convolutional Network in the Context Aware Feature Extraction Module to obtain the first intermediate block features. In a specific implementation, if the target image patch WSI contains k image patches, then the initial block features obtained after performing feature extraction processing on the k image patches can be denoted as X, and each initial block feature X satisfies the following equation 1.

[0113] X∈R k×d ={x1,x2,...,x k Equation 1

[0114] Where X represents the initial block feature set of each image patch in the target image, x i The initial block feature of any image patch in the target image is represented by i, where i ranges from 1 to k, and k is the total number of image patches contained in the target image.

[0115] Furthermore, the adjacency matrix determined by the computer device based on the positional relationships between image patches in the target image can be represented by A. And as can be seen from the above-described adjacency matrix generation process, the adjacency matrix satisfies: A∈[0,1] k×k Therefore, based on the determined initial block feature X and adjacency matrix A, the computer device inputs the initial block feature X and adjacency matrix A into the context-aware block feature extraction module to obtain the first intermediate feature, which is GCN1(X,A).

[0116] After the computer device calls the first graph convolutional network GCN1 to obtain the first intermediate block features of each image block, it can further call the first rectified network ReLU in the context-aware block feature extraction module to perform linear rectification processing on the first intermediate block features to obtain the second intermediate block features of each image block. The second intermediate block features obtained after the computer device calls the rectified network ReLU to perform rectification processing on the first intermediate block features can be shown in Equation 2.

[0117] Z = RELU(GCN1(X,A)) + X (Equation 2)

[0118] Where Z represents all the second intermediate block features obtained based on all the first intermediate block features, GCN1 is the first graph convolutional network layer in the context-aware block feature extraction module CAFE, X is the initial intermediate image block of each image block, and A is the adjacency matrix mentioned above.

[0119] After the computer device obtains the second intermediate block features of each image block, it can call the second graph convolutional network and the second rectified network in the context-aware block feature extraction module to perform recognition processing on the second intermediate block features and adjacency matrix of each image block, and use the recognized block features as the perception block features. The process by which the computer device calls the second graph convolutional network and the second rectified network to obtain the perception block features based on the second intermediate block features can be shown in Equation 3.

[0120] X CA =RELU(GCN2(Z,A))+Z Equation 3

[0121] Among them, X CA The perceptual block features obtained through the context-aware block feature extraction module are represented by GCN2 (the second graph convolutional network), ReLU (the second rectified network), and Z (the first intermediate block feature). It can be understood that the perceptual block features X obtained after the two graph convolutional networks and two rectified networks of the context-aware block feature extraction module are... CA It also satisfies Equation 4 below.

[0122]

[0123] in, Then, it is used to represent the perceptual block feature of the i-th image block in the target image, where i takes the value from 1 to k, and k is the total number of image blocks contained in the target image.

[0124] In one embodiment, the process by which the computer device inputs the initial block features of an image patch into the context-aware block feature extraction module's two graph convolutional networks and two rectified networks for separate processing can be as follows: Figure 5eAs shown. The graph convolutional network in the context-aware block feature extraction module has input features including the initial block features and intermediate block features of each image block. When the graph convolutional network performs recognition processing on the input features and the adjacency matrix, it can first determine the feature sum of the corresponding input features of each image block in the target image based on the adjacency matrix and the input features (including the initial block features X and the second intermediate block features Z of the image blocks mentioned above). Then, the computer device can determine the degree matrix corresponding to the positional relationship based on the adjacency matrix. Then, the degree matrix can be used to scale the adjacency matrix by row and column respectively to obtain the feature mean corresponding to the feature sum. The feature mean is used to generate the recognition processing result of the graph convolutional network. In specific implementation, the graph convolutional network (including the first graph convolutional network GCN1 and the second graph convolutional network GCN2 mentioned above) can use the following Equation 5 to integrate information from neighboring image blocks to obtain the corresponding aggregated features.

[0125]

[0126] in, It is the sum of the adjacency matrix A and the identity matrix. for The degree matrix is ​​given by W, which represents the learnable weights, and H is the input feature. The value of H can be either the initial block feature X or the second intermediate block feature Z. This is the mean of the feature vectors of all neighbors of the input feature H, scaled by row and column operations using the adjacency matrix. This scaling ensures that the input features maintain their original distribution within the feature matrix during transmission through the GCN network and prevents excessive feature differences between different image patches. The degree matrix... It can be calculated using Equation 6.

[0127]

[0128] After obtaining the perceptual block features of each image patch in the target image, the computer device can integrate the contextual image information of the target image into the block features of each image patch. Therefore, the computer device can use the perceptual block features of each image patch in the target image as the image features of the target image. This allows the computer device to perform image patch-level prediction on the information of the corresponding image patch, avoiding the neglect of the contextual information around the image patch. By introducing a context-aware block feature extraction module that considers the comprehensive information around each image patch, the extraction of complex diagnostic patterns can be achieved. After obtaining the perceptual block features of each image patch, the context-aware image patch features can be input into a Logit-based Log-Sum-Exp Aggregator (L-LSE) to predict the classification result of the target image.

[0129] S403, in the latent space, classify and predict the corresponding image blocks according to the block features of each image block to obtain the block classification result of each image block.

[0130] S404: Aggregate the block classification results of each image patch in the latent space to obtain the classification prediction result of the target image in the latent space.

[0131] S405, transform the classification prediction result of the target image in the latent space to obtain the image classification result of the target image in the probability space.

[0132] In steps S403 to S405, the block features processed by the computer device in the latent space are the perceptual block features X of the image block mentioned above. CA In other words, computer devices utilize multilayer perceptrons (MLPs) to analyze the perceptual block features X of each image patch in the latent space. CA Classification and prediction processing is performed to obtain the block classification result of each image patch in the latent space. This is achieved by using an MLP to analyze the perceptual block features X of each image patch. CA The process of classification and prediction is shown in Equation 7.

[0133]

[0134] in, Then, the perceptual block feature is used to represent the i-th image patch in the target image, where i ranges from 1 to k, k is the total number of image patches in the target image, and h i This represents the block classification result of the i-th image block obtained from the perceptual block features of the i-th image block.

[0135] After obtaining the block classification result for each image patch, the computer device can aggregate these results in the latent space to obtain the classification prediction result of the target image in the latent space. In a specific implementation, when aggregating the block classification results for each image patch in the latent space to obtain the classification prediction result of the target image, the computer device can first perform exponential operations based on the block classification results for each image patch in the latent space, and then sum the exponential operation results corresponding to each image patch to obtain the parameter exponent sum; then, it can perform a logarithmic operation on the parameter exponent sum in the latent space, and use the result of the logarithmic operation as the classification prediction result of the target image in the latent space. In one embodiment, the process of aggregating the block classification results of each image patch in the latent space by the computer device can be as shown in Equation 8.

[0136]

[0137] Where h represents the classification prediction result of the target image in the latent space, and γ is a hyperparameter. That is to say, based on the latent space smoothing maximization aggregator L-LSE, which includes the aforementioned multilayer perceptron (MLP) and the aggregation processing module Log-Sum-Exp, the process by which this computer device performs classification prediction processing on the perceptual block features of each image patch based on the latent space smoothing maximization aggregator L-LSE can be described as follows: Figure 6a As shown, Figure 6a As shown, after obtaining the classification prediction result h of the target image patch in the latent space, the latent space smooth maximization aggregator L-LSE can transform the classification prediction result in the latent space into the probability space, that is, perform a sigmoid operation on the classification prediction result h in the latent space to obtain the image classification result of the target image in the probability space.

[0138] In one embodiment, when a computer device performs image classification prediction on the classification prediction result of a target image in the latent space using the latent space smooth maximization aggregator L-LSE, it directly performs a sigmoid operation on the classification prediction result in the latent space, as shown in Equation 9.

[0139]

[0140] Here, P represents the classification probability (i.e., the image classification result of the target image) after transforming the classification prediction result in the latent space to the probability space. It can be seen that the latent space smooth maximization aggregator L-LSE reduces the redundant contribution of normal image patches in the target image to the aggregation result while increasing the contribution of abnormal image patches in the target image. This ensures that all abnormal image patches contribute according to their degree of abnormality, rather than overemphasizing the most prominent one as in max pooling. This method, based on latent space smooth maximization, performs a smooth maximization operation in the latent space before the output probability, rather than the actual probability space, to obtain the aggregated value (i.e., the classification prediction result of the target image in the latent space). Then, a sigmoid operation is performed on the aggregated value to obtain the final image classification probability in the probability space. However, because the aggregation process involves exponential operations, many normal image patches will produce asymptotically zero values ​​when performing exponential operations in the latent space. Figure 6b As shown, the generation of a large number of asymptotically zero values ​​can effectively offset their contribution to the aggregation process. Simultaneously, the curve of anomalous image patches is strengthened at the upper end after exponential operations. In other words, by performing exponential operations in the latent space, the contribution of normal and anomalous image patches in the aggregation process is adjusted, allowing the computer to contribute according to the degree of anomalousness of the anomalous image patches. This contrasts sharply with aggregation processing in the probability space. Similarly, as... Figure 6bAs shown, during aggregation in probability space, many normal image patches consistently contribute a unit value. By adjusting the contribution of each image patch during the aggregation process, the computer can focus more on analyzing and processing anomalous images. Specifically, the difference in contribution between normal and anomalous image patches during aggregation in latent space and probability space can be illustrated as follows: Figure 6c As shown.

[0141] Below, in conjunction with Figure 6d This paper provides an overall description of the process by which computer equipment performs classification and prediction processing on target images. Specifically, this process includes the following steps ① to ③.

[0142] ① Image preprocessing.

[0143] The target image is divided into multiple non-overlapping image patches, and the initial block features of each image patch are extracted. In addition, an 8-neighborhood adjacency matrix is ​​constructed by combining the positional relationships of each image patch in the target image.

[0144] ② The perceptual block features of each image block are obtained through the context-aware feature extraction module CAFE.

[0145] Using the initial block features of each pre-extracted image patch and the 8-neighbor adjacency matrix as input, context-aware block features are obtained through the context-aware feature extraction module.

[0146] ③ Classification prediction is performed using the L-LSE aggregator based on latent space smoothness maximization.

[0147] The final predicted probability is obtained by combining the latent space smoothing maximization aggregator with perceptual block features.

[0148] In one embodiment, the image classification result obtained by the computer device includes the anomaly type of the target object in the target image and the probability of it being an anomaly type. There are one or more anomaly types, and each anomaly type corresponds to a probability. The computer device can then select the maximum probability from the one or more probabilities included in the image classification result and determine the anomaly type corresponding to the maximum probability. The anomaly type corresponding to the maximum probability can then be used as the anomaly type of the target object in the target image, and the anomaly type can be output. In another implementation, the image classification result obtained by the computer device is also a binary classification result. When the image classification result is binary, it is either 0 or 1. A classification result of 0 indicates that the target image is an anomalous image, while a classification result of 1 indicates that the target image is a normal image. Therefore, when the target image is a pathological image, a normal image indicates that the target object in the pathological image does not have a pathological problem, while an anomalous image indicates that the target object in the pathological image has a pathological problem, such as a possible tumor.

[0149] In one embodiment, the image classification result is obtained by calling a trained target classification model to recognize the target image. The image classification result obtained by the computer device can be a binary classification result. When the image classification result is a binary classification prediction result, the computer device can use the classification label Y of the target image combined with the binary cross-entropy loss to supervise the classification prediction probability P of the target image to construct the loss function. The constructed loss function is shown in Equation 10.

[0150] Loss=YlogP+(1-Y)log(1-P) Equation 10

[0151] In one embodiment, the process of training the target classification model by the computer device can be as follows: the computer device first acquires sample images and category labels for the sample images; the category labels are used to indicate the reference probability that a sample object in the sample image is an anomalous object; then, the computer device can call the target classification model to perform recognition processing on the sample images to obtain image classification results for the sample images, which include the predicted probability that a sample object in the sample image is an object category; further, the computer device can construct a loss function as shown in Equation 10 based on the difference between the predicted probability and the reference probability, and train the target classification model based on the loss function to obtain a trained target classification model. It can be understood that the training process of the target classification model is an end-to-end model optimization training method.

[0152] The process of model training is the process of machine learning. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence, and machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0153] In other words, a computer device can train a target classification model and then use that model to classify and predict target images. When the target image is a pathological image, the computer device can deploy the trained target classification model to a pathological diagnosis service after the model has been tested on a dataset. This allows the computer device to use the trained target classification model to perform pathological diagnosis on the image. The process of the computer device using the trained target classification model for pathological diagnosis can be as follows: Figure 6e As shown, the specific process is as follows: the user uploads a pathological image through terminal device A, the background uses a pre-trained target classification model to perform classification prediction processing, and outputs the obtained image classification result to terminal device B. When the image classification result determines that the pathological image is an abnormal image (that is, the tissue structure contained in the pathological image is an abnormal tissue structure), a lesion heat map can also be displayed.

[0154] In one embodiment, when a computer device performs classification prediction processing on a pathological image and the resulting image classification indicates that the target object in the pathological image is an abnormal object, the computer device, when outputting a lesion heatmap, can first obtain the logical value (i.e., the classification prediction probability of each image block) corresponding to the block classification result of each image block during the aggregation processing of the block classification results of each image block in the latent space; then, the computer device can generate a classification heatmap of the target image based on the logical value corresponding to each image block. When generating the classification heatmap of the target image based on the logical value corresponding to each image block, the computer device can perform color processing on the corresponding image blocks in the target image based on the magnitude of the logical value corresponding to each image block; wherein, the magnitude of the logical value is positively correlated with the color intensity of the color processing; then, the color-processed target image can be used as the classification heatmap of the target image. The computer device performs classification prediction processing on the pathological image to obtain the corresponding image classification result and the generated lesion heatmap as follows: Figure 6fAs shown, the display of lesion heat maps using computer equipment can effectively assist in assessing the condition of relevant users.

[0155] In one embodiment, to evaluate the effectiveness of the classification prediction process mentioned in this application embodiment, experiments were conducted on two datasets: CAMELYON16 (a dataset in the medical field) and TCGA-NSCLC (another dataset in the medical field). CAMELYON16 is a breast cancer classification dataset consisting of 399 pixel-level labeled H&E-stained WSIs, including 239 normal WSIs and 160 WSIs with sentinel lymph nodes (i.e., abnormal objects). The experiments used the official training and testing splits. The official training set was further randomly split into training and validation sets in a 9:1 ratio. TCGA-NSCLC is a dataset extracted from two lung cancer subtypes from the Cancer Genome Atlas project, including lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD). It includes 541 lung squamous cell carcinoma WSIs and 512 lung adenocarcinoma WSIs, and was randomly split into training, validation, and testing sets in a 65:10:25 ratio.

[0156] (1) Implementation details and evaluation indicators:

[0157] The framework was implemented using an NVIDIA TelsaV100 GPU with 32GB of RAM, and was run in Python and PyTorch. In all experiments, the latent dimension of the context-aware feature extraction module was 512. Optimization was performed using the Adam optimizer (an adaptive algorithm-based optimizer) with a weight decay of 10. -4 The learning rates for CAMELYON16 and TCGA-NSCLC are 10%. -4 and 2×10 -4 All methods used cosine learning rate decay and early stopping for a 200-round training process, with early stopping tolerances of 30 and 20 rounds for CAMELYON16 and TCGA-NSCLC, respectively. Batch size was set to 1.

[0158] The experiments report WSI classification and F1 score (F1) to evaluate classification performance. Since TCGA-NSCLC lacks fine-grained labels, the experiments only report the area under the curve (patch AUC) for image patch classification on the CAMELYON16 dataset to evaluate lesion localization ability. All experiments were conducted triplicate with three random seeds, and the mean and standard deviation are reported. The inference computation and parameter count for all methods on the CAMELYON16 dataset Tumor001WSI are also reported.

[0159] (2) Baseline comparison method:

[0160] The proposed method in the application embodiments is comprehensively compared with recent instance-based and feature-based methods. Classic instance-based methods include: Max Pooling, Mean Pooling, LSE (a smoothed form of max pooling), MIL-RNN (a recurrent neural network), and WENO (an instance-based classification prediction method). Feature-based methods include: ABMIL (aggregate classification), DSMIL (self-supervised comparison learning prediction method), CLAM (a feature extraction prediction method), TransMIL (encoding dependency method), and PatchGCN (a graph convolutional network-based method).

[0161] The following is a brief description of the process by which a computer device trains a model using instance-based and feature-based methods to perform target image recognition and prediction. In one embodiment, when training the model using an instance-based method, the computer device typically segments each WSI into small-resolution image patches and uses a pre-trained encoder to extract low-dimensional features for processing. WSI classification can be formulated as a Multiple Instance Learning (MIL) problem, where each WSI is considered a bag, and its image patch is considered an instance. In a binary classification scenario, MIL searches for a set of features from instances X = {x...} i ∈R D Let the mapping from instance {i = 1, 2, ..., K} to the bag label Y ∈ {0, 1} be defined. Assume each instance has a label y. i ∈{0,1}, but cannot be accessed during training. The package label is defined in Equation 11.

[0162]

[0163] Among them, y i Let Y represent the label of each instance, K represent the package label, and K represent the total number of instances contained in the package. Then, based on the MIL method, the prediction of packages can be achieved by modeling a mapping function f(·) and an aggregation function g(·), and the prediction algorithm can be expressed as Equation 12.

[0164] Y = g(f(x1),f(x2),...,f(x) n Formula 12

[0165] Based on the choice of f(·) and g(·), weakly supervised pathological image classification methods can be divided into feature-based methods and instance-based methods. In feature-based methods, f(·) is a feature extraction module used to model image patch features containing higher-order semantics, and g(·) aggregates the features of all image patches into pathological image features and then predicts the classification result. In instance-based methods, f(·) is a classifier for image patches that provides classification predictions based on the features of each image patch, and g(·) is an aggregator that generates classification predictions for the pathological image based on the classification predictions of all instances.

[0166] Feature-based methods focus on designing aggregation techniques for pathological image features. ABMIL uses an attention mechanism as the aggregation function, leveraging a multilayer perceptron to obtain attention weights for all image patches, and linearly combining image patch features based on these attention weights to obtain pathological image features. DSMIL uses another branch to explicitly model the relationship between key image patches and other image patches, addressing the modeling problem of inter-instance relationships to some extent. CLAM improves classification accuracy by imposing constraints on the features of the image patches with the highest and lowest predicted probabilities using positive and negative pseudo-labels as instance-level supervision. TransMIL generates pathological image features by adding classification tokens and using a sequence-to-sequence linear Transformer model. While this method achieves high-accuracy classification by explicitly modeling the correlations of all instances, it is prone to overfitting. Furthermore, PatchGCN represents pathological images as graph structures, utilizing graph neural networks to model the contextual relationships between image patches. Although feature-based methods perform well, they suffer from high computational complexity, susceptibility to overfitting, and poor interpretability.

[0167] Instance-based methods first use p i =f(x) i The algorithm predicts the classification probability of each instance and then aggregates all instance probabilities using g(·) to obtain the bag classification probability P. Existing instance-based methods generate image patch-level predictions only based on the corresponding image patch, ignoring the contextual information around the image patch, resulting in poor image patch-level prediction performance. In addition, the aggregator of existing methods has several problems.

[0168] Mean pooling calculates the classification prediction probability of a pathological image by averaging the classification prediction probabilities of all image patches. This classification prediction probability can be represented by Equation 13.

[0169]

[0170] Where P represents the prediction probability, p i This represents the predicted probability of the i-th instance, where i can range from i to K, and K is the total number of instances.

[0171] However, practice has shown that the mean pooling method has the following two problems:

[0172] (1) In pathological images, the proportion of lesion areas in the entire image varies greatly from 0.02% to 80%, and mean pooling cannot model the classification task of lesion areas with such different proportions. If the number of normal tissue image blocks in a pathological image far exceeds the number of tumor tissue image blocks, the aggregation probability obtained by mean pooling will be significantly reduced, leading to misclassification of the pathological image.

[0173] (2) Mean pooling can be interpreted as using the labels of pathological images to supervise all image patches, which inevitably introduces a lot of supervision noise and reduces the accuracy of image patch classification.

[0174] Max pooling selects the highest image patch classification probability as the pathological image classification probability, which can be represented by Equation 14.

[0175] P = max(p1, p2, ..., p K ) = max 1≤i≤K p i Formula 14

[0176] While max pooling overcomes the problems of mean pooling, conforms to the definition of multi-instance learning, and is suitable for pathological image classification tasks, it still performs poorly in this area. The reason for max pooling's poor performance in pathological image classification is the difficulty of optimization. Each pathological image contains tens of thousands of image patches, but during training, only one patch is used for optimization per round. This leads to a skewed optimization trajectory and difficulty in converging to the optimal solution.

[0177] A smoothed form of max pooling, LSE, constructs the classification probability of a pathological image by merging the classification probabilities of all image patches. The classification probability obtained using LSE can be represented by Equation 15.

[0178]

[0179] Where γ is a hyperparameter. The upper and lower bounds of the probability P calculated using the LSE algorithm satisfy the following equation 16.

[0180]

[0181] In one embodiment, while the method performs well on small-scale multi-instance learning datasets, its performance remains poor in pathological image classification. In pathological images, the average number of processed image patches is as high as 10,000, therefore logK is significantly greater than p. i∈[0,1]. On the one hand, if γ is small, LSE is essentially close to mean pooling because logK / γ becomes a large constant, making max... i p i The impact of γ is negligible. On the other hand, if γ is large, the settlement result of LSE will asymptotically approach max pooling. Different values ​​of γ were tried in the experiment, but LSE still performed poorly. In addition, the calculation result of LSE is easily affected by the excessive contribution of normal instances.

[0182] (3) Experimental results:

[0183] like Figure 6g As shown in the chart, Mean Pooling fails to model the proportion of different tissues in different WSIs, and MaxPooling is affected by optimization problems, resulting in poor performance for both. Although LSE solves the problem of Max Pooling, it is still plagued by the problem of excessive contribution from normal image patches, leading to poor performance. The Latent Space Smooth Maximization Aggregator (L-LSE) proposed in this application alleviates the optimization problem and the problem of excessive contribution from normal image patches by performing a smooth maximization operation in the latent space. It achieves an AUC of 0.8781 and an F1 score of 0.8109 on the CAMELYON16 dataset, and an AUC of 0.9048 and an F1 score of 0.8339 on the TCGA-NSCLC dataset. It not only surpasses previous instance-based methods in WSI classification, but also outperforms all comparative methods with a patch AUC of 0.8823 in image patch classification. By introducing a context-aware feature extraction module (CAFE) to model the contextual information of image patches, the classification and localization performance is further improved. On CAMELYON16, the AUC is improved by 0.0119, the F1 score by 0.0092, and the patch AUC by 0.0399, respectively; on TCGA-NSCLC, the AUC and F1 score are improved by 0.0100 and 0.0215, respectively. Furthermore, by integrating the CAFE module, current instance-based methods, namely Max Pooling, Mean Pooling, and LSE, are also enhanced, further confirming the effectiveness of contextual information. Therefore, the method proposed in this application significantly outperforms existing instance-based and feature-based methods while maintaining a low parameter count of 0.79M and a low computational complexity of 12.43 GFLOPs (floating-point operations).

[0184] The above experimental comparisons show that the Latent Space Smooth Maximization Aggregator (L-LSE) solves the problems of existing instance-based methods and has accurate image patch prediction performance. Combined with context-aware feature extraction (CAFE) module for modeling contextual information, it can effectively reduce false alarms, thereby improving the computer's ability to locate target objects. Figure 6h As shown, Figure 6h The image marked with an A is a WSI image, and the area outlined by the black line in the image (i.e. Figure 6h The area marked with 60 in the middle represents the actual lesion area in the WSI image, while Figure 6h Image B in the image shows the lesion region output after classifying and predicting the WSI image using only the L-LSE algorithm. Figure 6h Image C in the diagram represents the lesion image region output after classifying and predicting the WSI image using the L-LSE algorithm combined with CAFE as described in this embodiment. Figure 6h In images B and C, darker colors indicate a higher likelihood of a lesion at the corresponding location. Therefore, combining... Figure 6h Figures B and C in the document also demonstrate that by using the L-LSE algorithm combined with the CAFE algorithm in this embodiment, the location of lesions in WSI images can be more accurately determined, thereby improving the accuracy of the output lesion images.

[0185] In this embodiment, the computer device can employ a weakly supervised classification method based on latent space smooth maximization. By using a smooth maximization aggregator in the latent space to obtain block classification results for each image patch in the pathological image (i.e., the target image), and then aggregating these block classification results to obtain the image classification result for the pathological image, the contribution of normal image patches to pathological image classification prediction can be effectively suppressed, thus improving the accuracy of the computer device in classifying and predicting the target image. Furthermore, to enable the computer device to effectively model the neighborhood information of image patches, a context-aware feature extraction module is introduced. This module can model the micro-environmental relationships of adjacent image patches, providing contextual information for image patch classification prediction. Therefore, by incorporating contextual image information and the predictive contribution of normal image patches to pathological image classification into the features of image patches, the computer device can effectively improve the accuracy and reliability of image classification and prediction processing.

[0186] Based on the description of the above image processing method embodiments, this invention also proposes an image processing apparatus, which can be a computer program (including program code) running on the aforementioned computer device. This image processing apparatus can be used to perform tasks such as... Figure 2 and Figure 4 For the image processing method described above, please refer to [link / reference]. Figure 7 The image processing apparatus includes an acquisition unit 701 and a processing unit 702.

[0187] Acquisition unit 701 is used to acquire a target image, the target image containing a target object, and the target image being divided into multiple image blocks;

[0188] Processing unit 702 is used to perform feature extraction processing on the target image to obtain image features of the target image; the image features include block features of each image patch in the latent space;

[0189] The processing unit 702 is further configured to perform classification prediction processing on the corresponding image block in the latent space according to the block features of each image block, so as to obtain the block classification result of each image block;

[0190] The processing unit 702 is further configured to perform aggregation processing on the block classification results of each image block in the latent space, and generate an image classification result of the target image based on the aggregation processing result; wherein, the aggregation processing in the latent space is used to adjust the contribution of the block classification results of different image blocks in the generation of the image classification result, and the image classification result is used to indicate the probability that the target object in the target image is an abnormal object.

[0191] In one embodiment, when the processing unit 702 performs feature extraction processing on the target image to obtain the image features of the target image, it is specifically used for:

[0192] Multiple image blocks obtained after dividing the target image are acquired, and the positional relationship between any two image blocks is determined based on the position of each image block in the target image.

[0193] Extract the initial block features of each image block and determine the relationship features corresponding to the positional relationship;

[0194] Based on the relational features, the initial block features of each image block are perceptually fused to obtain perceptual block features, and the obtained perceptual block features are used as the image features of the target image.

[0195] In one embodiment, the relational features corresponding to the positional relationships include an adjacency matrix, and the perceptual fusion processing is performed by calling a context-aware block feature extraction module, which includes a graph convolutional network and a rectified network; when the processing unit 702 performs perceptual fusion processing on the initial block features of each image block according to the relational features to obtain perceptual block features, it is specifically used for:

[0196] The initial block features and the adjacency matrix of each image block are input into the first graph convolutional network in the context-aware block feature extraction module to obtain the first intermediate block features;

[0197] The first rectified network in the context-aware block feature extraction module is invoked to perform linear rectification on the first intermediate block features to obtain the second intermediate block features of each image block.

[0198] The second graph convolutional network and the second rectified network in the context-aware block feature extraction module are invoked to identify the second intermediate block features and the adjacency matrix of each image block, and the identified block features are used as the perceptual block features.

[0199] In one embodiment, the input features of the graph convolutional network include initial block features of each image block and intermediate block features of each image block. When the processing unit 702 uses the graph convolutional network to perform recognition processing on the input features and the adjacency matrix, the graph convolutional network is specifically used for:

[0200] Based on the adjacency matrix and the input features, determine the sum of the input features corresponding to each image patch in the target image;

[0201] The degree matrix corresponding to the positional relationship is determined based on the adjacency matrix;

[0202] The adjacency matrix is ​​scaled by row and column using the degree matrix to obtain the feature mean corresponding to the feature sum; the feature mean is used to generate the recognition processing result of the graph convolutional network.

[0203] In one embodiment, the processing unit 702 is further configured to perform foreground segmentation processing on the target image to extract the corresponding foreground image region from the target image;

[0204] The processing unit 702 is further configured to enlarge the foreground image region and crop the enlarged foreground image region according to the target pixel size to obtain multiple cropped image regions; wherein, the cropped image region is an image block of the target image.

[0205] In one embodiment, the relational feature corresponding to the positional relationship includes an adjacency matrix, which contains multiple elements, any one of which describes the positional relationship between any two image patches of the target image; when determining the relational feature corresponding to the positional relationship, the processing unit 702 is specifically used to:

[0206] Obtain the first position of any image patch in the target image, and based on the first position, obtain the associated image patches that have an eight-adjacent relationship with any image patch;

[0207] The elements in the adjacency matrix that describe the positional relationship between any image block and the associated image block are set to 1, while the elements that describe the positional relationship between any image block and other image blocks are set to 0.

[0208] In one embodiment, when the processing unit 702 aggregates the block classification results of each image block in the latent space and generates the image classification result of the target image based on the aggregation result, it is specifically used for:

[0209] The block classification results of each image patch in the latent space are aggregated to obtain the classification prediction result of the target image in the latent space.

[0210] The classification prediction result of the target image in the latent space is transformed to obtain the image classification result of the target image in the probability space.

[0211] In one embodiment, when the processing unit 702 aggregates the block classification results of each image block in the latent space to obtain the classification prediction result of the target image in the latent space, it is specifically used for:

[0212] In the latent space, an exponential operation is performed based on the block classification result of each image block, and the exponential operation results corresponding to each image block are summed to obtain the parameter exponential sum;

[0213] Logarithmic operations are performed on the exponents of the parameters in the latent space, and the result of the logarithmic operation is used as the classification prediction result of the target image in the latent space.

[0214] In one embodiment, the image classification result includes the anomaly type of the target object in the target image and the probability of being the anomaly type, wherein there are one or more anomaly types, and one anomaly type corresponds to one probability;

[0215] The processing unit 702 is further configured to select the maximum probability from one or more probabilities contained in the image classification result, and determine the anomaly type corresponding to the maximum probability.

[0216] The processing unit 702 is further configured to take the anomaly type corresponding to the maximum probability as the anomaly type of the target object in the target image, and output the anomaly type.

[0217] In one embodiment, the processing unit 702 is further configured to obtain the logical value of the block classification result of each image block during the process of aggregating the block classification results of each image block in the latent space;

[0218] The processing unit 702 is further configured to generate a classification heatmap of the target image based on the logical value corresponding to each image block.

[0219] In one embodiment, when the processing unit 702 generates a classification heatmap of the target image based on the logical value corresponding to each image patch, it is specifically used for:

[0220] Based on the value of the logical value corresponding to each image block, the corresponding image block in the target image is colored; wherein, the value of the logical value is positively correlated with the color intensity of the coloring process.

[0221] The colored target image is used as the classification heatmap of the target image.

[0222] In one embodiment, the image classification result is obtained by calling a trained target classification model to perform recognition processing on the target image; the processing unit 702 is further configured to obtain a sample image and a category label of the sample image; the category label is used to indicate the reference probability that the sample object in the sample image is an abnormal object;

[0223] The processing unit 702 is further configured to call the target classification model to perform recognition processing on the sample image, and obtain the image classification result of the sample image, wherein the image classification result of the sample image includes the predicted probability that the sample object in the sample image is the object category;

[0224] The processing unit 702 is further configured to train the target classification model based on the difference between the predicted probability and the reference probability, so as to obtain a trained target classification model.

[0225] In this embodiment of the application, after the acquisition unit 701 acquires the target image containing the target object, the processing unit 702 can perform segmentation processing on the target image to obtain multiple image blocks contained in the target image. After obtaining multiple image blocks, the processing unit 702 can perform feature extraction processing on each image block to obtain the block features of each image block in the latent space used to describe the image features of the target image. After obtaining the block features of each image patch in the latent space of the target image, the processing unit 702 can further perform classification prediction processing on the corresponding image patch based on the block features in the latent space to obtain the block classification result of the corresponding image patch. Since the contribution of normal image patches in multiple image patches can be effectively suppressed when performing aggregation processing on the block prediction results of each image patch in the latent space, the weight of the block classification results of normal image patches in multiple image patches can be effectively adjusted when performing classification prediction on the target image. Therefore, after obtaining the block classification results of each image patch in the latent space, the processing unit 702 can further perform aggregation processing on the block classification results of each image patch in the latent space to obtain the classification prediction result of the target image in the latent space. Based on the determination of the classification prediction result of the target image in the latent space, the processing unit 702 can transform the classification prediction result of the target image in the latent space into the probability space, thereby obtaining the image classification result of the target image. Since the processing unit 702 can effectively adjust the contribution of normal image blocks in the multiple image blocks when aggregating the block classification results of each image block in the latent space, the classification prediction result of the target image obtained by the processing unit 702 is obtained after effectively amplifying the proportion of the block classification results of abnormal image blocks in the multiple image blocks. Therefore, when using the classification prediction result of the target image in the latent space to obtain the final image classification result, it can effectively rely on the abnormal image regions in the target image blocks for prediction processing, thereby improving the accuracy and reliability of the classification prediction processing of the target image.

[0226] Please see Figure 8 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 8 The computer device shown in this embodiment may include: one or more processors 801; one or more input devices 802; one or more output devices 803; and a memory 804. The processors 801, input devices 802, output devices 803, and memory 804 are connected via a bus 805. The memory 804 stores a computer program, which includes program instructions, and the processor 801 executes the program instructions stored in the memory 804.

[0227] The memory 804 may include volatile memory, such as random-access memory (RAM); the memory 804 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 804 may also include a combination of the above types of memory.

[0228] The processor 801 may be a central processing unit (CPU). The processor 801 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. The PLD may be a field-programmable gate array (FPGA), generic array logic (GAL), etc. The processor 801 may also be a combination of the above structures.

[0229] In this embodiment of the invention, the memory 804 is used to store a computer program, the computer program including program instructions, and the processor 801 is used to execute the program instructions stored in the memory 804 to implement the above-mentioned... Figure 2 and Figure 4 The steps of the corresponding method.

[0230] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0231] Acquire a target image, the target image containing a target object, and the target image is divided into multiple image blocks;

[0232] The target image is subjected to feature extraction processing to obtain the image features of the target image; the image features include the block features of each image patch in the latent space;

[0233] In the latent space, the corresponding image blocks are classified and predicted according to the block features of each image block to obtain the block classification result of each image block;

[0234] The block classification results of each image patch are aggregated in the latent space, and the image classification result of the target image is generated based on the aggregation result. The aggregation process in the latent space is used to adjust the contribution of the block classification results of different image patches in the generation of the image classification result. The image classification result is used to indicate the probability that the target object in the target image is an anomalous object.

[0235] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0236] Multiple image blocks obtained after dividing the target image are acquired, and the positional relationship between any two image blocks is determined based on the position of each image block in the target image.

[0237] Extract the initial block features of each image block and determine the relationship features corresponding to the positional relationship;

[0238] Based on the relational features, the initial block features of each image block are perceptually fused to obtain perceptual block features, and the obtained perceptual block features are used as the image features of the target image.

[0239] In one embodiment, the relational features corresponding to the positional relationship include an adjacency matrix, and the perceptual fusion processing is performed by calling a context-aware block feature extraction module, which includes a graph convolutional network and a rectified network; the processor 801 is configured to call the program instructions to execute:

[0240] The initial block features and the adjacency matrix of each image block are input into the first graph convolutional network in the context-aware block feature extraction module to obtain the first intermediate block features;

[0241] The first rectified network in the context-aware block feature extraction module is invoked to perform linear rectification on the first intermediate block features to obtain the second intermediate block features of each image block.

[0242] The second graph convolutional network and the second rectified network in the context-aware block feature extraction module are invoked to identify the second intermediate block features and the adjacency matrix of each image block, and the identified block features are used as the perceptual block features.

[0243] In one embodiment, the input features of the graph convolutional network include initial block features of each image block and intermediate block features of each image block. The graph convolutional network performs recognition processing on the input features and the adjacency matrix in the following manner:

[0244] Based on the adjacency matrix and the input features, determine the sum of the input features corresponding to each image patch in the target image;

[0245] The degree matrix corresponding to the positional relationship is determined based on the adjacency matrix;

[0246] The adjacency matrix is ​​scaled by row and column using the degree matrix to obtain the feature mean corresponding to the feature sum; the feature mean is used to generate the recognition processing result of the graph convolutional network.

[0247] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0248] The target image is subjected to foreground segmentation processing to extract the corresponding foreground image region from the target image;

[0249] The foreground image region is magnified, and the magnified foreground image region is cropped according to the target pixel size to obtain multiple cropped image regions; wherein, the cropped image region is an image block of the target image.

[0250] In one embodiment, the relational feature corresponding to the positional relationship includes an adjacency matrix, which contains multiple elements, any one of which describes the positional relationship between any two image patches of the target image; the processor 801 is configured to call the program instructions to execute:

[0251] Obtain the first position of any image patch in the target image, and based on the first position, obtain the associated image patches that have an eight-adjacent relationship with any image patch;

[0252] The elements in the adjacency matrix that describe the positional relationship between any image block and the associated image block are set to 1, while the elements that describe the positional relationship between any image block and other image blocks are set to 0.

[0253] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0254] The block classification results of each image patch in the latent space are aggregated to obtain the classification prediction result of the target image in the latent space.

[0255] The classification prediction result of the target image in the latent space is transformed to obtain the image classification result of the target image in the probability space.

[0256] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0257] In the latent space, an exponential operation is performed based on the block classification result of each image block, and the exponential operation results corresponding to each image block are summed to obtain the parameter exponential sum;

[0258] Logarithmic operations are performed on the exponents of the parameters in the latent space, and the result of the logarithmic operation is used as the classification prediction result of the target image in the latent space.

[0259] In one embodiment, the image classification result includes the anomaly type of the target object in the target image and the probability of belonging to the anomaly type, wherein there are one or more anomaly types, and each anomaly type corresponds to a probability; the processor 801 is configured to call the program instructions to execute:

[0260] Select the maximum probability from one or more probabilities included in the image classification result, and determine the anomaly type corresponding to the maximum probability;

[0261] The anomaly type corresponding to the maximum probability is taken as the anomaly type of the target object in the target image, and the anomaly type is output.

[0262] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0263] During the process of aggregating the block classification results of each image block in the latent space, the logical value of the block classification result of each image block is obtained;

[0264] A classification heatmap of the target image is generated based on the logical value corresponding to each image block.

[0265] In one embodiment, the processor 801 is configured to invoke the program instructions to execute:

[0266] Based on the value of the logical value corresponding to each image block, the corresponding image block in the target image is colored; wherein, the value of the logical value is positively correlated with the color intensity of the coloring process.

[0267] The colored target image is used as the classification heatmap of the target image.

[0268] In one embodiment, the image classification result is obtained by calling a trained target classification model to perform recognition processing on the target image; the processor 801 is configured to call the program instructions to execute:

[0269] Obtain a sample image and its category label; the category label is used to indicate the reference probability that a sample object in the sample image is an anomalous object.

[0270] The target classification model is invoked to perform recognition processing on the sample image to obtain the image classification result of the sample image, which includes the predicted probability that the sample object in the sample image belongs to the object category;

[0271] Based on the difference between the predicted probability and the reference probability, the target classification model is trained to obtain a trained target classification model.

[0272] Furthermore, it should be noted that an exemplary embodiment of this application also provides a computer-readable storage medium storing a computer program for the aforementioned image processing method. This computer program includes program instructions that, when loaded and executed by one or more processors, can implement the image processing method described in the embodiments, which will not be repeated here. The beneficial effects of employing the same method will also not be repeated here. It is understood that the program instructions can be deployed on one or more computer devices capable of communicating with each other for execution.

[0273] The aforementioned computer-readable storage medium can be an internal storage unit of the image processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory in the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0274] One aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform actions as described in the embodiments of this application. Figure 2 and Figure 4 The method described in [the document / article].

[0275] One aspect of this application provides another computer program product, which includes a computer program or computer instructions that, when executed by a processor, implement the implementation provided in the embodiments of this application. Figure 2 and Figure 4 The steps of the image processing method described above.

[0276] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0277] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0278] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of the invention.

Claims

1. An image processing method, characterized in that, include: Acquire a target image, the target image containing a target object, and the target image is divided into multiple image blocks; The target image is subjected to feature extraction processing to obtain the image features of the target image; the image features include the block features of each image patch in the latent space; In the latent space, the corresponding image blocks are classified and predicted according to the block features of each image block to obtain the block classification result of each image block; The block classification results of each image patch are aggregated in the latent space, and the image classification result of the target image is generated based on the aggregation result. The aggregation in the latent space is used to adjust the contribution of the block classification results of different image patches in the generation of the image classification result. The image classification result is used to indicate the probability that the target object in the target image is an anomalous object.

2. The method as described in claim 1, characterized in that, The step of performing feature extraction processing on the target image to obtain the image features of the target image includes: Multiple image blocks obtained after dividing the target image are acquired, and the positional relationship between any two image blocks is determined based on the position of each image block in the target image. Extract the initial block features of each image block and determine the relationship features corresponding to the positional relationship; Based on the relational features, the initial block features of each image block are perceptually fused to obtain perceptual block features, and the obtained perceptual block features are used as the image features of the target image.

3. The method as described in claim 2, characterized in that, The relational features corresponding to the positional relationships include an adjacency matrix. The perceptual fusion processing is performed by calling a context-aware block feature extraction module, which includes a graph convolutional network and a rectified network. The perceptual fusion processing of the initial block features of each image block based on the relational features to obtain perceptual block features includes: The initial block features and the adjacency matrix of each image block are input into the first graph convolutional network in the context-aware block feature extraction module to obtain the first intermediate block features; The first rectified network in the context-aware block feature extraction module is invoked to perform linear rectification on the first intermediate block features to obtain the second intermediate block features of each image block. The second graph convolutional network and the second rectified network in the context-aware block feature extraction module are invoked to identify the second intermediate block features and the adjacency matrix of each image block, and the identified block features are used as the perceptual block features.

4. The method as described in claim 3, characterized in that, The input features of the graph convolutional network include the initial block features of each image block and the intermediate block features of each image block. The graph convolutional network performs recognition processing on the input features and the adjacency matrix in the following ways: Based on the adjacency matrix and the input features, determine the sum of the input features corresponding to each image patch in the target image; The degree matrix corresponding to the positional relationship is determined based on the adjacency matrix; The adjacency matrix is ​​scaled by row and column using the degree matrix to obtain the feature mean corresponding to the feature sum; the feature mean is used to generate the recognition processing result of the graph convolutional network.

5. The method as described in claim 2, characterized in that, The method further includes: The target image is subjected to foreground segmentation processing to extract the corresponding foreground image region from the target image; The foreground image region is magnified, and the magnified foreground image region is cropped according to the target pixel size to obtain multiple cropped image regions; wherein, the cropped image region is an image block of the target image.

6. The method as described in claim 2, characterized in that, The relational features corresponding to the positional relationship include an adjacency matrix, which contains multiple elements, any one of which is used to describe the positional relationship between any two image blocks of the target image; The determination of the relational features corresponding to the positional relationship includes: Obtain the first position of any image patch in the target image, and based on the first position, obtain the associated image patches that have an eight-adjacent relationship with any image patch; The elements in the adjacency matrix that describe the positional relationship between any image block and the associated image block are set to 1, while the elements that describe the positional relationship between any image block and other image blocks are set to 0.

7. The method as described in claim 1, characterized in that, The step of aggregating the block classification results of each image patch in the latent space and generating the image classification result of the target image based on the aggregation result includes: The block classification results of each image patch in the latent space are aggregated to obtain the classification prediction result of the target image in the latent space. The classification prediction result of the target image in the latent space is transformed to obtain the image classification result of the target image in the probability space.

8. The method as described in claim 7, characterized in that, The step of aggregating the block classification results of each image patch in the latent space to obtain the classification prediction result of the target image in the latent space includes: In the latent space, an exponential operation is performed based on the block classification result of each image block, and the exponential operation results corresponding to each image block are summed to obtain the parameter exponential sum; Logarithmic operations are performed on the exponents of the parameters in the latent space, and the result of the logarithmic operation is used as the classification prediction result of the target image in the latent space.

9. The method as described in claim 1, characterized in that, The image classification result includes the anomaly type of the target object in the target image and the probability of belonging to the anomaly type, wherein there are one or more anomaly types, and each anomaly type corresponds to a probability; the method further includes: Select the maximum probability from one or more probabilities included in the image classification result, and determine the anomaly type corresponding to the maximum probability; The anomaly type corresponding to the maximum probability is taken as the anomaly type of the target object in the target image, and the anomaly type is output.

10. The method as described in claim 1, characterized in that, The method further includes: During the process of aggregating the block classification results of each image block in the latent space, the logical value of the block classification result of each image block is obtained; A classification heatmap of the target image is generated based on the logical value corresponding to each image block.

11. The method as described in claim 10, characterized in that, The step of generating a classification heatmap of the target image based on the logical value corresponding to each image patch includes: Based on the value of the logical value corresponding to each image block, the corresponding image block in the target image is colored; wherein, the value of the logical value is positively correlated with the color intensity of the coloring process. The colored target image is used as the classification heatmap of the target image.

12. The method as described in claim 1, characterized in that, The image classification result is obtained by calling a trained target classification model to recognize the target image; the method further includes: Obtain a sample image and its category label; the category label is used to indicate the reference probability that a sample object in the sample image is an anomalous object. The target classification model is invoked to perform recognition processing on the sample image to obtain the image classification result of the sample image, which includes the predicted probability that the sample object in the sample image belongs to the object category; Based on the difference between the predicted probability and the reference probability, the target classification model is trained to obtain a trained target classification model.

13. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire a target image, the target image containing a target object, and the target image being divided into multiple image blocks; The processing unit is used to perform feature extraction processing on the target image to obtain image features of the target image; the image features include block features of each image patch in the latent space; The processing unit is further configured to perform classification prediction processing on the corresponding image block in the latent space according to the block features of each image block, so as to obtain the block classification result of each image block; The processing unit is further configured to perform aggregation processing on the block classification results of each image block in the latent space, and generate an image classification result of the target image based on the aggregation processing result; wherein, the aggregation processing in the latent space is used to adjust the contribution of the block classification results of different image blocks in the generation of the image classification result, and the image classification result is used to indicate the probability that the target object in the target image is an abnormal object.

14. A computer device, characterized in that, The system includes a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which are executed by a processor using the image processing method as described in any one of claims 1 to 12.

Images