Image classification method and device, computer device and storage medium

Abstract

This application discloses an image classification method, apparatus, computer device, and storage medium, belonging to the field of computer technology. The method includes: acquiring image features of a pathological image to be classified; for each of multiple scales, extracting at least one local feature corresponding to that scale from the image features; aggregating the at least one local feature corresponding to that scale to obtain aggregated features; stitching together the obtained aggregated features to obtain stitched image features; and classifying the stitched image features to obtain the category to which the pathological image belongs. The method provided in this application utilizes the fact that local features corresponding to different scales contain different information, resulting in multiple aggregated features containing feature information corresponding to different scales. After stitching together multiple aggregated features into stitched image features, the feature information of the stitched image features is enriched. Based on these stitched image features, the category to which the pathological image belongs is determined, ensuring the accuracy of the category.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image classification method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of computer technology, image classification technology is being applied more and more widely in various scenarios, such as face recognition and medical image classification. Related technologies involve extracting features from the image to be classified, obtaining image features, and then directly classifying these features to determine the image's category. However, because the above methods directly classify based on the extracted image features, which contain limited information, the classification accuracy is poor. Summary of the Invention

[0003] This application provides an image classification method, apparatus, computer device, and storage medium, which can improve classification accuracy. The technical solution is as follows:

[0004] On the one hand, an image classification method is provided, the method comprising:

[0005] Obtain image features of the pathological images to be classified;

[0006] For each of the multiple scales, at least one local feature corresponding to the scale is extracted from the image features, and the at least one local feature corresponding to the scale is aggregated to obtain aggregated features;

[0007] The obtained multiple aggregated features are concatenated to obtain the concatenated image features;

[0008] The features of the stitched image are classified to determine the category to which the pathological image belongs.

[0009] In one possible implementation, acquiring the image features of the pathological image to be classified includes:

[0010] The pathological image is segmented to obtain multiple sub-images;

[0011] Feature extraction is performed on each sub-image to obtain the image features of each sub-image;

[0012] Based on the positions of the multiple sub-images in the pathological image, the image features of the multiple sub-images are stitched together to obtain the image features of the pathological image.

[0013] On the other hand, an image classification apparatus is provided, the apparatus comprising:

[0014] The acquisition module is used to acquire image features of the pathological images to be classified;

[0015] An aggregation module is used to extract at least one local feature corresponding to each scale from the image features for each scale among multiple scales, and to aggregate the at least one local feature corresponding to the scale to obtain aggregated features;

[0016] The stitching module is used to stitch together multiple aggregated features to obtain stitched image features;

[0017] The classification module is used to classify the features of the stitched image to obtain the category to which the pathological image belongs.

[0018] In one possible implementation, the stitched image features include first sub-features located at multiple positions; the classification module includes:

[0019] The update unit is used to update each first sub-feature based on multiple first sub-features to obtain the first updated feature corresponding to each first sub-feature;

[0020] The constituting unit is used to construct the updated stitched image features by combining the first updated features corresponding to the plurality of first sub-features based on their positions in the stitched image features.

[0021] A classification unit is used to classify the updated stitched image features to obtain the category to which the pathological image belongs.

[0022] In another possible implementation, the updating unit is configured to obtain the weights of the target sub-feature among the plurality of first sub-features, wherein the weights indicate the degree of correlation between the corresponding first sub-feature and the target sub-feature; and to perform weighted fusion of the plurality of first sub-features based on the weights of the plurality of first sub-features to obtain the first updated feature corresponding to the target sub-feature.

[0023] In another possible implementation, the updating unit is configured to, for the target sub-feature among the plurality of first sub-features, obtain a distance feature between each first sub-feature and the target sub-feature, the distance feature indicating the distance between the positions of the first sub-feature and the target sub-feature in the stitched image features; and obtain a weight for each first sub-feature based on the similarity between each first sub-feature and the target sub-feature and the distance feature.

[0024] In another possible implementation, the updating unit is configured to determine the target distance corresponding to each first sub-feature based on the position of each first sub-feature and the target sub-feature in the stitched image features, wherein the target distance represents the distance between the positions of the first sub-feature and the target sub-feature; and to perform mapping processing on the target distance corresponding to each first sub-feature to obtain the distance feature between each first sub-feature and the target sub-feature.

[0025] In another possible implementation, the first updated feature is a vector; the constituting unit is used to construct a three-dimensional feature matrix based on the positions of the plurality of first sub-features in the stitched image features, and to determine the three-dimensional feature matrix as the updated stitched image features.

[0026] In another possible implementation, the stitched image features include first sub-features located at multiple locations; the apparatus further includes:

[0027] The fusion module is used to fuse each first sub-feature with its corresponding position feature to obtain a second updated feature corresponding to each first sub-feature, wherein the position feature indicates the position of the corresponding first sub-feature in the stitched image feature;

[0028] The composition module is used to construct updated stitched image features by combining the second updated features corresponding to the multiple first sub-features based on their positions in the stitched image features.

[0029] In another possible implementation, the second updated feature is a vector; the constituting module is used to construct a three-dimensional feature matrix based on the positions of the plurality of first sub-features in the stitched image features, and to determine the three-dimensional feature matrix as the updated stitched image features.

[0030] In another possible implementation, the local features include at least one second sub-feature located at at least one position; the aggregation module is used to concatenate the second sub-features in each extracted local feature to obtain a first feature vector corresponding to each local feature; based on the first feature vectors corresponding to multiple local features, each first feature vector is updated to obtain a second feature vector corresponding to each first feature vector; based on the positions of the multiple local features in the image features, the multiple second feature vectors are used to form a three-dimensional feature matrix, and the three-dimensional feature matrix is ​​determined as the aggregated feature.

[0031] In another possible implementation, the acquisition module is used to segment the pathological image to obtain multiple sub-images; extract features from each sub-image to obtain image features of each sub-image; and stitch together the image features of the multiple sub-images based on their positions in the pathological image to obtain the image features of the pathological image.

[0032] In another possible implementation, the multiple aggregated features have the same feature size; the stitching module is used to stitch together features at the same position among the multiple aggregated features to obtain feature vectors corresponding to multiple positions; the three-dimensional feature matrix formed by the feature vectors corresponding to the multiple positions is determined as the stitched image feature.

[0033] In another possible implementation, the steps of acquiring image features of the pathological image to be classified; extracting at least one local feature corresponding to each scale from the image features for each scale; aggregating the at least one local feature corresponding to the scale to obtain aggregated features; stitching together the obtained aggregated features to obtain stitched image features; and classifying the stitched image features to obtain the category to which the pathological image belongs are implemented based on a classification model.

[0034] In another possible implementation, the classification model includes a feature extraction sub-model and a classification sub-model; the apparatus further includes:

[0035] The acquisition module is also used to acquire sample pathological images and sample labels, wherein the sample labels indicate the category to which the sample pathological images belong;

[0036] The acquisition module is further configured to acquire sample image features of the sample pathological image based on the feature extraction sub-model;

[0037] The prediction module is configured to, based on the classification sub-model, extract at least one local feature corresponding to each scale from the sample image features for each of the multiple scales, aggregate the at least one local feature corresponding to the scale to obtain aggregated sample features, stitch together the obtained aggregated sample image features to obtain stitched sample image features, classify the stitched sample image features to obtain a predicted label to which the sample pathological image belongs, and the predicted label indicates the predicted category to which the sample pathological image belongs.

[0038] The training module is used to train the classification model based on the predicted label and the sample label.

[0039] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the image classification method as described above.

[0040] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the image classification method as described above.

[0041] In another aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the operations performed by the image classification method described above.

[0042] The method, apparatus, computer equipment, and storage medium provided in this application, after acquiring the image features of a pathological image, process the image features of the pathological image according to multiple scales to obtain aggregated features corresponding to multiple scales. Then, the aggregated features corresponding to multiple scales are stitched together into a single stitched image feature. This stitched image feature is then classified to obtain the category of the pathological image. Since the aggregated features corresponding to different scales are obtained by aggregating local features at different scales, and the information contained in the local features at different scales is different, multiple aggregated features contain feature information corresponding to different scales. After stitching multiple aggregated features into a stitched image feature, the stitched image feature contains feature information corresponding to different scales, enriching the feature information of the stitched image feature. Based on this stitched image feature, the category to which the pathological image belongs is determined to ensure the accuracy of the category. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the structure of an implementation environment provided in an embodiment of this application;

[0045] Figure 2 This is a flowchart of an image classification method provided in an embodiment of this application;

[0046] Figure 3 This is a flowchart of an image classification method provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating the extraction of local features according to an embodiment of this application;

[0048] Figure 5 This is a flowchart provided in an embodiment of this application for obtaining any aggregation feature;

[0049] Figure 6 This is a flowchart illustrating how to obtain the first updated feature corresponding to a target sub-feature, as provided in an embodiment of this application.

[0050] Figure 7 This is a flowchart illustrating image classification based on a classification model, provided in an embodiment of this application.

[0051] Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0054] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0056] The terms "first," "second," etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first eigenvector can be referred to as a second eigenvector, and similarly, a second eigenvector can be referred to as a first eigenvector.

[0057] As used in this application, the terms "at least one", "multiple", "each", and "any" mean that at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any means refers to any one of the multiple. For example, multiple scales include three scales, where each refers to each of the three scales, and any means refers to any one of the three scales, which could be the first, the second, or the third.

[0058] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0059] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0060] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0061] The solution provided in this application embodiment is based on artificial intelligence machine learning technology to train a classification model, and uses the trained classification model to classify pathological images.

[0062] The image classification method provided in this application is executed by a computer device. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, and vehicle terminal, but is not limited to these.

[0063] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0064] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes terminal 101 and server 102. Terminal 101 and server 102 are directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0065] Terminal 101 is used to acquire pathological images and send them to server 102. Server 102 is used to classify the pathological images sent by terminal 101 to determine the category to which the pathological images belong.

[0066] In some embodiments, a target application provided by a server 102 is installed on the terminal 101, enabling the terminal 101 to perform functions such as data transmission and message interaction. Optionally, the target application is a target application in the terminal 101's operating system or a target application provided by a third party. For example, the target application is an image classification application that has image classification functionality. Of course, the image classification application may also have other functions, such as review functionality and navigation functionality.

[0067] Terminal 101 is used to acquire pathological images, log in to a target application based on a user identifier, and send the pathological images to server 102 through the target application. Server 102 is used to receive the pathological images sent by terminal 101, classify the pathological images sent by terminal 101, and obtain the category to which the pathological images belong. Optionally, server 102 sends the classification result to terminal 101, which indicates the category to which the pathological images belong. Terminal 101 receives the classification result sent by server 102 and displays it.

[0068] Figure 2 This is a flowchart of an image classification method provided in an embodiment of this application, executed by a computer device, such as... Figure 2 As shown, the method includes:

[0069] 201. Computer equipment acquires image features of pathological images to be classified.

[0070] Pathological images are used to present the pathological morphology of organs, tissues, or cells in an organism. For example, a pathological image is a WSI (Whole Slide Image, digital pathological image). Image features of a pathological image are used to characterize the image. These image features can be two-dimensional or three-dimensional and can be represented in any form. For example, the image features can be represented as a feature matrix or as a feature vector. Furthermore, the image features can be represented as a three-dimensional feature matrix or a two-dimensional feature matrix.

[0071] 202. For each of multiple scales, the computer device extracts at least one local feature corresponding to that scale from the image features, and aggregates the at least one local feature corresponding to that scale to obtain aggregated features.

[0072] Here, the scale is used to determine the feature size of the local features extracted from the image features. Multiple scales have different sizes; for example, multiple scales can be two-dimensional, such as 3×3, 5×5, or 7×7. Each local feature is a partial feature within the image features. Any local feature corresponding to any scale indicates that the feature size of that local feature matches that scale, meaning that the local feature was extracted from the image features based on that scale. For any scale among the multiple scales, at least one local feature corresponding to that scale is extracted from the image features, and these at least one local feature are aggregated to obtain an aggregated feature. Following this method, an aggregated feature can be obtained based on each of the multiple scales, thus obtaining multiple aggregated features.

[0073] 203. The computer device stitches together the multiple aggregated features to obtain stitched image features.

[0074] The multiple aggregated features are obtained based on multiple scales. Different aggregated features contain feature information corresponding to different scales. By stitching these multiple aggregated features together, it can be ensured that the stitched image features contain feature information corresponding to multiple scales.

[0075] 204. The computer equipment classifies the features of the spliced ​​image to obtain the category to which the pathological image belongs.

[0076] The method provided in this application, after acquiring the image features of a pathological image, processes the image features of the pathological image according to multiple scales to obtain aggregated features corresponding to multiple scales. Then, the aggregated features corresponding to multiple scales are stitched together into a single stitched image feature. This stitched image feature is then classified to obtain the category of the pathological image. Since the aggregated features corresponding to different scales are obtained by aggregating local features at different scales, and the information contained in the local features at different scales is different, multiple aggregated features contain feature information corresponding to different scales. After stitching multiple aggregated features into a stitched image feature, the stitched image feature contains feature information corresponding to different scales, enriching the feature information of the stitched image feature. Based on this stitched image feature, the category to which the pathological image belongs is determined to ensure the accuracy of the category.

[0077] exist Figure 2 Based on the illustrated embodiment, the image features of the pathological image can also be obtained by segmenting and then stitching together the images. A self-attention mechanism is adopted to obtain multiple aggregated features and update the obtained stitched image features. Then, the category to which the pathological image belongs is obtained based on the updated stitched image features. For details of the process, please refer to the following embodiment.

[0078] Figure 3 This is a flowchart of an image classification method provided in an embodiment of this application, executed by a computer device, such as... Figure 3 As shown, the method includes:

[0079] 301. Computer equipment segments the pathological images to be classified to obtain multiple sub-images.

[0080] Each sub-image is a portion of the pathological image, and multiple sub-images can be combined to form the pathological image. Optionally, the multiple sub-images have the same image size. For example, if the pathological image has an image size of 40000×40000 and each sub-image has a size of 4000×4000, then after segmenting the pathological image, 100 sub-images are obtained.

[0081] In one possible implementation, step 301 includes: segmenting the pathological image based on the target size to obtain multiple sub-images. Each sub-image has the same size as the target size. For example, if the target size is 1120×1120, then each segmented sub-image will have a size of 1120×1120.

[0082] Optionally, the process of segmenting the pathological image includes: moving a first sliding window on the pathological image based on a first moving step size, and determining a portion of the pathological image within the first sliding window after each movement as a sub-image.

[0083] In this design, the size of the first sliding window is the same as the target size, and the first movement step represents the distance the first sliding window moves each time. For example, the first movement step represents the number of pixels moved each time the first sliding window is moved. Optionally, the target size includes a first dimension and a second dimension, which represent the length and width of the first sliding window, respectively. The first movement step is the same as the first dimension and the second dimension. For example, if the target size is 1120×1120 and the first movement step is 1120, multiple sub-images in the pathological image are obtained according to this first movement step and the target size, and each sub-image has a size of 1120×1120.

[0084] Optionally, the pathological image is represented in the form of a three-dimensional feature matrix, and each segmented sub-image is also represented in the form of a three-dimensional feature matrix. These multiple sub-images can be stitched together to form the pathological image. Then, the pathological image and the multiple sub-images satisfy the following relationship:

[0085]

[0086]

[0087]

[0088]

[0089] in, Representing pathological images, The image size of the pathological image is indicated as follows: , , Indicates the length and width of the pathological image. , It can be any positive integer; , , Each represents one of a set of sub-images. Indicates the image size of each sub-image as , , This represents the length and width of each sub-image. , All are arbitrary positive integers, and Less than , Less than ; Indicates the sequence number of multiple sub-images. Greater than 0 is not significant integers, greater than 0 and not greater than Positive integers; To determine the image size of the sub-image Image size in pathological images The number of segments above To determine the image size of the sub-image Image size of pathological images The number of segments above This represents the total number of sub-images segmented from a pathological image.

[0090] 302. The computer equipment extracts features from each sub-image to obtain the image features of each sub-image.

[0091] Each sub-image's image features are used to characterize the corresponding sub-image. These image features can be two-dimensional or three-dimensional and can be represented in any form. For example, the image features of a sub-image can be represented as a feature vector or a feature matrix. Alternatively, the image features of a sub-image can be represented as a two-dimensional feature matrix or a three-dimensional feature matrix.

[0092] In one possible implementation, each sub-image and its corresponding image features satisfy the following relationship:

[0093]

[0094] in, Represents any one of multiple sub-images. Indicates the sequence number of multiple sub-images. Represents the first sub-image among multiple sub-images Sub-images; This represents the feature extraction function, based on which... It can extract features from each sub-image separately to obtain the image features of each sub-image; Representing image features for 3D eigenvectors.

[0095] 303. The computer device stitches together the image features of the multiple sub-images based on their positions in the pathological image to obtain the image features of the pathological image.

[0096] In this context, the image features of a pathological image are used to characterize the image, and these image features can be represented in any form. For example, if the image features of multiple sub-images are all two-dimensional feature matrices, then the image features of the pathological image are also two-dimensional feature matrices; if the image features of multiple sub-images are all three-dimensional feature matrices, then the image features of the pathological image are also three-dimensional feature matrices.

[0097] Since different sub-images are located in different positions within the pathological image, the image features of multiple sub-images are stitched together based on their respective positions to obtain the image features of the pathological image. This ensures that the image features of the pathological image correspond to the pathological image itself, and the image features of the pathological image retain the positional information of each sub-image, thus guaranteeing the accuracy of the image features. Furthermore, by first segmenting the pathological image into multiple small sub-images, obtaining the image features of each sub-image, and then stitching them together to form the image features of the pathological image, there is no need to directly extract features from the large-sized pathological image. This reduces the demands on equipment performance and the requirements for image classification equipment, ensuring the versatility of this feature extraction method.

[0098] Optionally, when the image features of multiple sub-images are all two-dimensional feature matrices, the feature size of the image features of the pathological image is the sum of the feature sizes of the image features of the multiple sub-images. For example, if a pathological image includes 4 sub-images, and the feature size of the image features of each sub-image is 2×2, then the feature size of the image features of the pathological image is 8×8.

[0099] Optionally, when the image features of multiple sub-images are all three-dimensional feature matrices, the feature dimensions of the image features of the sub-images include a first dimension, a second dimension, and a third dimension, and the feature dimensions of the image features of the pathological image include a fourth dimension, a fifth dimension, and a sixth dimension. The fourth dimension is the sum of the first dimensions corresponding to the multiple sub-images, the fifth dimension is the sum of the second dimensions corresponding to the multiple sub-images, and the sixth dimension is the same as the third dimension. For example, if a pathological image includes four sub-images, and the feature dimension of the image features of each sub-image is 2×2×10, then the feature dimension of the image features of the pathological image is 8×8×10.

[0100] Optionally, the image features of multiple sub-images and the image features of the pathological image satisfy the following relationship:

[0101]

[0102] in, Image features representing pathological images, , , Each represents an image feature of one of a plurality of sub-images; Indicates the sequence number of multiple sub-images. greater than 0 and not greater than integers, greater than 0 and not greater than Positive integers; To be based on the feature size of the sub-image In pathological images The number of segments in the dimension To be based on the feature size of the sub-image In pathological images The number of segments in the dimension This represents the total number of sub-images segmented from a pathological image; This represents the vector dimension of the image features of each sub-image, assuming the image features of each sub-image are vectors.

[0103] It should be noted that in this embodiment, the pathological image is divided into multiple sub-images by first segmenting and then stitching, and the image features of the pathological image are obtained based on the multiple sub-images. In another embodiment, steps 301-303 do not need to be performed, and other methods can be used to obtain the image features of the pathological image to be classified.

[0104] 304. For each of the multiple scales, the computer device extracts multiple local features corresponding to that scale from the image features of the pathological image, and concatenates the second sub-features in each of the extracted local features to obtain the first feature vector corresponding to each local feature.

[0105] In this embodiment, the image features of the pathological image include second sub-features located at multiple positions. These second sub-features are one-dimensional or two-dimensional features. For example, a second sub-feature may include a single feature value, or it may be a feature vector composed of multiple feature values. When the image features of the pathological image are two-dimensional, the second sub-feature includes a single feature value; when the image features of the pathological image are three-dimensional, the second sub-feature includes multiple feature values, i.e., it is a feature vector composed of multiple feature values. For example, the feature size of the image features of the pathological image is 3×3×10, and the image features include second sub-features at 9 positions, each second sub-feature being a feature vector composed of 10 feature values.

[0106] For any of multiple scales, at least one local feature is extracted from the image features of the pathological image based on that scale. Each local feature includes a second sub-feature located at at least one location. Optionally, any scale corresponds to multiple local features, each local feature including second sub-features located at multiple locations. Different local features may include the same second sub-feature. The image features of the pathological image include second sub-features located at 3×3 locations, such as... Figure 4 As shown, each box represents a position; the scale is 2×2. Based on this scale, four local features 402 are extracted from the image features 401 of the pathological image. Each local feature 402 includes a second sub-feature located at four positions. That is, in the image features of the pathological image, the second sub-features at the four positions in the upper left corner constitute a local feature 402, the second sub-features at the four positions in the upper right corner constitute a local feature 402, the second sub-features at the four positions in the lower left corner constitute a local feature 402, and the second sub-features at the four positions in the lower right corner constitute a local feature 402. The local feature 402 at the upper left corner and the local feature 402 at the upper right corner include two identical second sub-features.

[0107] For any local feature, the second sub-features in the local feature are concatenated into a first feature vector. The feature size of the first feature vector is the sum of the feature sizes of at least one second sub-feature in the local feature. For example, if any local feature includes four second sub-features, each with a feature size of 1×3, the four second sub-features are concatenated into a first feature vector, and the size of the first feature vector corresponding to the local feature is 1×12.

[0108] In one possible implementation, the process of extracting local features corresponding to any scale includes: moving a second sliding window corresponding to the scale on the image features of the pathological image based on a second moving step size, and constructing a local feature from the second sub-features at at least one position within the second sliding window after each movement.

[0109] The size of the second sliding window corresponding to this scale is the same as the scale itself. Optionally, the scale is the side length of the second sliding window. For example, if the scale is one-dimensional, the length and width of the second sliding window are the same as the scale; if the scale is 3, the size of the second sliding window is 3×3. As another example, if the scale is two-dimensional, including length and width, the length and width of the second sliding window corresponding to this scale are equal to the length and width included in the two-dimensional scale, respectively. The second movement step size represents the distance the second sliding window moves each time. For example, the second movement step size represents the number of positions moved each time the second sliding window is moved; if the second movement step size is 1, then each time the second sliding window is moved, it only needs to move one position. Figure 4As shown, the size of the second sliding window is 2×2, the second moving step is 1, and each extracted local feature 402 includes a second sub-feature located at 4 positions.

[0110] Optionally, the process of extracting local features corresponding to any scale includes: determining the augmentation parameter corresponding to the scale, augmenting the image features of the pathological image based on the augmentation parameter to obtain the augmented image features, moving the second sliding window corresponding to the scale on the image features of the pathological image based on the second moving step size, and constructing a local feature by taking the second sub-feature at at least one position within the second sliding window after each movement.

[0111] The augmentation parameter indicates the number of positions in the image features of the pathological image to be augmented. For example, the augmentation parameter is Padding. For any given scale, the image features of the pathological image are augmented by the number of positions indicated by the augmentation parameter, thus increasing the feature size of the augmented image features. For example, after augmenting the image features of a pathological image, both the length and width of the augmented image features are increased by twice the number of positions indicated by the augmentation parameter. For instance, if the image features of a pathological image are a 3×3 feature matrix, and the augmentation parameter for any scale is 1, then the augmented image features are a 5×5 feature matrix; if the augmentation parameter for any scale is 2, then the augmented image features are a 7×7 feature matrix.

[0112] Optionally, any scale is a one-dimensional scale. The process of determining the augmentation parameter corresponding to the scale includes: determining the difference between the scale and the first value, and determining the ratio of the difference to the second value as the augmentation parameter corresponding to the scale.

[0113] In this case, the first and second values ​​are arbitrary values, for example, the first value is 1 and the second value is 2.

[0114] Optionally, any scale and its corresponding augmentation parameters satisfy the following relationship:

[0115]

[0116] in, Indicates scale. Representing scale In this embodiment of the application, the corresponding extended parameters are: the first value is 1 and the second value is 2.

[0117] For example, the feature size of the image features in a pathological image is 4×4. The first scale is 3, meaning the size of the second sliding window corresponding to the first scale is 3×3. The first expansion parameter corresponding to the first scale is 1, meaning the feature size of the image features expanded based on the first expansion parameter is 6×6. Based on the second sliding window corresponding to the first scale, 16 local features with a feature size of 3×3 can be extracted from it. The second scale is 5, meaning the size of the second sliding window corresponding to the second scale is 5×5. The second expansion parameter corresponding to the second scale is 2, meaning the feature size of the image features expanded based on the second expansion parameter is 8×8. Based on the second sliding window corresponding to the second scale, 16 local features with a feature size of 5×5 can be extracted from it. That is, 16 local features are extracted based on both the first and second scales.

[0118] When extracting local features from the image features of a pathological image based on each scale, the image features of the pathological image are first expanded based on the expansion parameters corresponding to each scale to ensure that the number of local features extracted based on each scale is the same, so as to facilitate the fusion of features obtained based on multiple scales in the future.

[0119] In one possible implementation, for any of the multiple scales, after obtaining the first eigenvector corresponding to each local feature of that scale, the multiple first eigenvectors are used to form the first feature matrix corresponding to that scale.

[0120] Optionally, for any one of the multiple scales, the first feature matrix corresponding to that scale and the image features of the pathological image satisfy the following relationship:

[0121]

[0122]

[0123]

[0124] in, This represents the first feature matrix corresponding to this scale; Image features representing pathological images, This represents a feature reconstruction function used to reconstruct the image features of pathological images. This constitutes the first characteristic matrix; This represents the concatenation function, used to construct the first feature matrix corresponding to this scale from multiple first feature vectors; This represents the expansion function, which is used to concatenate the second sub-features in the local features corresponding to this scale into a first feature vector; The size of the first feature matrix corresponding to this scale is . , This indicates the scale. It is a positive integer; this scale The feature size of each corresponding local feature is The feature size of the image features in the pathological image is , The length and width of the image features representing pathological images. The dimension representing the second sub-feature in the image features of a pathological image. All are arbitrary positive integers; Indicates according to scale When extracting local features from the image features of a pathological image, the image features of the pathological image... The number of dimensions; Indicates according to scale When extracting local features from the image features of a pathological image, the image features of the pathological image... The number of dimensions; This represents the vector dimension of each second sub-feature; This represents the vector dimension of each first feature vector; Representing scale The number of corresponding local features also represents the scale. The number of corresponding first feature vectors; Representing scale The corresponding extended parameters; This indicates the second movement step size.

[0125] 305. The computer device updates each first feature vector based on the first feature vector corresponding to multiple local features to obtain the second feature vector corresponding to each first feature vector.

[0126] In this embodiment of the application, for any scale among multiple scales, there are multiple local features corresponding to the scale. Considering the correlation between different local features, after obtaining the first feature vectors corresponding to multiple local features, each first feature vector is updated using the first feature vectors corresponding to multiple local features, so that the updated second feature vector incorporates the first feature vectors corresponding to other local features, thereby ensuring the accuracy of the second feature vector.

[0127] In one possible implementation, step 305 includes: for a target feature vector of multiple first feature vectors, obtaining the similarity between each first feature vector and the target feature vector, and performing weighted fusion on the multiple first feature vectors based on the similarity corresponding to each first feature vector to obtain a second feature vector corresponding to the target feature vector.

[0128] Here, the target feature vector is any one of multiple first feature vectors. The similarity between any first feature vector and the target feature vector represents the degree of similarity between the two first feature vectors. This similarity can be obtained using cosine similarity or other similarity acquisition methods. Since the similarity corresponding to each first feature vector can represent the degree of similarity between that first feature vector and the target feature vector, a self-attention mechanism is adopted. Based on the similarity between multiple first feature vectors and the target feature vector, the first feature vector is weighted and fused to obtain the second feature vector corresponding to the target feature vector. This ensures the accuracy of the obtained second feature vector by incorporating multiple first feature vectors according to their similarity. By following the above method for obtaining the second feature vector corresponding to the target feature vector, the second feature vector corresponding to each first feature vector can be obtained.

[0129] Optionally, the process of obtaining the second feature vector corresponding to the target feature vector includes: for the target feature vector of multiple first feature vectors, obtaining the similarity between each first feature vector and the target feature vector; based on the similarity corresponding to each first feature vector, performing weighted fusion on the multiple first feature vectors to obtain a third feature vector; and performing feature transformation on the third feature vector to obtain the second feature vector corresponding to the target feature vector.

[0130] In this embodiment, each first feature vector and its corresponding second feature vector have different feature dimensions. For each of the multiple scales, the feature dimensions of the second feature vectors corresponding to different scales are the same. During the process of updating the multiple first feature vectors corresponding to each scale, the feature vectors are converted into feature vectors with the same feature dimension to ensure that the feature size of the aggregated features obtained subsequently based on each scale is the same.

[0131] Optionally, the process of performing feature transformation on the third feature vector includes: multiplying the third feature vector with the feature transformation matrix to determine the second feature vector corresponding to the target feature vector.

[0132] The feature transformation matrix is ​​used to transform the third feature vector, and this feature transformation matrix is ​​an arbitrary constant matrix.

[0133] In one possible implementation, for any scale among multiple scales, the first eigenvectors corresponding to each local feature of that scale constitute the first feature matrix corresponding to that scale. Then, step 305 includes: updating each first eigenvector in the first feature matrix based on the multiple first eigenvectors in the first feature matrix to obtain the second eigenvectors corresponding to each first eigenvector, and forming a second feature matrix from the multiple second eigenvectors.

[0134] Optionally, for any one of the multiple scales, the first feature matrix and the second feature matrix corresponding to that scale satisfy the following relationship:

[0135]

[0136]

[0137] in, This represents the second feature matrix corresponding to this scale; This represents a multilayer perceptron function, used to perform fully connected transformations on features; Represents a multi-head self-attention function; This represents the first feature matrix corresponding to this scale; the feature size of the image features of the pathological image is... , The length and width of the image features representing pathological images. The dimension representing the second sub-feature in the image features of a pathological image. All are arbitrary positive integers; This indicates that when extracting local features from the image features of a pathological image according to this scale, the image features of the pathological image will be... The number of dimensions; This indicates that when extracting local features from the image features of a pathological image according to this scale, the image features of the pathological image will be... The number of dimensions; Representing scale The number of corresponding local features is also the number of multiple second feature vectors; This represents the vector dimension of each second feature vector; Indicates the join function; This represents the number of heads in the multi-head self-attention function. It is a positive integer greater than 1; This represents the feature output by any head in a multi-head self-attention function. This indicates the index of each head self-attention function in the multi-head self-attention function. greater than 0 and not greater than Positive integers; This represents a mapping matrix, which is also a constant matrix; All are the first The constant matrix in the individual self-attention function; In the multi-head self-attention function, the first... Size self-attention function; This represents the vector dimension of each first eigenvector in the first feature matrix corresponding to this scale.

[0138] 306. Based on the positions of the multiple local features in the image features of the pathological image, the computer device constructs a three-dimensional feature matrix from the multiple second feature vectors, and determines the three-dimensional feature matrix as the aggregate feature.

[0139] In this context, different local features are located at different positions within the image features of a pathological image. The second feature vector corresponding to any local feature represents the feature of the pixel at that location in the pathological image. Optionally, the second feature vector corresponding to any local feature may represent the feature of one or more pixels corresponding to that local feature in the pathological image. For example, if the image features in the pathological image are used to characterize the pathological image, and any local feature in the pathological image represents the feature of pixels in a portion of the pathological image, where that portion of the pathological image contains one or more pixels, then the second feature vector corresponding to that local feature represents the feature of one or more pixels in that portion of the pathological image.

[0140] Based on the positions of multiple local features within the image features of the pathological image, the second feature vectors corresponding to these local features are used to construct a three-dimensional feature matrix, which is a collective feature. For example, if the number of local features at any scale is 4, and the dimension of the second feature vector corresponding to each local feature is 10, the feature size of the collective feature constructed based on the second feature vectors corresponding to these local features is 2×2×10. Following steps 304-306 above, the image features of the pathological image are processed at multiple scales to obtain multiple collective features.

[0141] In the process of processing the image features of the pathological image based on each scale, the second sub-features in each local feature are concatenated into a feature vector. Considering the correlation between different local features, each first feature vector is updated using the first feature vectors corresponding to multiple local features. This ensures the accuracy of the second feature vector by incorporating the first feature vectors corresponding to other local features into the updated second feature vector. Furthermore, according to the position of multiple local features in the image features of the pathological image, the three-dimensional feature matrix composed of the feature vectors corresponding to multiple local features is determined as the aggregated feature. This ensures that the aggregated feature retains the positional information of the original image features of the pathological image, thus guaranteeing the accuracy of the aggregated feature.

[0142] In one possible implementation, the number of local features corresponding to different scales is the same, and the feature dimension of the second feature vector corresponding to different scales is the same. Therefore, the feature size of the aggregated features obtained based on different scales is the same.

[0143] In one possible implementation, for any one of multiple scales, the multiple second feature vectors corresponding to that scale constitute a second feature matrix. Then step 306 includes: based on the position of the multiple local features in the image features of the pathological image, constructing a three-dimensional feature matrix from the multiple second feature vectors in the second feature matrix, and determining the three-dimensional feature matrix as an aggregated feature.

[0144] Optionally, the second feature matrix and the aggregated feature satisfy the following relationship:

[0145]

[0146] in, This represents the aggregated features obtained based on this scale; This represents the reshaping function, used to reshape the second feature matrix, recover spatial location information, and obtain a new feature map; Indicates aggregated features Feature size; the feature size of image features in pathological images is , The length and width of the image features representing pathological images. The dimension representing the second sub-feature in the image features of a pathological image. All are arbitrary positive integers; This indicates that when extracting local features from the image features of a pathological image according to this scale, the image features of the pathological image will be... The number of dimensions; This indicates that when extracting local features from the image features of a pathological image according to this scale, the image features of the pathological image will be... The number of dimensions; This represents the vector dimension of each second feature vector.

[0147] In the process of obtaining aggregated features based on any scale, such as Figure 5 As shown, taking a 4×4 matrix of image features of a pathological image as an example, with any scale of 3, four local features 501 are extracted from the image features of the pathological image. Each local feature 501 is fused into a first feature vector 502. Using a multi-head attention mechanism and a multilayer perceptron, each first feature vector is updated according to the above steps 305-306, and multiple second feature vectors are combined to form an aggregate feature 503.

[0148] It should be noted that the embodiments of this application are illustrated with the example of multiple local features corresponding to each scale. In another embodiment, if any scale corresponds to a local feature, the sub-features in the local feature corresponding to that scale are concatenated to obtain the first feature vector corresponding to that local feature, and the first feature vector is determined as the aggregate feature.

[0149] It should be noted that the embodiments of this application take the example of local features including at least one second sub-feature at at least one location, and obtain multiple aggregated features based on the image features of each scale in multiple scales and the pathological image. In another embodiment, it is not necessary to perform the above steps 304-306. Other methods can be adopted to extract at least one local feature corresponding to each scale from the image features in multiple scales, and aggregate the at least one local feature corresponding to the scale to obtain aggregated features.

[0150] 307. The computer device stitches together the multiple aggregated features to obtain stitched image features.

[0151] In this embodiment of the application, the image features of the pathological image are processed based on multiple scales to obtain multiple aggregated features. Different aggregated features contain feature information corresponding to different scales. The multiple aggregated features are then stitched together so that the stitched image features contain feature information corresponding to multiple scales.

[0152] In one possible implementation, step 307 includes: stitching together features at the same position among the multiple aggregated features to obtain feature vectors corresponding to multiple positions; and determining the three-dimensional feature matrix formed by the feature vectors corresponding to multiple positions as the stitched image feature.

[0153] In this embodiment, the multiple aggregated features have the same feature size, and each aggregated feature includes features located at multiple positions, with each feature at a position being a second feature vector. For different aggregated features, the features at the same position are the features of pixels at the same position in the pathological image. Therefore, the features at the same position in the multiple aggregated features are concatenated to obtain feature vectors corresponding to multiple positions. Based on the multiple positions, the feature vectors corresponding to the multiple positions are used to construct a three-dimensional feature matrix, and this three-dimensional feature matrix is ​​determined as the concatenated image feature.

[0154] For example, multiple scales include 3, each aggregated feature has a feature size of 4×4×10, and the feature size of the stitched image feature obtained by stitching the 3 aggregated features is 4×4×30.

[0155] Optionally, the stitched image features obtained based on multiple aggregation features satisfy the following relationship:

[0156]

[0157]

[0158]

[0159] in, Indicates the features of the stitched image; Indicates the concatenation function; This represents aggregated features obtained based on different scales. Indicates the number of multiple scales; The feature size representing the features of the stitched image is The feature size of the image features in the pathological image is , The length and width of the image features representing pathological images. The dimension representing the second sub-feature in the image features of a pathological image. All are arbitrary positive integers; This indicates that when extracting local features from the image features of a pathological image at each scale, the image features of the pathological image are... The number of dimensions; This indicates that when extracting local features from the image features of a pathological image at each scale, the image features of the pathological image are... The number of dimensions; This represents the feature size of each first sub-feature in the stitched image features; This represents the vector dimension of each second feature vector.

[0160] 308. The computer device updates each of the first sub-features in the stitched image features to obtain the first updated feature corresponding to each first sub-feature.

[0161] The stitched image features include first sub-features located at multiple positions, meaning the stitched image features comprise multiple first sub-features, each containing feature information corresponding to different scales. In this embodiment, the first sub-features at different positions are correlated. A self-attention mechanism is adopted, updating each first sub-feature through multiple first sub-features, so that the updated first feature incorporates other first sub-features, thereby ensuring the accuracy of the updated first feature.

[0162] In one possible implementation, step 308 involves: for the target sub-feature among the plurality of first sub-features, obtaining the weights of the plurality of first sub-features; and based on the weights of the plurality of first sub-features, performing weighted fusion on the plurality of first sub-features to obtain the first updated feature corresponding to the target sub-feature.

[0163] The weight of any first sub-feature indicates the degree of correlation between that first sub-feature and the target sub-feature. A larger weight indicates a stronger correlation between the first sub-feature and the target sub-feature; a smaller weight indicates a weaker correlation. Since the weight of each first sub-feature represents the degree of correlation between it and the target sub-feature, multiple first sub-features are weighted and merged into a first updated feature based on their correlation with the target sub-feature, ensuring the accuracy of the obtained first updated feature. Following the above method for obtaining the first updated feature corresponding to the target sub-feature, the first updated feature corresponding to each first sub-feature can be obtained.

[0164] Optionally, the process of obtaining the weight of each first sub-feature includes: for the target sub-feature among the plurality of first sub-features, obtaining the distance feature between each first sub-feature and the target sub-feature; and obtaining the weight of each first sub-feature based on the similarity and distance features between each first sub-feature and the target sub-feature.

[0165] The distance feature between any first sub-feature and the target sub-feature indicates the distance between the positions of the first sub-feature and the target sub-feature in the stitched image features; the similarity between any first sub-feature and the target sub-feature represents the degree of similarity between the first sub-feature and the target sub-feature.

[0166] In this embodiment, features at different locations in a pathological image may be correlated. For any first sub-feature, the similarity between the first sub-feature and the target sub-feature indicates the degree of similarity between them; while the distance between the first sub-feature and the target sub-feature indicates their distribution in the stitched image features. Based on the similarity and distance between the first sub-feature and the target sub-feature, the weight of the first sub-feature is obtained. This considers not only the similarity between the first sub-feature and the target in the features, but also their relative positional relationship, thus fully taking into account the correlation between the first sub-feature and the target in the features to ensure the accuracy of the obtained weight.

[0167] Optionally, the process of obtaining distance features includes: determining the target distance corresponding to each first sub-feature based on the position of each first sub-feature and the target sub-feature in the stitched image features; and performing mapping processing on the target distance corresponding to each first sub-feature to obtain the distance features between each first sub-feature and the target sub-feature.

[0168] Here, the target distance corresponding to any first sub-feature represents the distance between the location of the first sub-feature and the location of the target sub-feature. The position of each first sub-feature in the stitched image features can be represented in any form, for example, the position of each first sub-feature in the stitched image features can be represented in the form of two-dimensional coordinates. For any first sub-feature, based on the location corresponding to the first sub-feature and the location corresponding to the target sub-feature, the distance between the two locations can be determined, which is the target distance corresponding to the first sub-feature. The target distance corresponding to the first sub-feature is mapped to a distance feature so that the weight of the first sub-feature can be obtained subsequently based on this distance feature.

[0169] Optionally, the process of obtaining the weight of each first sub-feature based on the similarity and distance features between each first sub-feature and the target sub-feature includes: for each first sub-feature among multiple first sub-features, fusing the similarity and distance features between the first sub-feature and the target sub-feature to obtain the first matching degree corresponding to the first sub-feature; determining the sum of the first matching degrees corresponding to multiple first sub-features; and determining the ratio of the first matching degree corresponding to each first sub-feature to the sum as the weight of each first sub-feature.

[0170] Here, the first matching degree corresponding to any first sub-feature indicates the degree of matching between that first sub-feature and the target sub-feature. When obtaining the weight of any sub-feature, the first matching degree between each first sub-feature and the target sub-feature is first obtained. Considering the matching degrees between multiple first sub-features and the target sub-feature, the ratio of the first matching degree corresponding to each first sub-feature to the sum is determined as the weight of each first sub-feature. This ensures that the first sub-feature that matches the target sub-feature more closely has a larger weight, thereby ensuring the accuracy of the weights.

[0171] Optionally, the process of obtaining the first matching degree between any first sub-feature and the target sub-feature includes: fusing the similarity and distance features between the first sub-feature and the target sub-feature to obtain a second matching degree; and performing feature transformation on the second matching degree to obtain the first matching degree corresponding to the first sub-feature.

[0172] By first obtaining the second matching degree and then performing feature transformation on the second matching degree, it is ensured that the first matching degree corresponding to each first sub-feature belongs to the same interval, so that the first matching degree corresponding to multiple first sub-features can be compared in the future.

[0173] Optionally, for the target sub-feature among the plurality of first sub-features, the weight of any first sub-feature satisfies the following relationship:

[0174]

[0175]

[0176]

[0177] ;

[0178]

[0179] Among the multiple first sub-features, the target sub-feature For the first sub-feature in multiple sub-features The first sub-feature; any first sub-feature is the first among multiple first sub-features. First sub-feature ; Indicates the first sub-feature The weights; This represents a feature matrix composed of multiple first sub-features. It is a constant matrix; Indicates the first sub-feature The corresponding first matching degree; This represents an exponential function with the natural constant e as its base. Indicates the first sub-feature With target sub-features The second degree of matching between them; This represents the sum of the first matching degrees corresponding to multiple first sub-features; Indicates the total number of multiple first sub-features; Indicates the first sub-feature among multiple first sub-features First sub-feature and target sub-feature The second degree of matching between them; Indicates the first sub-feature With target sub-features The similarity between them; For any constant; This indicates the transpose of the matrix; Indicates the first sub-feature With target sub-features Distance characteristics between them; This represents a mapping function used to map distance to distance features. Indicates the first sub-feature The corresponding target distance; Representing target sub-features Location in the features of the stitched image Representing target sub-features Two-dimensional coordinates of the position in the features of the stitched image; Indicates the first sub-feature Location in the features of the stitched image; Indicates the first sub-feature Two-dimensional coordinates of the position in the features of the stitched image.

[0180] Optionally, the first updated feature corresponding to the target sub-feature satisfies the following relationship:

[0181]

[0182]

[0183] in, This represents the first updated feature corresponding to the target sub-feature; Indicates the features of the stitched image; All are constant matrices; Indicates the total number of multiple first sub-features; Represents the first of multiple first sub-features The first sub-feature; Indicates the first sub-feature The weights; This indicates that the multiple first sub-features are weighted and fused based on their respective weights. This represents the first updated feature corresponding to the target sub-feature. for dimensional feature vectors It can be any positive integer; Represents a constant matrix The size is , All are arbitrary positive integers. The process of obtaining the first updated feature corresponding to the target sub-feature based on multiple first sub-features in the stitched image features is as follows: Figure 6 As shown.

[0184] 309. Based on the positions of the multiple first sub-features in the stitched image features, the computer device constructs a three-dimensional feature matrix of the first updated features corresponding to the multiple first sub-features, and determines the three-dimensional feature matrix as the updated stitched image features.

[0185] In this system, the first updated feature is a vector, meaning that the first updated feature corresponding to each first sub-feature is a vector. By constructing a three-dimensional feature matrix from the multiple first sub-features according to their positions in the stitched image features, this three-dimensional feature matrix becomes the updated stitched image features. This ensures that the updated stitched image features retain the positional information between different features, thereby guaranteeing the accuracy of the updated stitched image features.

[0186] For example, the feature size of the stitched image features before the update is 3×3×10, and each first sub-feature is a feature vector with a dimension of 10, meaning that the stitched image features before the update include 9 positions; the first update feature corresponding to each first sub-feature is a feature vector with a dimension of 8, meaning that the feature size of the stitched image features after the update, composed of multiple first update features, is 3×3×8.

[0187] In one possible implementation, the updated stitched image features satisfy the following relationship:

[0188]

[0189] in, This represents the updated features of the stitched image. These represent the first updated feature, the second updated feature, and the third updated feature in the updated stitched image features, respectively. The first updated feature, This represents the total number of multiple first-update features. It is a positive integer greater than 1.

[0190] In one possible implementation, the process of obtaining the updated stitched image features based on the stitched image features according to steps 308-309 above satisfies the following relationship:

[0191]

[0192]

[0193] in, This represents the updated features of the stitched image; Indicates the features of the stitched image; Represents a linear transformation function; This represents a multi-head self-attention function, used to obtain the first updated feature corresponding to each first sub-feature according to step 308 above; This represents a multilayer perceptron function, used to perform fully connected transformations on features; This represents the first sub-feature in the spliced ​​image features; This represents the total number of first sub-features in the stitched image features.

[0194] It should be noted that this embodiment uses a vector as an example. In another embodiment, the first update feature is not a vector, so step 309 is unnecessary. Instead, other methods can be used to construct the updated stitched image features based on the positions of the multiple first sub-features within the stitched image features. For example, if the first update feature includes a single feature element, then based on the positions of the multiple first sub-features within the stitched image features, the first update features corresponding to the multiple first sub-features are constructed into a two-dimensional feature matrix, which is then determined as the updated stitched image features.

[0195] 310. The computer equipment classifies the features of the updated stitched image to obtain the category to which the pathological image belongs.

[0196] Because each first updated feature in the updated stitched image features not only incorporates features from local features at different scales, but also integrates the first sub-features at each position in the entire stitched image features according to the degree of correlation between the first sub-features at each position in the entire stitched image features, the updated stitched image features are enriched, thereby ensuring the accuracy of the updated stitched image features. By classifying the updated stitched features, the accuracy of the determined categories is ensured.

[0197] In one possible implementation, the category to which the pathological image belongs indicates the state of the target region in the pathological image.

[0198] In one possible implementation, step 310 includes: classifying the updated stitched image features to obtain a classification feature vector, and determining the category corresponding to the highest probability in the classification feature vector as the category to which the pathological image belongs.

[0199] In this embodiment, the classification feature vector includes probabilities in multiple dimensions, with each probability indicating the likelihood that the pathological image belongs to the category corresponding to that dimension. By classifying the updated stitched image features, a classification feature vector is obtained to indicate the likelihood that the pathological image belongs to multiple categories. The category corresponding to the highest probability in the classification feature vector is determined as the category to which the pathological image belongs, thus ensuring the accuracy of the determined category.

[0200] In one possible implementation, the updated mosaic image features and the category to which the pathological image belongs satisfy the following relationship:

[0201]

[0202] in, Indicates the category to which the pathological image belongs; This represents the updated features of the stitched image; Indicates the features of the stitched image; This represents the classification function.

[0203] It should be noted that in this embodiment, after obtaining the stitched image features, the stitched image features are updated according to steps 308-309 above, and the updated image features are directly classified to obtain the category to which the pathological image belongs. In another embodiment, after step 309, the updated stitched image can be updated again according to steps 308-309 above, and this process can be repeated multiple times. After obtaining the last updated stitched image features, the last updated stitched image features are classified to obtain the category to which the pathological image belongs. After obtaining the stitched image features, considering the correlation between features at different positions, the stitched image features are updated multiple times to ensure that features at different positions in the stitched image features are fully integrated, so that the positional information between different features in the final updated stitched image features is accurate.

[0204] In one possible implementation, after obtaining the stitched image features, the stitched image features are updated multiple times, and the last updated stitched image features are classified to determine the category to which the pathological image belongs. This process satisfies the following relationship:

[0205]

[0206] in, Indicates the category to which the pathological image belongs; This represents the updated stitched image features obtained after updating the stitched image features once. Indicates the features of the stitched image; This indicates the number of times the stitched image features are updated. For any positive integer, for example, It is 6; This represents the classification function.

[0207] It should be noted that the method provided in this application embodiment is applicable to classification tasks of various types of pathological images, such as cancer classification, lymph node metastasis, and gene mutation prediction. For example, for any type of cancer corresponding to a pathological image, the state of the cancerous region corresponding to the pathological image can be determined based on the method provided in this application embodiment.

[0208] The method provided in this application, after acquiring the image features of a pathological image, processes the image features of the pathological image according to multiple scales to obtain aggregated features corresponding to multiple scales. Then, the aggregated features corresponding to multiple scales are stitched together into a single stitched image feature. This stitched image feature is then classified to obtain the category of the pathological image. Since the aggregated features corresponding to different scales are obtained by aggregating local features at different scales, and the information contained in the local features at different scales is different, multiple aggregated features contain feature information corresponding to different scales. After stitching multiple aggregated features into a stitched image feature, the stitched image feature contains feature information corresponding to different scales, enriching the feature information of the stitched image feature. Based on this stitched image feature, the category to which the pathological image belongs is determined to ensure the accuracy of the category.

[0209] Furthermore, by first segmenting the pathological image into multiple small sub-images, and then stitching together the image features of multiple sub-images to form the image features of the pathological image after obtaining the image features of each sub-image, there is no need to directly extract features from the large-sized pathological image. This reduces the requirements for equipment performance and can be implemented on various devices, thus ensuring the versatility of this feature extraction method.

[0210] Furthermore, during the processing of image features of the pathological image based on each scale, the second sub-features in each local feature are concatenated into a feature vector. Considering the correlation between different local features, each first feature vector is updated using the first feature vectors corresponding to multiple local features. This ensures the accuracy of the second feature vector by incorporating the first feature vectors corresponding to other local features into the updated second feature vector. In addition, the three-dimensional feature matrix formed by the feature vectors corresponding to multiple local features is determined as the aggregated feature according to the position of multiple local features in the image features of the pathological image. This ensures that the aggregated feature retains the positional information in the original image features of the pathological image, thus guaranteeing the accuracy of the aggregated feature.

[0211] Furthermore, since each first updated feature in the updated stitched image features not only incorporates features from local features at different scales, but also integrates the first sub-features at each position in the entire stitched image features according to the degree of correlation between the first sub-features at each position in the entire stitched image features, the updated stitched image features are enriched, thereby ensuring the accuracy of the updated stitched image features. By classifying the updated stitched features, the accuracy of the determined category is ensured.

[0212] It should be noted that the above Figure 3The illustrated embodiment involves updating each first sub-feature based on the first sub-feature located at multiple positions in the stitched image features after obtaining the stitched image features, to obtain the updated stitched image features, and then classifying the updated stitched image features. In another embodiment, steps 308-310 are not required. After obtaining the stitched image features, other methods can be used to classify the stitched image features and obtain the category to which the pathological image belongs.

[0213] In one possible implementation, the process of obtaining the category to which the pathological image belongs based on the stitched image features after obtaining the stitched image features includes the following steps 1-3:

[0214] Step 1: The computer device fuses each first sub-feature in the stitched image features with the corresponding positional features to obtain the second updated feature corresponding to each first sub-feature.

[0215] In the embodiments of this application, the stitched image features include first sub-features located at multiple positions; the position feature corresponding to any first sub-feature indicates the position of the first sub-feature in the stitched image features, and the position feature can be represented in any form, for example, the position feature is represented in the form of a vector, or the position feature is represented in the form of coordinates.

[0216] Each second updated feature is obtained by fusing the corresponding first sub-feature with the corresponding location feature, in order to enhance the spatial information between the various first sub-features in the stitched image features, enhance the distinction between different first sub-features, and improve the accuracy of the obtained second updated features.

[0217] Step 2: Based on the positions of multiple first sub-features in the stitched image features, the computer device constructs the updated stitched image features from the second updated features corresponding to these multiple first sub-features.

[0218] After updating each first sub-feature, each second updated feature incorporates the positional features of the corresponding first sub-feature to enhance the distinction between different first sub-features. Based on the positions of the multiple first sub-features in the stitched image features, the multiple second updated features are arranged according to their positions within the stitched image features, forming the updated stitched image features. The updated stitched image features include second updated features located at multiple positions. Because each second updated feature in the updated stitched image features incorporates its own positional features, the distinction between different first sub-features is enhanced, thereby improving the accuracy of the updated stitched image features.

[0219] In one possible implementation, the second updated feature is a vector; step 2 includes: based on the positions of multiple first sub-features in the stitched image features, constructing a three-dimensional feature matrix from the second updated features corresponding to the multiple first sub-features, and determining the three-dimensional feature matrix as the updated stitched image features.

[0220] In this embodiment, the first sub-feature is a vector. The second updated feature, obtained by fusing each first sub-feature with its corresponding positional feature, is also a vector. The dimensions of any first sub-feature and its corresponding second updated feature may be the same or different. For example, if the feature size of the stitched image feature is 3×3×10, meaning the stitched image feature includes 9 first sub-features, each a 10-dimensional feature vector; if the dimensions of each first sub-feature and its corresponding second updated feature are the same, the updated stitched image feature obtained by stitching the 9 second updated features has a feature size of 3×3×10; if the dimensions of each first sub-feature and its corresponding second updated feature are different, and each second updated feature has a dimension of 20, the updated stitched image feature obtained by stitching the 9 second updated features has a feature size of 3×3×20.

[0221] Step 3: The computer equipment classifies the features of the updated stitched image to obtain the category to which the pathological image belongs.

[0222] Step 3 is similar to step 310 above, and will not be repeated here.

[0223] The method provided in this application, after obtaining the stitched image features, fuses each first sub-feature in the stitched image features with its corresponding positional features to obtain a second updated feature corresponding to each first sub-feature, thereby enhancing the distinction between different first sub-features. This results in each second updated feature in the updated stitched image features incorporating its own positional features, further enhancing the distinction between different first sub-features and improving the accuracy of the updated stitched image features. Classification is then performed based on these updated stitched image features to ensure the accuracy of the determined categories.

[0224] It should be noted that the various methods described above for updating the features of the stitched image and obtaining the updated features can be combined arbitrarily. In one possible implementation, after obtaining the features of the stitched image, the process of obtaining the category to which the pathological image belongs based on the features of the stitched image includes the following steps 4-8:

[0225] Step 4: The computer device fuses each first sub-feature in the stitched image features with the corresponding positional features to obtain the second updated feature corresponding to each first sub-feature.

[0226] Step 4 is the same as step 1 above, and will not be repeated here.

[0227] Step 5: Based on the positions of multiple first sub-features in the stitched image features, the computer device constructs the first stitched image features from the second updated features corresponding to the multiple first sub-features.

[0228] The first stitched image feature is equivalent to the updated stitched image feature obtained in step 2 above. Step 5 is the same as step 2 above, and will not be repeated here.

[0229] Step 6: The computer device updates each of the multiple second update features to obtain the third update feature corresponding to each second update feature.

[0230] Step 6 is similar to step 308 above, and will not be repeated here.

[0231] Step 7: The computer device constructs a second stitched image feature from multiple third update features based on the positions of multiple second update features in the first stitched image feature.

[0232] In this embodiment, each third update feature may include a feature element or a vector. When each third update feature includes a feature element, the positions of the multiple second update features in the first stitched image feature are used to construct a two-dimensional feature matrix, which is then determined as the second stitched image feature. When each third update feature is a vector, step 7 is analogous to step 309 described above, and will not be repeated here.

[0233] Step 8: The computer equipment classifies the features of the second stitched image to obtain the category to which the pathological image belongs.

[0234] This step is similar to step 310 above, and will not be repeated here.

[0235] The method provided in this application, during the process of updating the features of the stitched image, incorporates not only the corresponding positional information but also the correlation degree between the first sub-features at each position for each feature in the stitched image features. This means that not only the absolute positional information of the features at each position in the stitched image features is considered, but also the relative positional relationship between different positions. This approach can capture both local and global information in the pathological image, thereby improving the accuracy of the second stitched image features. By classifying the updated stitched features, the accuracy of the determined category is ensured.

[0236] It should be noted that, in the above Figure 2 and Figure 3 Based on the embodiments shown, pathological images can also be classified according to a classification model to obtain the category to which the pathological image belongs.

[0237] In one possible implementation, the classification model includes a feature extraction sub-model and a classification sub-model, wherein the classification sub-model includes an absolute position encoding layer, at least one spatial encoder, and a linear layer. The process of classifying pathological images based on the classification model is as follows: Figure 7 As shown, based on the feature extraction sub-model, following steps 301-303 above, the pathological image to be classified is segmented into multiple sub-images, and then the image features of each sub-image are extracted. Finally, the image features of multiple sub-images are spliced ​​together to form the image features of the pathological image.

[0238] Based on the sub-classification model, following steps 304-306 above, multiple aggregated features are obtained using multiple scales; then, following step 307 above, the multiple aggregated features are stitched together to obtain stitched image features.

[0239] Based on the absolute position encoding layer, each first sub-feature in the stitched image features is fused with the corresponding position feature to obtain the second updated feature corresponding to each first sub-feature. Based on the position of multiple first sub-features in the stitched image features, the second updated features corresponding to multiple first sub-features in the stitched image features are used to form the first stitched image features.

[0240] Based on the first spatial encoder, following steps 308-309 above, each second update feature is updated based on multiple second update features in the first stitched image features to obtain a third update feature corresponding to each second update feature; based on the positions of the multiple second update features in the first stitched image features, the multiple third update features are combined to form a second stitched image feature.

[0241] Based on the q-th spatial encoder, the stitched image features output by the (q-1)-th spatial encoder are updated again according to steps 308-309 above, until the last spatial encoder outputs the updated stitched image features, where q is an integer greater than 1.

[0242] Based on the linear layer in the classifier sub-model, the updated stitched image features output by the last spatial encoder are classified to obtain the category to which the pathological image belongs.

[0243] Optionally, each spatial encoder includes a normalization layer, a multi-head attention layer, a fusion layer, and a multilayer perceptron.

[0244] Taking the first spatial encoder as an example, based on the normalization layer, feature transformation is performed on the first stitched image features to obtain the transformed first stitched image features; based on the multi-head self-attention layer, the transformed first stitched image features are processed according to steps 308-309 above to obtain the updated first stitched image features; based on the first fusion layer, the first stitched image features are fused with the updated first stitched image features to obtain fused image features, thereby enhancing the accuracy of the fused image features; based on the second normalization layer, feature transformation is performed on the fused image features to obtain the transformed fused image features; based on the multilayer perceptron, the transformed fused image features are mapped to obtain mapped features, and the mapped features are fused with the transformed fused image features to obtain the second stitched image features.

[0245] It should be noted that before classifying pathological images based on the classification model, the classification model needs to be trained. Taking a classification model that includes a feature extraction sub-model and a classification sub-model as an example, the process of training the classification model includes the following steps 1-4:

[0246] Step 1: Obtain the pathological image of the sample and the sample label, which indicates the category to which the pathological image of the sample belongs.

[0247] The pathological image of the sample is a pathological image that presents the pathological morphology of organs, tissues or cells of the body, and the category indicated by the sample label is the category to which the pathological image of the sample belongs.

[0248] Step 2: Based on the feature extraction sub-model, obtain the sample image features of the pathological image of the sample.

[0249] Step 2 is similar to steps 301-303 above, and will not be repeated here.

[0250] Step 3: Based on the classification sub-model, for each scale among multiple scales, extract at least one sample local feature corresponding to that scale from the sample image features, aggregate the at least one sample local feature corresponding to that scale to obtain sample aggregate features; stitch the obtained multiple sample aggregate image features to obtain sample stitched image features; classify the sample stitched image features to obtain the predicted label to which the sample pathological image belongs, and the predicted label indicates the predicted category to which the sample pathological image belongs.

[0251] Step 3 is similar to steps 304-310 above, and will not be repeated here.

[0252] Step 4: Train the classification model based on the predicted label and the sample label.

[0253] Since the predicted label indicates the category to which the pathological image of the sample belongs based on the classification model, and the sample label indicates the category to which the pathological image of the sample belongs, comparing the predicted label with the sample label can reflect the accuracy of the classification model. Based on the predicted label and the sample label, the classification model is trained to improve the accuracy of the classification model.

[0254] In one possible implementation, step 4 includes: determining a loss value based on the predicted label and the sample label, and training the classification model based on the loss value.

[0255] The loss value represents the degree of difference between the predicted label and the sample label. The classification model is trained based on this loss value to improve its accuracy.

[0256] In one possible implementation, based on multiple sample pathological images and their corresponding sample labels, the classification model is trained iteratively in multiple rounds according to steps 1-4 above. If the loss value obtained in the current iteration is less than the loss value threshold, the training of the classification model is stopped; or, if the number of iterations reaches the target number, the training of the classification model is stopped.

[0257] By setting a threshold for the loss value or a target number of iterations when training the classification model, the accuracy of the trained classification model can be ensured to be as high as possible.

[0258] It should be noted that during the training of the classification model described above, the feature extraction sub-model can be any pre-trained image feature extraction model, such as ResNet (Residual Neural Network), DenseNet (Densely Connected Convolutional Networks), or Inception Networks. Based on this feature extraction sub-model, the feature extraction sub-model and the classification sub-model are combined to form a classification model. The classification model is then trained according to steps 1-4 above to save training time and ensure the accuracy of the classification model.

[0259] Based on the classification model provided in this application embodiment, a comparison is made with various classification models in related technologies. These related technologies include RNN-MIL (Recurrent Neural Network-Multiple Instance Learning), CNN-MIL (Convolutional Neural Network-Multiple Instance Learning), and ViT-MIL (VisionTransformer-Multiple Instance Learning). As shown in Table 1, the classification model provided in this application embodiment is compared with various related technologies on different datasets and for different classification tasks. The classification model provided in this application embodiment achieves an AUC (Area Under Curve) of over 90% on the first two datasets and over 90% on the lung adenocarcinoma-gene mutation dataset. Based on Table 1, it can be seen that the classification model provided in this application performs better than the classification models in related technologies.

[0260] Table 1

[0261]

[0262] The classification model provided in this application includes a location encoding layer, an encoder-based pyramid multi-scale fusion, and a spatial encoder. For example, the encoder-based pyramid multi-scale fusion is TPMF (Transformer-based Pyramid Multi-Scale Fusion); the spatial encoder is SET (Spatial Encoding Transformer). To determine the importance of each part of the classification model when performing image classification based on the model provided in this application, the model was tested on four different datasets, as shown in Table 2. Based on the data in Table 2, it can be seen that all parts of the classification model provided in this application are important, and the classification model composed of these parts has higher classification performance.

[0263] Table 2

[0264]

[0265] In the method provided in this application embodiment, the performance of the classification model is evaluated using various positional encoding strategies. When testing the classification model on four different datasets, two positional encoding strategy configurations were set: absolute positional encoding and relative positional encoding. As shown in Table 3, using both absolute and relative positional encoding mechanisms can improve the accuracy of image classification.

[0266] Table 3

[0267]

[0268] Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0269] The acquisition module 801 is used to acquire the image features of the pathological images to be classified;

[0270] The aggregation module 802 is used to extract at least one local feature corresponding to each scale from the image features for each scale among multiple scales, and to aggregate the at least one local feature corresponding to the scale to obtain aggregated features.

[0271] The stitching module 803 is used to stitch together multiple aggregated features to obtain stitched image features;

[0272] The classification module 804 is used to classify the features of the stitched images to obtain the category to which the pathological images belong.

[0273] In one possible implementation, the stitched image features include first sub-features located at multiple locations; such as Figure 9 As shown, classification module 804 includes:

[0274] The update unit 8041 is used to update each first sub-feature based on multiple first sub-features to obtain the first updated feature corresponding to each first sub-feature;

[0275] The constituting unit 8042 is used to construct the updated stitched image features by combining the first updated features corresponding to the multiple first sub-features based on their positions in the stitched image features.

[0276] Classification unit 8043 is used to classify the updated spliced ​​image features to obtain the category to which the pathological image belongs.

[0277] In another possible implementation, the update unit 8041 is used to obtain the weights of the target sub-feature among the multiple first sub-features, where the weights indicate the degree of correlation between the corresponding first sub-feature and the target sub-feature; and to perform weighted fusion of the multiple first sub-features based on their weights to obtain the first updated feature corresponding to the target sub-feature.

[0278] In another possible implementation, the update unit 8041 is used to obtain the distance feature between each first sub-feature and the target sub-feature among multiple first sub-features, the distance feature indicating the distance between the positions of the first sub-feature and the target sub-feature in the stitched image features; and to obtain the weight of each first sub-feature based on the similarity and distance feature between each first sub-feature and the target sub-feature.

[0279] In another possible implementation, the update unit 8041 is used to determine the target distance corresponding to each first sub-feature based on the position of each first sub-feature and the target sub-feature in the stitched image features. The target distance represents the distance between the positions of the first sub-feature and the target sub-feature. The target distance corresponding to each first sub-feature is mapped to obtain the distance feature between each first sub-feature and the target sub-feature.

[0280] In another possible implementation, the first updated feature is a vector; the constituting unit 8042 is used to construct a three-dimensional feature matrix based on the positions of the multiple first sub-features in the stitched image features, and to determine the three-dimensional feature matrix as the updated stitched image features.

[0281] In another possible implementation, the stitched image features include first sub-features located at multiple locations; such as Figure 9 As shown, the device also includes:

[0282] The fusion module 805 is used to fuse each first sub-feature with the corresponding position feature to obtain the second updated feature corresponding to each first sub-feature. The position feature indicates the position of the corresponding first sub-feature in the stitched image features.

[0283] The constituting module 806 is used to construct the updated constituting image features by combining the second updated features corresponding to the multiple first sub-features based on their positions in the constituting image features.

[0284] In another possible implementation, the second updated feature is a vector; the constituting module 806 is used to construct a three-dimensional feature matrix based on the positions of the multiple first sub-features in the stitched image features, and to determine the three-dimensional feature matrix as the updated stitched image features.

[0285] In another possible implementation, the local features include at least one second sub-feature located at at least one position; the aggregation module 802 is used to concatenate the second sub-features in each extracted local feature to obtain a first feature vector corresponding to each local feature; based on the first feature vectors corresponding to multiple local features, each first feature vector is updated to obtain a second feature vector corresponding to each first feature vector; based on the positions of multiple local features in the image features, multiple second feature vectors are used to form a three-dimensional feature matrix, and the three-dimensional feature matrix is ​​determined as the aggregated feature.

[0286] In another possible implementation, the acquisition module 801 is used to segment the pathological image to obtain multiple sub-images; to extract features from each sub-image to obtain the image features of each sub-image; and to stitch together the image features of the multiple sub-images based on their positions in the pathological image to obtain the image features of the pathological image.

[0287] In another possible implementation, the feature sizes of multiple aggregated features are the same; the stitching module 803 is used to stitch together the features in the same position among the multiple aggregated features to obtain feature vectors corresponding to multiple positions; the three-dimensional feature matrix formed by the feature vectors corresponding to multiple positions is determined as the stitched image feature.

[0288] In another possible implementation, the steps of obtaining image features of the pathological image to be classified, extracting at least one local feature corresponding to each scale from the image features for each scale, aggregating the at least one local feature corresponding to the scale to obtain aggregated features, stitching together the obtained aggregated features to obtain stitched image features, and classifying the stitched image features to obtain the category to which the pathological image belongs are implemented based on a classification model.

[0289] In another possible implementation, the classification model includes a feature extraction sub-model and a classification sub-model; such as Figure 9 As shown, the device also includes:

[0290] The acquisition module 801 is also used to acquire sample pathological images and sample labels, whereby the sample labels indicate the category to which the sample pathological images belong.

[0291] The acquisition module 801 is also used to acquire sample image features of the sample pathological image based on the feature extraction sub-model;

[0292] The prediction module 807 is used to extract at least one local feature corresponding to the scale from the sample image features for each scale in multiple scales based on the classification sub-model, aggregate the at least one local feature corresponding to the scale to obtain sample aggregate features, stitch the obtained multiple sample aggregate image features to obtain sample stitched image features, classify the sample stitched image features to obtain the prediction label to which the sample pathological image belongs, and the prediction label indicates the predicted category to which the sample pathological image belongs.

[0293] Training module 808 is used to train the classification model based on the predicted label and the sample label.

[0294] It should be noted that the image classification device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image classification device and the image classification method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0295] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed by the image classification method of the above embodiments.

[0296] Optionally, the computer device is provided as a terminal. Figure 10 This illustration shows a structural block diagram of a terminal 1000 provided in an exemplary embodiment of this application. The terminal may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, etc.

[0297] Terminal 1000 includes a processor 1001 and a memory 1002.

[0298] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0299] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one computer program, which is executed by the processor 1001 to implement the image classification method provided in the method embodiments of this application.

[0300] In some embodiments, the terminal 1000 may also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1009.

[0301] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0302] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1004 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0303] Display screen 1005 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1005, disposed on the front panel of terminal 1000; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1000 or in a folded design; in still other embodiments, display screen 1005 may be a flexible display screen, disposed on a curved or folded surface of terminal 1000. Furthermore, display screen 1005 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0304] The camera assembly 1006 is used to acquire images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0305] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0306] Power supply 1009 is used to power the various components in terminal 1000. Power supply 1009 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1009 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0307] In some embodiments, the terminal 1000 further includes one or more sensors 1010. The one or more sensors 1010 include, but are not limited to: an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1010, an optical sensor 1014, and a proximity sensor 1015.

[0308] Accelerometer 1011 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 1000. For example, accelerometer 1011 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1001 can control display screen 1005 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1011. Accelerometer 1011 can also be used for games or for acquiring user motion data.

[0309] The gyroscope sensor 1012 can detect the orientation and rotation angle of the terminal 1000. The gyroscope sensor 1012, in conjunction with the accelerometer sensor 1011, can collect 3D motion data from the user on the terminal 1000. Based on the data collected by the gyroscope sensor 1012, the processor 1001 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0310] The pressure sensor 1010 can be disposed on the side bezel of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1010 is disposed on the side bezel of the terminal 1000, it can detect the user's grip signal on the terminal 1000, and the processor 1001 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1010. When the pressure sensor 1010 is disposed on the lower layer of the display screen 1005, the processor 1001 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0311] An optical sensor 1014 is used to collect ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 based on the ambient light intensity collected by the optical sensor 1014. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 based on the ambient light intensity collected by the optical sensor 1014.

[0312] The proximity sensor 1015, also known as a distance sensor, is installed on the front panel of the terminal 1000. The proximity sensor 1015 is used to detect the distance between the user and the front of the terminal 1000. In one embodiment, when the proximity sensor 1015 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from a screen-on state to a screen-off state; when the proximity sensor 1015 detects that the distance between the user and the front of the terminal 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from a screen-off state to a screen-on state.

[0313] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on terminal 1000 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0314] Optionally, the computer equipment is provided as a server. Figure 11This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The memory 1102 stores at least one computer program, which is loaded and executed by the processor 1101 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0315] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the image classification method of the above embodiments.

[0316] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the operations performed by the image classification method described above.

[0317] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0318] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. An image classification method, characterized in that, The method includes: Obtain image features of the pathological images to be classified; For each of the multiple scales, at least one local feature corresponding to the scale is extracted from the image features, and the at least one local feature corresponding to the scale is aggregated to obtain aggregated features; The obtained multiple aggregated features are concatenated to obtain the concatenated image features; The features of the stitched image are classified to obtain the category to which the pathological image belongs; The local features include at least one second sub-feature located at at least one position; the aggregation of at least one local feature corresponding to the scale to obtain aggregated features includes: The second sub-features of each extracted local feature are concatenated to obtain the first feature vector corresponding to each local feature. Based on the first feature vectors corresponding to multiple local features, each first feature vector is updated to obtain the second feature vector corresponding to each first feature vector. The second feature vector corresponding to each first feature vector incorporates the first feature vectors corresponding to other local features. Based on the positions of the multiple local features in the image features, the multiple second feature vectors are used to form the aggregated feature.

2. The method according to claim 1, characterized in that, The stitched image features include first sub-features located at multiple positions; classifying the stitched image features to obtain the category to which the pathological image belongs includes: Based on multiple first sub-features, each first sub-feature is updated to obtain the first updated feature corresponding to each first sub-feature. Based on the positions of the plurality of first sub-features in the stitched image features, the first updated features corresponding to the plurality of first sub-features constitute the updated stitched image features; The updated stitched image features are classified to determine the category to which the pathological image belongs.

3. The method according to claim 2, characterized in that, The step of updating each first sub-feature based on multiple first sub-features to obtain a first updated feature corresponding to each first sub-feature includes: For the target sub-feature among the plurality of first sub-features, the weights of the plurality of first sub-features are obtained, and the weights indicate the degree of correlation between the corresponding first sub-feature and the target sub-feature; Based on the weights of the multiple first sub-features, the multiple first sub-features are weighted and fused to obtain the first updated feature corresponding to the target sub-feature.

4. The method according to claim 3, characterized in that, The step of obtaining the weights of the target sub-features among the plurality of first sub-features includes: For the target sub-feature among the plurality of first sub-features, a distance feature between each first sub-feature and the target sub-feature is obtained, wherein the distance feature indicates the distance between the positions of the first sub-feature and the target sub-feature in the stitched image features; The weight of each first sub-feature is obtained based on the similarity between each first sub-feature and the target sub-feature and the distance feature.

5. The method according to claim 4, characterized in that, For each of the plurality of first sub-features, the step of obtaining the distance feature between each first sub-feature and the target sub-feature includes: Based on the positions of each first sub-feature and the target sub-feature in the stitched image features, a target distance corresponding to each first sub-feature is determined, whereby the target distance represents the distance between the positions of the first sub-feature and the target sub-feature. The target distance corresponding to each first sub-feature is mapped to obtain the distance feature between each first sub-feature and the target sub-feature.

6. The method according to claim 2, characterized in that, The first updated feature is a vector; the step of constructing the updated stitched image features by constructing the first updated features corresponding to the plurality of first sub-features based on their positions in the stitched image features includes: Based on the positions of the plurality of first sub-features in the stitched image features, the first updated features corresponding to the plurality of first sub-features are used to form a three-dimensional feature matrix, and the three-dimensional feature matrix is ​​determined as the updated stitched image features.

7. The method according to claim 1, characterized in that, The stitched image features include first sub-features located at multiple positions; before classifying the stitched image features to obtain the category to which the pathological image belongs, the method further includes: Each first sub-feature is fused with its corresponding position feature to obtain a second updated feature corresponding to each first sub-feature, wherein the position feature indicates the position of the corresponding first sub-feature in the stitched image features; Based on the positions of multiple first sub-features in the stitched image features, the second updated features corresponding to the multiple first sub-features are used to form the updated stitched image features.

8. The method according to claim 7, characterized in that, The second updated feature is a vector; the step of constructing the updated stitched image features by constructing the second updated features corresponding to the multiple first sub-features based on their positions in the stitched image features includes: Based on the positions of the plurality of first sub-features in the stitched image features, the second updated features corresponding to the plurality of first sub-features are used to form a three-dimensional feature matrix, and the three-dimensional feature matrix is ​​determined as the updated stitched image features.

9. The method according to claim 1, characterized in that, The step of constructing the aggregated feature from multiple second feature vectors includes: Multiple second feature vectors are used to form a three-dimensional feature matrix, and the three-dimensional feature matrix is ​​determined as the aggregated feature.

10. The method according to claim 1, characterized in that, The multiple aggregated features have the same feature size; the process of concatenating the obtained multiple aggregated features to obtain concatenated image features includes: By concatenating features that are in the same position among multiple aggregated features, we obtain feature vectors corresponding to multiple positions. The three-dimensional feature matrix formed by the feature vectors corresponding to the multiple positions is determined as the feature of the stitched image.

11. The method according to any one of claims 1-10, characterized in that, The steps of acquiring image features of the pathological image to be classified; extracting at least one local feature corresponding to each scale from the image features for each scale; aggregating the at least one local feature corresponding to the scale to obtain aggregated features; stitching together the obtained aggregated features to obtain stitched image features; and classifying the stitched image features to obtain the category to which the pathological image belongs are implemented based on a classification model.

12. The method according to claim 11, characterized in that, The classification model includes a feature extraction sub-model and a classification sub-model; the method further includes: Obtain the pathological images and labels of the samples, wherein the labels indicate the category to which the pathological images of the samples belong; Based on the feature extraction sub-model, the sample image features of the sample pathological image are obtained; Based on the classification sub-model, for each of the multiple scales, at least one sample local feature corresponding to the scale is extracted from the sample image features. The at least one sample local feature corresponding to the scale is aggregated to obtain sample aggregate features. The obtained multiple sample aggregate features are stitched together to obtain sample stitched image features. The sample stitched image features are classified to obtain the predicted label to which the sample pathological image belongs. The predicted label indicates the predicted category to which the sample pathological image belongs. The classification model is trained based on the predicted labels and the sample labels.

13. An image classification device, characterized in that, The device includes: The acquisition module is used to acquire image features of the pathological images to be classified; An aggregation module is used to extract at least one local feature corresponding to each scale from the image features for each scale among multiple scales, and to aggregate the at least one local feature corresponding to the scale to obtain aggregated features; The stitching module is used to stitch together multiple aggregated features to obtain stitched image features; A classification module is used to classify the features of the stitched image to obtain the category to which the pathological image belongs; The local feature includes at least one second sub-feature located at at least one position; the aggregation module is used for: The second sub-features of each extracted local feature are concatenated to obtain the first feature vector corresponding to each local feature. Based on the first feature vectors corresponding to multiple local features, each first feature vector is updated to obtain the second feature vector corresponding to each first feature vector. The second feature vector corresponding to each first feature vector incorporates the first feature vectors corresponding to other local features. Based on the positions of the multiple local features in the image features, the multiple second feature vectors are used to form the aggregated feature.

14. The apparatus according to claim 13, characterized in that, The stitched image features include first sub-features located at multiple positions; The classification module includes: The update unit is used to update each first sub-feature based on multiple first sub-features to obtain the first updated feature corresponding to each first sub-feature; The constituting unit is used to construct the updated stitched image features by combining the first updated features corresponding to the plurality of first sub-features based on their positions in the stitched image features. A classification unit is used to classify the updated stitched image features to obtain the category to which the pathological image belongs.

15. The apparatus according to claim 14, characterized in that, The update unit is used for: For the target sub-feature among the plurality of first sub-features, the weights of the plurality of first sub-features are obtained, and the weights indicate the degree of correlation between the corresponding first sub-feature and the target sub-feature; Based on the weights of the multiple first sub-features, the multiple first sub-features are weighted and fused to obtain the first updated feature corresponding to the target sub-feature.

16. The apparatus according to claim 15, characterized in that, The update unit is used for: For the target sub-feature among the plurality of first sub-features, a distance feature between each first sub-feature and the target sub-feature is obtained, wherein the distance feature indicates the distance between the positions of the first sub-feature and the target sub-feature in the stitched image features; The weight of each first sub-feature is obtained based on the similarity between each first sub-feature and the target sub-feature and the distance feature.

17. The apparatus according to claim 16, characterized in that, The update unit is used for: Based on the positions of each first sub-feature and the target sub-feature in the stitched image features, a target distance corresponding to each first sub-feature is determined, whereby the target distance represents the distance between the positions of the first sub-feature and the target sub-feature. The target distance corresponding to each first sub-feature is mapped to obtain the distance feature between each first sub-feature and the target sub-feature.

18. The apparatus according to claim 14, characterized in that, The first updated feature is a vector; the constituting unit is used for: Based on the positions of the plurality of first sub-features in the stitched image features, the first updated features corresponding to the plurality of first sub-features are used to form a three-dimensional feature matrix, and the three-dimensional feature matrix is ​​determined as the updated stitched image features.

19. The apparatus according to claim 13, characterized in that, The stitched image features include first sub-features located at multiple positions; the device further includes: The fusion module is used to fuse each first sub-feature with its corresponding position feature to obtain a second updated feature corresponding to each first sub-feature, wherein the position feature indicates the position of the corresponding first sub-feature in the stitched image feature; The composition module is used to construct updated stitched image features by combining the second updated features corresponding to the multiple first sub-features based on their positions in the stitched image features.

20. The apparatus according to claim 19, characterized in that, The second updated feature is a vector; the constituting module is used to construct a three-dimensional feature matrix based on the positions of the plurality of first sub-features in the stitched image features, and to determine the three-dimensional feature matrix as the updated stitched image features.

21. The apparatus according to claim 13, characterized in that, The aggregation module is used to construct a three-dimensional feature matrix from multiple second feature vectors, and to determine the three-dimensional feature matrix as the aggregated feature.

22. The apparatus according to claim 13, characterized in that, The multiple aggregated features have the same feature size; the splicing module is used for: By concatenating features that are in the same position among multiple aggregated features, we obtain feature vectors corresponding to multiple positions. The three-dimensional feature matrix formed by the feature vectors corresponding to the multiple positions is determined as the feature of the stitched image.

23. The apparatus according to any one of claims 13-22, characterized in that, The steps of acquiring image features of the pathological image to be classified; extracting at least one local feature corresponding to each scale from the image features for each scale; aggregating the at least one local feature corresponding to the scale to obtain aggregated features; stitching together the obtained aggregated features to obtain stitched image features; and classifying the stitched image features to obtain the category to which the pathological image belongs are implemented based on a classification model.

24. The apparatus according to claim 23, characterized in that, The classification model includes a feature extraction sub-model and a classification sub-model; the device further includes: The acquisition module is also used to acquire sample pathological images and sample labels, wherein the sample labels indicate the category to which the sample pathological images belong; The acquisition module is further configured to acquire sample image features of the sample pathological image based on the feature extraction sub-model; The prediction module is configured to, based on the classification sub-model, extract at least one local feature corresponding to each scale from the sample image features for each of the multiple scales, aggregate the at least one local feature corresponding to the scale to obtain aggregated sample features, stitch the obtained aggregated sample features to obtain stitched sample image features, classify the stitched sample image features to obtain a predicted label to which the sample pathological image belongs, and the predicted label indicates the predicted category to which the sample pathological image belongs. The training module is used to train the classification model based on the predicted label and the sample label.

25. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations of the image classification method as claimed in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the image classification method as described in any one of claims 1 to 12.

27. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it performs the operations of the image classification method as described in any one of claims 1 to 12.

Images