Image processing methods, apparatus, computer equipment and storage media

Abstract

This application provides an image processing method, apparatus, computer device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: segmenting a medical image containing a target object into multiple sub-images; extracting features from each of the multiple sub-images to obtain multiple sub-image features; determining a coding feature corresponding to the first sub-image feature based on a first sub-image feature and multiple second sub-image features, wherein the first sub-image feature is any one of the multiple sub-image features, and the multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature; and determining a state recognition result of the target object based on the multiple coding features corresponding to the multiple sub-image features. This application, by effectively integrating the sub-image features, obtains a more accurate feature representation, thereby improving the accuracy of state recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, computer device, and storage medium. Background Technology

[0002] Currently, artificial intelligence technology is widely used in image processing. For example, in medical image processing, artificial intelligence can be applied to process medical images containing target objects to obtain state recognition results for the target objects; however, the current state recognition results are not accurate enough. Summary of the Invention

[0003] This application provides an image processing method, apparatus, computer device, and storage medium that can improve the accuracy of state recognition. The technical solution is as follows.

[0004] On the one hand, an image processing method is provided, the method comprising:

[0005] The medical image containing the target object is divided into multiple sub-images, and feature extraction is performed on each of the multiple sub-images to obtain multiple sub-image features;

[0006] Based on a first sub-image feature and multiple second sub-image features, a coding feature corresponding to the first sub-image feature is determined. The first sub-image feature is any one of the multiple sub-image features. The multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature.

[0007] Based on the multiple encoded features corresponding to the multiple sub-image features, the state recognition result of the target object is determined.

[0008] In one possible implementation, before determining the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features, the method further includes:

[0009] Based on the coding features corresponding to the first sub-image features and the multiple coding features corresponding to the multiple second sub-image features respectively, the updated coding features corresponding to the first sub-image features are determined.

[0010] In one possible implementation, the key features and value features of the encoded features are obtained based on a second parameter matrix, and the method further includes:

[0011] In the presence of multiple second parameter matrices, the multiple decoding features obtained based on the multiple second parameter matrices are concatenated and compressed to obtain the updated decoding features.

[0012] In one possible implementation, the query features, key features, and value features of the decoded features, as well as the query features, key features, and value features of the initial identifier features, are obtained based on a third parameter matrix. The method further includes:

[0013] In the presence of multiple third parameter matrices, the multiple second reference query features obtained based on the multiple third parameter matrices are concatenated and compressed to obtain the updated second reference query features.

[0014] In one possible implementation, before determining the state recognition result of the target object based on multiple obtained decoding features, the method further includes:

[0015] Each time a decoding feature is obtained, a third reference query feature is determined based on the obtained decoding feature; the third reference query feature and the key and value features of the multiple encoded features are cross-processed to obtain the updated decoding feature.

[0016] In one possible implementation, the encoder includes a first self-attention unit;

[0017] The step of invoking the encoder to determine the encoding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes:

[0018] The first self-attention unit is invoked to perform cross-processing on the query features of the first sub-image features and the key and value features of each second sub-image feature to obtain the encoded features corresponding to the first sub-image features.

[0019] In one possible implementation, the encoder further includes a first feature update unit;

[0020] After invoking the first self-attention unit to perform cross-processing on the query features of the first sub-image features and the key and value features of each second sub-image feature to obtain the encoded features corresponding to the first sub-image features, the method further includes:

[0021] The first feature update unit is invoked to update the encoded features corresponding to the first sub-image features, thereby obtaining the updated encoded features corresponding to the first sub-image features.

[0022] In one possible implementation, the feature processing network includes multiple encoders;

[0023] The step of determining the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features includes: calling a first encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features; and calling a second encoder connected after the first encoder to determine the updated encoding feature corresponding to the first sub-image feature based on the encoding feature corresponding to the first sub-image feature and multiple encoding features respectively corresponding to the multiple second sub-image features.

[0024] In one possible implementation, the decoder further includes a second feature update unit, and the method further includes:

[0025] Each time a decoded feature is obtained, the second feature update unit is called to update the decoded feature, and the updated decoded feature is obtained.

[0026] In one possible implementation, the feature processing network includes multiple decoders, and before invoking the classifier to determine the state recognition result of the target object based on the obtained multiple decoded features, the method further includes:

[0027] Each time the first decoder is called to obtain the decoded features, the second decoder connected after the first decoder is called. Based on the obtained decoded features, a third reference query feature is determined. The third reference query feature and the key and value features of the multiple encoded features are cross-processed to obtain the updated decoded features.

[0028] In one possible implementation, the step of invoking the feature extraction network to extract features from the plurality of sub-images respectively, thereby obtaining multiple sub-image features, includes:

[0029] The feature extraction network is invoked to compress the size of the initial feature map of each sub-image until a feature map of the target size is obtained; the feature map of the target size is then compressed into a sub-image feature map of size 1×1.

[0030] In one possible implementation, the number of channels for each sub-image feature is a first quantity, and the feature processing network further includes a feature compressor;

[0031] Before invoking the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features, the method further includes:

[0032] The feature compressor is invoked to perform convolution processing on each of the sub-image features to obtain convolution-processed sub-image features. The number of channels of the convolution-processed sub-image features is a second number, which is less than the first number.

[0033] The step of invoking the encoder to determine the encoding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes:

[0034] The encoder is invoked to determine the encoded features corresponding to the first sub-image features based on the first sub-image features after convolution and multiple second sub-image features after convolution.

[0035] In one possible implementation, the plurality of sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image to form a feature map of the medical image, wherein the number of channels of the feature map is a first quantity; the feature processing network further includes a feature compressor;

[0036] Before invoking the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features, the method further includes:

[0037] The feature compressor is invoked to perform convolution processing on the feature map of the medical image to obtain a convolution-processed feature map. The number of channels in the convolution-processed feature map is a second number, which is less than the first number.

[0038] The step of invoking the encoder to determine the encoding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes:

[0039] The encoder is invoked to determine the encoded features corresponding to the first sub-image features based on the first sub-image features in the convolutional feature map and multiple second sub-image features in the convolutional feature map.

[0040] On one hand, an image processing apparatus is provided, the apparatus comprising:

[0041] The sub-image feature extraction module is used to divide a medical image containing a target object into multiple sub-images, and extract features from each of the multiple sub-images to obtain multiple sub-image features;

[0042] The coding feature determination module is used to determine the coding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features. The first sub-image feature is any one of the multiple sub-image features, and the multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature.

[0043] The state recognition module is used to determine the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features respectively.

[0044] In one possible implementation, the encoding feature determination module is configured to:

[0045] The query features of the first sub-image feature and the key and value features of each second sub-image feature are cross-processed to obtain the encoded features corresponding to the first sub-image feature.

[0046] In one possible implementation, the encoding feature determination module is configured to:

[0047] Based on the query features of the first sub-image features and the key features of each second sub-image feature, the correlation parameters between the first sub-image features and each second sub-image feature are determined.

[0048] According to the multiple correlation parameters corresponding to the multiple second sub-image features, the value features of the multiple second sub-image features are weighted and fused to obtain the encoded features corresponding to the first sub-image features.

[0049] In one possible implementation, the query features, key features, and value features of the sub-image features are obtained based on a first parameter matrix, and the apparatus further includes:

[0050] The first coding feature update module is used to concatenate and compress multiple coding features corresponding to the first sub-image features obtained based on the multiple first parameter matrices when multiple first parameter matrices exist, so as to obtain the updated coding features corresponding to the first sub-image features.

[0051] In one possible implementation, the device further includes:

[0052] The associated location determination module is used to determine a second location associated with the first location based on the first location of the first sub-image in the medical image, wherein the features of the first sub-image are obtained by feature extraction of the first sub-image;

[0053] The associated sub-image feature determination module is used to determine the sub-image features corresponding to the second sub-image located at the second position as the associated sub-image features.

[0054] In one possible implementation, the device further includes:

[0055] The second coding feature update module is used to determine the updated coding feature corresponding to the first sub-image feature based on the coding feature corresponding to the first sub-image feature and the multiple coding features corresponding to the multiple second sub-image features respectively.

[0056] In one possible implementation, the state recognition module includes:

[0057] The first decoding submodule is used to determine a first reference query feature based on the initial identifier feature, and to perform cross-processing on the key features and value features of the first reference query feature and the plurality of encoded features to obtain the decoding feature, wherein the initial identifier feature is used to indicate the start of decoding;

[0058] The second decoding submodule is used to determine the second reference query feature based on the initial identifier feature and the obtained decoding feature, and to perform cross-processing on the second reference query feature and the key features and value features of the multiple encoded features to obtain the next decoding feature, until the obtained decoding feature indicates the end of decoding;

[0059] The state recognition submodule is used to determine the state recognition result of the target object based on multiple decoded features obtained.

[0060] In one possible implementation, the key features and value features of the encoded features are obtained based on a second parameter matrix, and the apparatus further includes:

[0061] The first decoding feature update module is used to concatenate and compress multiple decoding features obtained based on the multiple second parameter matrices when multiple second parameter matrices exist, so as to obtain the updated decoding features.

[0062] In one possible implementation, the second decoding submodule is used for:

[0063] Based on the latest obtained decoding feature query feature and the key feature of the initial identifier feature, determine the relevance parameter corresponding to the initial identifier feature; and based on the latest obtained decoding feature query feature and the key feature of each obtained decoding feature, determine the relevance parameter corresponding to each obtained decoding feature.

[0064] The value features of the initial identifier feature and the value features of each obtained decoded feature are weighted and fused according to the relevance parameters corresponding to the initial identifier feature and the relevance parameters corresponding to each obtained decoded feature to obtain the second reference query feature.

[0065] In one possible implementation, the query features, key features, and value features of the decoded features, as well as the query features, key features, and value features of the initial identifier features, are obtained based on a third parameter matrix. The apparatus further includes:

[0066] The query feature update module is used to concatenate and compress multiple second reference query features obtained based on the multiple third parameter matrices when multiple third parameter matrices exist, so as to obtain updated second reference query features.

[0067] In one possible implementation, the device further includes:

[0068] The second decoding feature update module is used to determine a third reference query feature based on the obtained decoding features after each decoding feature is obtained; and to perform cross-processing on the third reference query feature and the key features and value features of the multiple encoded features to obtain the updated decoding features.

[0069] In one possible implementation, the number of channels for each sub-image feature is a first quantity, and the apparatus further includes:

[0070] The feature compression module is used to perform convolution processing on each of the sub-image features to obtain convolution-processed sub-image features, wherein the number of channels of the convolution-processed sub-image features is a second number, which is less than the first number.

[0071] The coding feature determination module is used to determine the coding features corresponding to the first sub-image features based on the first sub-image features after convolution processing and multiple second sub-image features after convolution processing.

[0072] In one possible implementation, the device further includes:

[0073] The location encoding module is used to encode the location information of each of the plurality of sub-images to obtain location features, and to embed the location features into the sub-image features of the sub-image. The location information is used to represent the location of the sub-image in the medical image.

[0074] In one possible implementation, the sub-image feature extraction module is used to compress the size of the initial feature map of each sub-image until a feature map of the target size is obtained; and to compress the feature map of the target size into a sub-image feature of size 1×1.

[0075] In one possible implementation, the plurality of sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image, constituting a feature map of the medical image, wherein the number of channels in the feature map is a first quantity; the device further includes:

[0076] The feature map compression module is used to perform convolution processing on the feature map of the medical image to obtain a convolution-processed feature map. The number of channels in the convolution-processed feature map is a second number, which is less than the first number.

[0077] The encoding feature determination module is used to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature in the convolutional feature map and multiple second sub-image features in the convolutional feature map.

[0078] In one possible implementation, the image processing model includes a feature extraction network and a feature processing network, wherein the feature processing network includes an encoder and a classifier;

[0079] The sub-image feature extraction module is used to call the feature extraction network to extract features from the multiple sub-images respectively, thereby obtaining multiple sub-image features;

[0080] The encoding feature determination module is used to call the encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features;

[0081] The state recognition module is used to call the classifier to determine the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features.

[0082] In one possible implementation, the encoder includes a first self-attention unit; the encoding feature determination module is used to call the first self-attention unit to perform cross-processing on the query features of the first sub-image features and the key features and value features of each second sub-image feature to obtain the encoding features corresponding to the first sub-image features.

[0083] In one possible implementation, the encoder further includes a first feature update unit;

[0084] The device further includes:

[0085] The third coding feature update module is used to call the first feature update unit to update the coding features corresponding to the first sub-image features, so as to obtain the updated coding features corresponding to the first sub-image features.

[0086] In one possible implementation, the feature processing network includes multiple encoders;

[0087] The encoding feature determination module is used to call a first encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features; and to call a second encoder connected after the first encoder to determine the updated encoding feature corresponding to the first sub-image feature based on the encoding feature corresponding to the first sub-image feature and multiple encoding features respectively corresponding to the multiple second sub-image features.

[0088] In one possible implementation, the feature processing network further includes a decoder, which includes a second self-attention unit and a decoding unit;

[0089] The status recognition module includes:

[0090] The first decoding submodule is used to call the second self-attention unit to determine the first reference query feature based on the initial identifier feature; and to call the decoding unit to perform cross-processing on the first reference query feature and the key and value features of the plurality of encoded features to obtain the decoding feature, wherein the initial identifier feature is used to indicate the start of decoding.

[0091] The second decoding submodule is used to call the second self-attention unit to determine the second reference query feature based on the initial identifier feature and the obtained decoding feature; call the decoding unit to perform cross-processing on the second reference query feature and the key and value features of the multiple encoded features to obtain the next decoding feature; until the obtained decoding feature indicates the end of decoding;

[0092] The state recognition submodule is used to call the classifier and determine the state recognition result of the target object based on the obtained multiple decoded features.

[0093] In one possible implementation, the decoder further includes a second feature update unit, and the apparatus further includes:

[0094] The third decoding feature update module is used to call the second feature update unit to update the decoding feature each time a decoding feature is obtained, so as to obtain the updated decoding feature.

[0095] In one possible implementation, the device further includes:

[0096] The second decoding feature update module is used to call the second decoder connected after the first decoder after each call to obtain the decoding features, determine the third reference query feature based on the obtained decoding features, and perform cross processing on the third reference query feature and the key features and value features of the multiple encoded features to obtain the updated decoding features.

[0097] In one possible implementation, the number of channels for each sub-image feature is a first quantity, and the feature processing network further includes a feature compressor;

[0098] The device further includes:

[0099] The feature compression module is used to call the feature compressor to perform convolution processing on each of the sub-image features to obtain convolution-processed sub-image features. The number of channels of the convolution-processed sub-image features is a second number, which is less than the first number.

[0100] The encoding feature determination module is used to call the encoder to determine the encoding features corresponding to the first sub-image features based on the first sub-image features after convolution and multiple second sub-image features after convolution.

[0101] In one possible implementation, the plurality of sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image to form a feature map of the medical image, wherein the number of channels of the feature map is a first quantity; the feature processing network further includes a feature compressor;

[0102] The device further includes:

[0103] The feature map compression module is used to call the feature compressor to perform convolution processing on the feature map of the medical image to obtain a convolution-processed feature map. The number of channels in the convolution-processed feature map is a second number, which is less than the first number.

[0104] The encoding feature determination module is used to call the encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature in the convolutional feature map and multiple second sub-image features in the convolutional feature map.

[0105] In one possible implementation, the sub-image feature extraction module is used to call the feature extraction network to compress the size of the initial feature map of each sub-image until a feature map of the target size is obtained; and to compress the feature map of the target size into a sub-image feature of size 1×1.

[0106] In one possible implementation, the training process of the feature processing network includes:

[0107] Multiple sub-image features of a sample medical image and corresponding annotation information of the sample medical image are obtained, wherein the annotation information is used to represent the state of the target object;

[0108] The feature processing network is invoked to process multiple sub-image features of the sample medical image to obtain the sample recognition result;

[0109] The feature processing network is trained based on the sample recognition results and the annotation information.

[0110] On one hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the image processing method described in any of the above possible implementations.

[0111] On one hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the image processing method described in any of the possible implementations above.

[0112] On the one hand, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the image processing method described in any of the above possible implementations.

[0113] The technical solution provided in this application divides a medical image containing a target object into multiple sub-images. Considering the combined influence of multiple sub-images on the state recognition result of the target object, for each sub-image feature, multiple sub-image features are encoded into the sub-image feature, thereby achieving effective integration of sub-image features and obtaining a more accurate feature representation. Based on the more accurate feature representation, a more accurate state recognition result can be obtained, thus improving the accuracy of state recognition. Attached Figure Description

[0114] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0115] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application;

[0116] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0117] Figure 3 This is a schematic diagram of a feature map provided in an embodiment of this application;

[0118] Figure 4 This is a schematic diagram of the structure of an image processing model provided in an embodiment of this application;

[0119] Figure 5 This is a flowchart of an image processing method provided in an embodiment of this application;

[0120] Figure 6This is a schematic diagram of the structure of an image processing model provided in an embodiment of this application;

[0121] Figure 7 This is a flowchart of an image processing method provided in an embodiment of this application;

[0122] Figure 8 This is a flowchart of an image processing method provided in an embodiment of this application;

[0123] Figure 9 This is a flowchart of an image processing method provided in an embodiment of this application;

[0124] Figure 10 This is a block diagram of an image processing apparatus provided in an embodiment of this application;

[0125] Figure 11 This is a block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0126] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0127] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first sub-image feature may be referred to as a second sub-image feature, and a second sub-image feature may be referred to as a first sub-image feature.

[0128] As used in this application, the terms "at least one," "multiple," "each," and "any" mean, respectively, with "at least one" including one, two, or more, "multiple" including two or more, "each" referring to each of the corresponding multiple, and "any" referring to any one of the multiple. For example, multiple sub-images include three sub-images, with "each sub-image" referring to every single one of the three sub-images, and "any" referring to any one of the three sub-images, which could be the first, the second, or the third.

[0129] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0130] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0131] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0132] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0133] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0134] The solutions provided in this application involve artificial intelligence technologies such as computer vision and machine learning, which are specifically illustrated through the following embodiments.

[0135] The image processing method provided in this application is executed by a computer device. In some embodiments, the computer device is a terminal. The terminal may be a medical terminal, desktop computer, laptop computer, tablet computer, or smartphone, but is not limited thereto. In some embodiments, the computer device is a server. Optionally, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0136] In one possible implementation, a computer program for implementing the image processing method provided in the embodiments of this application is deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network. The multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0137] In one possible implementation, the computer device is a node in the blockchain system. The node executes the image processing method provided in the embodiments of this application to obtain the state recognition result of the target object. Then, the node or other nodes in the blockchain can store the state recognition result of the target object.

[0138] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application. The following is in conjunction with… Figure 1 This section provides a brief description of the image processing method, which is executed by a computer device and includes the following steps:

[0139] 101. The computer device divides a medical image containing the target object into multiple sub-images, extracts features from each sub-image, and obtains features from the sub-images.

[0140] The target object is the tissue or site used to represent a pathological condition. Due to the large size of medical images, to obtain more accurate feature representations, the medical image is first divided into multiple sub-images, and then feature extraction is performed on each sub-image to obtain multiple sub-image features. Specifically, the computer extracts features from each sub-image to obtain its sub-image features. One sub-image corresponds to one sub-image feature, and the number of sub-image features is equal to the number of sub-images.

[0141] 102. The computer device determines the coding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features.

[0142] A single sub-image contains local information about the target object, and the corresponding sub-image features represent independent information within that sub-image. Different sub-images are correlated, and these correlated sub-images collectively influence the state recognition result of the target object. Therefore, the computer device effectively integrates multiple sub-image features to obtain a more accurate feature representation, which in turn leads to a more accurate state recognition result. In this step, the computer device encodes multiple second sub-image features into first sub-image features, obtaining the encoded features corresponding to the first sub-image features, thus achieving effective integration of multiple sub-image features.

[0143] The first sub-image feature is any one of multiple sub-image features, and the multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature.

[0144] This step illustrates the process using the example of a computer device determining the encoded feature corresponding to a first sub-image feature. Furthermore, the computer device will determine the encoded feature corresponding to each sub-image feature using a similar process to determining the encoded feature corresponding to the first sub-image feature, thus obtaining multiple encoded features corresponding to multiple sub-image features. Here, one sub-image feature corresponds to one encoded feature, and the number of multiple encoded features is equal to the number of sub-image features.

[0145] 103. Based on multiple encoded features corresponding to multiple sub-image features, determine the state recognition result of the target object.

[0146] The state recognition result is used to indicate the state of the target object. Optionally, the state recognition result is used to indicate whether the target object is in a normal state or an abnormal state; or, the state recognition result is used to indicate the abnormal category of the target object; or, the state recognition result is the abnormal transition probability of the target object, where the abnormal transition probability represents the probability that a target object in an abnormal state will cause other objects to change to an abnormal state.

[0147] The technical solution provided in this application divides a medical image containing a target object into multiple sub-images. Considering the combined influence of multiple sub-images on the state recognition result of the target object, for each sub-image feature, multiple sub-image features are encoded into the sub-image feature, thereby achieving effective integration of sub-image features and obtaining a more accurate feature representation. Based on the more accurate feature representation, a more accurate state recognition result can be obtained, thus improving the accuracy of state recognition.

[0148] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application. The following is in conjunction with… Figure 2 The image processing method is described in detail below. This image processing method is executed by a computer device and includes the following steps:

[0149] 201. The computer device segments a medical image containing the target object into multiple sub-images.

[0150] The target object is a tissue or site used to represent a pathological condition. In some embodiments, the target object is a tissue of the body. Optionally, the target object can be any site of the body, such as breast tissue, colon tissue, rectal tissue, or lymphatic tissue. In some embodiments, the target object is a site of the body, such as the lungs or stomach. Optionally, the medical image is acquired by a computer device; or, the medical image is acquired by other devices and then sent to the computer device; this embodiment of the application does not impose any limitations on this.

[0151] In some embodiments, the target object is tissue of the organism, and the medical image containing the target object is obtained by image acquisition of pathological sections. Pathological sections are prepared by obtaining tissue of a certain size and using pathological histological methods. For example, the preparation process of a pathological section includes: embedding or freezing the obtained tissue in paraffin, slicing it into thin sections; adhering the thin sections to a glass slide, and staining them to obtain a pathological section. In some embodiments, the medical image is obtained by scanning the pathological section with a digital slide scanner; the medical image is also referred to as a WSI (Whole Slide Image). Optionally, the pathological section is a biopsy pathological section or a gross section.

[0152] In some embodiments, the target object is a part of the body, and the medical image containing the target object is obtained by X-ray irradiation; or, it is obtained by other means such as endoscopic imaging, and the embodiments of this application do not limit this.

[0153] In some embodiments, the computer device segments a medical image into sub-images of size n×m. Optionally, the computer device uses a sliding window of size n×m to slide within the medical image to obtain sub-images of size n×m. Here, n and m represent the number of pixels, and both n and m are positive integers. Optionally, n and m are equal, for example, both n and m are 512.

[0154] In some embodiments, to obtain a more detailed feature representation, the computer device magnifies the medical image at a target magnification to obtain a medical image at the target magnification; the medical image at the target magnification is then segmented into multiple sub-images. The target magnification can be flexibly configured, for example, a target magnification of 10x or 20x. In some embodiments, the computer device magnifies the medical image at different magnifications to obtain sub-images segmented at different magnifications; based on multiple sub-images at the same magnification, the state recognition result of the target object is determined; and the multiple state recognition results corresponding to the multiple magnifications are weighted and summed to obtain the final state recognition result.

[0155] 202. The computer equipment extracts features from multiple sub-images to obtain features of multiple sub-images.

[0156] In some embodiments, for each of the plurality of sub-images, the computer device performs feature extraction on the sub-image to obtain a one-dimensional feature with a first number of channels, that is, to obtain a 1×1×D sub-image feature. D is the first number, and D is a positive integer, for example, D is 1280.

[0157] In some embodiments, the computer device compresses the size of an initial feature map of a sub-image until a feature map of a target size is obtained. The target-size feature map is then compressed into a sub-image feature of size 1×1, which is a one-dimensional feature with a first number of channels. Specifically, the computer device compresses the initial feature map to obtain a feature map of a first size, where the number of channels in the first-size feature map is greater than the number of channels in the initial feature map. If the first size is greater than the target size, the size of the first-size feature map is further compressed until the feature map of the target size is obtained.

[0158] The initial feature map is the initial feature representation of the sub-image. For example, the initial feature map consists of features of the sub-image in three channels: R (Red), G (Green), and B (Blue). The initial feature map has a size of 512×512 and three channels. The target size can be flexibly configured; for example, the target size is 7×7. To preserve the information contained in the sub-image, the number of channels in the target-sized feature map will be greater than the number of channels in the initial feature map. Optionally, the number of channels in the target-sized feature map is equal to the number of channels in the 1×1 sub-image features.

[0159] In some embodiments, after obtaining the sub-image features of each sub-image, the computer device further performs dimensionality reduction on each sub-image feature, that is, reduces the number of channels of the sub-image feature to reduce the amount of feature data. Subsequent processing of the dimensionality-reduced sub-image features can improve processing efficiency.

[0160] In some embodiments, the number of channels for each sub-image feature is a first number. The computer device performs convolution processing on the sub-image features to obtain convolutionally processed sub-image features. The convolutionally processed sub-image features are the dimensionality-reduced sub-image features. The number of channels for the convolutionally processed sub-image features is a second number, which is less than the first number. The second number is a positive integer and can be flexibly configured; for example, the second number is 512. After obtaining the convolutionally processed sub-image features, the computer device performs subsequent processing steps on them. That is, based on the convolutionally processed first and second sub-image features, it determines encoding features, or encodes the positional information of the sub-image to obtain positional features, and embeds these positional features into the convolutionally processed sub-image features of that sub-image.

[0161] In this system, each sub-image shares the same positional information with its corresponding sub-image features. This positional information indicates the sub-image's location within the medical image. In some embodiments, after obtaining the sub-image features of each sub-image, the computer device arranges the sub-image features according to their positions within the medical image, resulting in a feature map of the medical image. The position of a sub-image feature in the feature map is the same as the position of the corresponding sub-image in the medical image. For example, if a sub-image's position in the medical image is represented as (2, 3), it means that the sub-image is located in the 2nd row and 3rd column among the multiple sub-images constituting the medical image. Correspondingly, the position of the sub-image feature in the feature map is also (2, 3). See also Figure 3 The position of sub-image feature 311 of sub-image 301 in feature map 310 is the same as the position of sub-image 301 in medical image 300, and the same applies to other sub-images and their corresponding sub-image features.

[0162] In some embodiments, the computer device performs convolution processing on the feature map of a medical image to obtain a convolutionally processed feature map. After obtaining the convolutionally processed feature map, the computer device performs subsequent processing steps on the sub-image features in the convolutionally processed feature map. That is, based on the first sub-image features and the second sub-image features in the convolutionally processed feature map, it determines the encoded features, or encodes the positional information of the sub-image to obtain positional features, and embeds the positional features into the sub-image features of the sub-image in the convolutionally processed feature map. The number of channels in the feature map before convolution processing is a first number, and the number of channels in the feature map after convolution processing is a second number, where the second number is less than the first number. It should be noted that the computer device performs convolution processing on the feature map of the medical image, that is, it performs convolution processing on each sub-image feature in the feature map separately.

[0163] The above technical solution reduces the number of channels in sub-image features by performing convolution processing on the sub-image features. This not only improves the efficiency of subsequent feature processing but also retains effective features that have a positive impact on the accuracy of state recognition results, thus ensuring the accuracy of state recognition.

[0164] 203. For each of the multiple sub-images, the computer device encodes the position information of the sub-image to obtain position features, and embeds the position features into the sub-image features of the sub-image.

[0165] The location information of the sub-image is used to represent its position within the medical image. The computer device encodes the location information of the sub-image to obtain location features with the same number of channels as the sub-image features. Each location feature consists of feature values ​​across multiple channels, and each sub-image feature also consists of feature values ​​across multiple channels. For each channel, the computer device embeds the location feature into the sub-image feature by adding the feature value of the location feature in that channel to the feature value of the sub-image feature in that channel.

[0166] In some embodiments, for any sub-image and any channel of its sub-image features, if the channel number is even, the positional information of the sub-image is processed based on a sine function to obtain the feature value of the positional feature in that channel; if the channel number is odd, the positional information of the sub-image is processed based on a cosine function to obtain the feature value of the positional feature in that channel. That is, the positional information of a sub-image is encoded using the following formula to obtain the positional feature:

[0167] Formula 1:

[0168] Where pos represents the position of the sub-image in the medical image; i represents the i-th channel, i is a positive integer or 0; j is a positive integer or 0; i = 2j means i is even; i = 2j + 1 means i is odd; Ω represents frequency, which can be flexibly configured as needed. For example, Ω can be equal to the number of channels of the position feature; PE(pos,i) represents the feature value of the position feature corresponding to the sub-image at position pos in the i-th channel.

[0169] In this context, the location information of the sub-image within the medical image is represented by two-dimensional coordinates, comprising two coordinate values. In some embodiments, the computer device encodes these two coordinate values ​​into different channels of the location feature. That is, the location feature includes multiple channels, and the computer device encodes one coordinate value into a subset of the location feature's channels and the other coordinate value into another subset of the location feature's channels. For example, if the location feature is a 100-dimensional feature with 100 channels, the horizontal coordinate value can be encoded into the first 50 channels, and the vertical coordinate value into the last 50 channels. In other words, in Formula 1 above, when i is not greater than 50, pos takes the horizontal coordinate value from the location information. When i is greater than 50, pos takes the vertical coordinate value from the location information.

[0170] In some embodiments, pos is a numerical value obtained by encoding the two coordinate values ​​mentioned above. Decoding pos according to a decoding method opposite to the encoding method can restore the two coordinate values. For example, a medical image is divided into x×y sub-images, and these x×y sub-images are arranged in the form of x rows and y columns to form a medical image. The pos of multiple sub-images is encoded as multiple positive integers between 1 and x×y. Correspondingly, when pos%y is 0, the horizontal coordinate of the sub-image is pos / y, and the vertical coordinate of the sub-image is y; when pos%y is not 0, the horizontal coordinate of the sub-image is pos / y+1, and the vertical coordinate of the sub-image is pos%y. Here, pos%y represents the remainder obtained by dividing pos by y, and x and y are both positive integers.

[0171] The above-described method of encoding and embedding location information into sub-image features is one example. Computer devices can also embed location information into sub-image features in other ways, and this application does not limit this.

[0172] The above technical solution also integrates location information to represent the features of the sub-images. In the subsequent processing of the sub-image features, it can combine the positional correlation between features to perform state recognition and improve the accuracy of state recognition.

[0173] In some embodiments, after the computer device performs step 202 and obtains multiple sub-image features, it may not perform step 203. That is, it may not embed the location information into the sub-image features, but instead directly determine the encoded features based on the sub-image features obtained in step 202.

[0174] 204. The computer device determines the coding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features.

[0175] Wherein, the first sub-image feature is any one of multiple sub-image features, and the multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature. Optionally, the first sub-image feature and the second sub-image feature are sub-image features obtained by feature extraction of the sub-image; or, the first sub-image feature and the second sub-image feature are sub-image features obtained by feature extraction of the sub-image followed by convolution processing; or, the first sub-image feature and the second sub-image feature are sub-image features obtained by feature extraction of the sub-image followed by positional information encoding; or, the first sub-image feature and the second sub-image feature are sub-image features obtained by feature extraction of the sub-image followed by convolution processing and then positional information encoding.

[0176] In some embodiments, the plurality of second sub-image features refer to the sub-image features of a medical image divided into a plurality of sub-images.

[0177] In some embodiments, the plurality of second sub-image features refer to a portion of the sub-image features among the sub-image features of a medical image divided into multiple sub-images. That is, the plurality of second sub-image features include the first sub-image feature and the associated sub-image features of the first sub-image feature. Before determining the encoded features corresponding to the first sub-image feature based on the first sub-image feature and the plurality of second sub-image features, the computer device also determines the associated sub-image features of the first sub-image feature. The step of the computer device determining the associated sub-image features of the first sub-image feature includes: the computer device determining a second position associated with the first position based on a first position of the first sub-image in the medical image, wherein the first sub-image feature is obtained by feature extraction of the first sub-image; and determining the sub-image feature corresponding to the second sub-image located at the second position as the associated sub-image feature.

[0178] In some embodiments, the computer device determines a second location associated with a first location from the association between locations. In some embodiments, the computer device determines a location offset associated with the first location from the association between locations and location offsets; the location offset is then added to the first location to obtain the second location.

[0179] The above technical solution determines a small number of key sub-image features based on the correlation between locations, and performs feature encoding based on these key sub-image features. This reduces the amount of data for feature encoding, lowers computational complexity, and improves feature processing efficiency.

[0180] In some embodiments, the computer device, based on a self-attention mechanism, encodes multiple second sub-image features into the first sub-image feature, thereby obtaining the encoded features corresponding to the first sub-image feature. That is, the computer device performs cross-processing on the query features of the first sub-image feature and the key and value features of each second sub-image feature to obtain the encoded features corresponding to the first sub-image feature.

[0181] It should be noted that the query feature, key feature, and value feature of any feature are obtained by decomposing the feature obtained by multiplying the feature with the parameter matrix. In the self-attention mechanism, the query feature of a feature is used to query the relevance of the feature with other features, and the key feature of the feature is used to query the corresponding relevance of other features. The dot product of the query feature and the key feature represents the relevance between the feature to which the query feature belongs and the feature to which the key feature belongs. The value feature of a feature represents the content represented by the feature. Based on the relevance represented by the query feature and the key feature, the value feature is weighted and fused to obtain a feature that incorporates the relevance between features.

[0182] The query features, key features, and value features of the sub-image features are obtained by processing the sub-image features based on the first parameter matrix. The first parameter matrix is ​​a 3-dimensional parameter matrix, which includes three sub-parameter matrices. The sub-image features are multiplied by these three sub-parameter matrices respectively to obtain the query features, key features, and value features.

[0183] Cross-processing refers to the weighted fusion of value features based on the relevance parameters obtained by the dot product of query features and key features. The process by which a computer device performs cross-processing on the query features of the first sub-image feature and the key and value features of each second sub-image feature to obtain the encoded features corresponding to the first sub-image feature is as follows: the computer device determines the relevance parameters between the first sub-image feature and each second sub-image feature based on the query features of the first sub-image feature and the key features of each second sub-image feature; and then, according to the multiple relevance parameters corresponding to each of the multiple second sub-image features, weighted fusion of the value features of the multiple second sub-image features is performed to obtain the encoded features corresponding to the first sub-image feature.

[0184] The process of determining the encoded features corresponding to each sub-image feature by the computer equipment is similar. Therefore, the query features, key features, and value features of multiple sub-image features can be used to construct query matrices, key matrices, and value matrices, respectively. By processing the query matrix, key matrix, and value matrix, the encoded matrix can be obtained. This encoded matrix includes multiple encoded features corresponding to multiple sub-image features. The above process is also the process by which the computer equipment performs feature encoding using the following formula two:

[0185] Formula 2:

[0186] Where Att(Q,K,V) represents the encoding matrix; Q represents the query matrix; K T V represents the transpose of the key matrix; V represents the value matrix; d k The dimension of the key feature, that is, the number of feature values ​​included in the key feature, d k It is a positive integer; softmax(·) represents normalization.

[0187] It should be noted that the above embodiments are illustrated using a computer device performing feature encoding based on a single first parameter matrix, that is, using a computer device performing feature encoding based on a single-head attention mechanism. In some embodiments, the computer device performs feature encoding based on a multi-head attention mechanism, that is, based on multiple first parameter matrices. Each first parameter matrix contains different parameters, and different first parameter matrices can capture information at different levels, resulting in different feature representations.

[0188] A computer device, based on a first parameter matrix, can obtain the query feature, key feature, and value feature corresponding to a sub-image feature. Then, by cross-processing the query feature of the first sub-image feature and the key and value features of each second sub-image feature, it can obtain a single encoded feature corresponding to the first sub-image feature. Therefore, when multiple first parameter matrices exist, multiple encoded features corresponding to the first sub-image feature will be obtained. After obtaining these multiple encoded features, they need to be integrated to obtain a single encoded feature corresponding to the first sub-image feature.

[0189] The process by which the computer device integrates multiple coded features corresponding to the first sub-image feature to obtain a single coded feature includes: the computer device concatenating and compressing the multiple coded features corresponding to the first sub-image feature obtained based on multiple first parameter matrices to obtain an updated coded feature corresponding to the first sub-image feature; subsequently, the state recognition result is determined based on this updated coded feature. In some embodiments, the computer device concatenates the above multiple coded features to obtain a concatenated feature; and multiplies the concatenated feature by a compression parameter matrix to obtain an updated coded feature. The number of rows in the compression parameter matrix is ​​equal to the number of columns in the concatenated feature, and the number of columns in the compression parameter matrix is ​​the same as the number of columns in the sub-image feature.

[0190] The process by which the computer device determines the encoded features corresponding to the first sub-image features based on the multi-head attention mechanism is represented by the following formula three:

[0191] Formula 3:

[0192] MHA(Q,K,V)=Concat(Head1,…,Head M W O

[0193] Head i =Att(QW i Q KW i K VW i V )

[0194] Where MHA(Q, K, V) represents the encoded features obtained after concatenating and compressing multiple encoded features; Concat(·) represents concatenation; Head i W represents the encoded feature obtained based on the i-th first parameter matrix, where i is a positive integer or 0; M represents the total number of first parameter matrices, where M is a positive integer; O W represents the compression parameter matrix. i Q W represents the sub-parameter matrix in the i-th first parameter matrix used to determine the query features; i K W represents the sub-parameter matrix used to determine the key features in the i-th first parameter matrix; i V This represents the sub-parameter matrix used to determine the value characteristics in the i-th first parameter matrix.

[0195] In some embodiments, after determining the multiple coded features corresponding to the multiple sub-image features, the computer device further encodes the obtained coded features to improve the accuracy of the feature representation. That is, the computer device determines the updated coded features corresponding to the first sub-image feature based on the coded features corresponding to the first sub-image feature and the multiple coded features corresponding to the multiple second sub-image features. The process of the computer device determining the updated coded features corresponding to the first sub-image feature based on the coded features corresponding to the first sub-image feature and the multiple coded features corresponding to the multiple second sub-image features is the same as the process of the computer device determining the coded features corresponding to the first sub-image feature based on the first sub-image feature and the multiple second sub-image features, and will not be described again here.

[0196] In some embodiments, after obtaining the coded features corresponding to the sub-image features, the computer device further decodes the obtained multiple coded features to obtain a more accurate feature representation. The process of the computer device decoding multiple coded features is described below.

[0197] 205. The computer device determines the first reference query feature based on the initial identification feature.

[0198] During the decoding process, the computer device uses a cross-attention mechanism to perform attention operations across both the encoding and decoding parts. This integrates the encoded features into the decoding process; that is, it uses the already obtained decoded features as a reference to determine a reference query feature. Based on the reference query feature, it decodes multiple encoded features to obtain the decoded feature. It's important to note that no decoded features exist before the first decoded feature is obtained. Therefore, an initial identifier feature is set to indicate the start of decoding. The computer device uses this initial identifier feature to perform the first decoding of multiple encoded features, obtaining the first decoded feature. This initial identifier feature is also called a token.

[0199] In some embodiments, the computer device performs an autoregressive process on the initial identifier features based on a self-attention mechanism to obtain a first reference query feature. That is, the computer device multiplies the third parameter matrix with the initial identifier features to obtain the query features, key features, and value features of the initial identifier features. The query features, key features, and value features of the initial identifier features are then cross-processed to obtain the first reference query feature.

[0200] The above embodiments illustrate the use of a computer device determining reference query features based on a single-head attention mechanism. In some embodiments, the computer device determines reference query features based on a multi-head attention mechanism, that is, based on multiple third-parameter matrices. Each third-parameter matrix contains different parameters, and different third-parameter matrices can capture information at different levels, resulting in different feature representations.

[0201] The computer device, based on a third parameter matrix, can determine a first reference query feature corresponding to the initial identifier feature. When multiple third parameter matrices exist, multiple first reference query features corresponding to the initial identifier feature are obtained. The computer device further concatenates and compresses these multiple first reference query features to obtain an updated first reference query feature corresponding to the initial identifier feature. Subsequent feature decoding is based on this updated first reference query feature. The process of concatenating and compressing multiple first reference query features is similar to the process of concatenating and compressing multiple encoded features corresponding to sub-image features, and will not be elaborated further here.

[0202] 206. The computer device performs cross-processing on the first reference query feature and the key and value features of multiple encoded features to obtain the decoded features.

[0203] The computer device performs a dot product of the first reference query feature and the key feature of each encoded feature to obtain the inner product corresponding to each encoded feature. Then, the inner product corresponding to each encoded feature is transformed by the softmax function to obtain the relevance parameter corresponding to each encoded feature. Subsequently, according to the multiple relevance parameters corresponding to multiple encoded features, the value features of multiple encoded features are weighted and fused to obtain the decoded features.

[0204] In some embodiments, the computer device processes multiple encoded features based on a single-head attention mechanism to obtain a decoded feature. The key and value features of the encoded feature are obtained based on a second parameter matrix.

[0205] In some embodiments, the computer device processes multiple encoded features based on a multi-head attention mechanism to obtain decoded features. That is, it processes multiple encoded features based on multiple second parameter matrices to obtain decoded features. Specifically, the computer device obtains one decoded feature based on one second parameter matrix; when multiple second parameter matrices exist, multiple decoded features are obtained. After obtaining multiple decoded features, the computer device further concatenates and compresses these features based on the multiple second parameter matrices to obtain updated decoded features, which are then applied for subsequent processing. The process of concatenating and compressing multiple decoded features is similar to the process of concatenating and compressing multiple encoded features corresponding to sub-image features, and will not be elaborated further here.

[0206] 207. The computer device determines a second reference query feature based on the initial identification feature and the obtained decoding feature. It then performs cross-processing on the second reference query feature and the key and value features of multiple encoded features to obtain the next decoding feature, until the obtained decoding feature indicates the end of the decoding process.

[0207] After obtaining the first decoded feature, the computer device can process the initial identifier feature and the obtained decoded features based on a self-attention mechanism to obtain a reference query feature. The steps for the computer device to determine the second reference query feature based on the initial identifier feature and the obtained decoded features include: the computer device determining the relevance parameter corresponding to the initial identifier feature based on the query feature of the latest obtained decoded features and the key feature of the initial identifier feature; and determining the relevance parameter corresponding to each obtained decoded feature based on the query feature of the latest obtained decoded features and the key feature of each obtained decoded feature; and weightedly fusing the value feature of the initial identifier feature and the value feature of each obtained decoded feature according to the relevance parameter corresponding to the initial identifier feature and the relevance parameter corresponding to each obtained decoded feature to obtain the second reference query feature.

[0208] In some embodiments, the computer device processes the initial identifier features and the obtained decoded features based on a single-head attention mechanism to obtain a second reference query feature. The query features, key features, and value features of the initial identifier features, as well as the query features, key features, and value features of the decoded features, are obtained based on a third parameter matrix.

[0209] In some embodiments, the computer device processes the initial identification features and the obtained decoded features based on a multi-head attention mechanism. That is, it processes the initial identification features and the obtained decoded features based on multiple third parameter matrices to obtain second reference query features. Each third parameter matrix corresponds to one second reference query feature. When multiple third parameter matrices exist, multiple second reference query features are obtained. The computer device then concatenates and compresses these multiple second reference query features obtained from the multiple third parameter matrices to obtain updated second reference query features. Subsequently, based on these updated second reference query features, multiple encoded features are processed to obtain decoded features.

[0210] The process by which the computer device performs cross-processing on the second reference query feature and the key and value features of multiple encoded features to obtain the next decoded feature is the same as the process by which the computer device performs cross-processing on the first reference query feature and the key and value features of multiple encoded features to obtain the decoded feature, and will not be described again here.

[0211] For each decoded feature obtained, the computer device executes the steps described above: based on the initial identifier feature and the previously obtained decoded features, determine the second reference query feature; perform cross-processing on the second reference query feature and the key and value features of multiple encoded features to obtain the next decoded feature; and continue until the obtained decoded feature indicates the end of decoding. The computer device can identify the feature that indicates the end of decoding. If it identifies the latest obtained decoded feature as indicating the end of decoding, it will no longer use previously obtained decoded features as a reference to determine the next decoded feature.

[0212] The above technical solution, after obtaining the encoded features, also spans both the encoding and decoding parts, using the obtained decoded features as a reference to process multiple encoded features to obtain the next decoded feature. This fully integrates the correlation between features and performs feature representation, further improving the accuracy of feature representation.

[0213] In some embodiments, after obtaining decoded features each time, the computer device further updates the obtained decoded features to improve the accuracy of the feature representation. The step of updating the obtained decoded features by the computer device includes: after obtaining decoded features each time, the computer device determines a third reference query feature based on the obtained decoded features; cross-processes the third reference query feature and the key features and value features of multiple encoded features to obtain updated decoded features; and subsequently, based on the updated decoded features, determines the state recognition result of the target object.

[0214] 208. The computer device determines the state recognition result of the target object based on the multiple decoded features obtained.

[0215] In some embodiments, the state recognition result indicates whether the target object is in a normal or abnormal state. A normal state indicates that the target object has not changed, while an abnormal state indicates that the target object has changed relative to its normal state. For example, in a scenario where the target object is identified as being diseased, a normal state indicates that the target object is not diseased, while an abnormal state indicates that the target object is diseased.

[0216] In some embodiments, the state identification result indicates the nature of the lesion in the target object. The state identification result is either a first category or a second category. The first category indicates that the lesion in the target object is benign, and the second category indicates that the lesion in the target object is malignant. Benign indicates that the lesion in the target object is developing slowly and has a relatively small negative impact on the body's health, while malignant indicates that the lesion in the target object is developing rapidly and has a high risk of metastasis to other parts of the body, resulting in a significant negative impact on the body's health.

[0217] In some embodiments, the state recognition result indicates the anomalous category of the target object. For example, if the target object is lung tissue, the state recognition result indicates that the anomalous category of the target object is non-small cell lung cancer or small cell lung cancer.

[0218] In some embodiments, the state recognition result is the abnormal metastasis probability of the target object, which represents the probability that a target object in an abnormal state will cause other objects to become abnormal. For example, in the scenario of predicting the metastasis probability of cancer cells, the abnormal metastasis probability represents the probability that cancer cells will reach other sites and continue to grow, causing lesions in those sites. The image processing method provided in this application embodiment can be used to predict the metastasis probability of cancer cells in cancers such as breast cancer, colon cancer, or rectal cancer, and can also be used to predict the metastasis probability of cancer cells in other cancer types. In addition, the image processing method provided in this application embodiment can be used to predict the probability of cancer cells metastasizing through the lymphatic system, and can also be used to predict the probability of cancer cells metastasizing through invasion or other means.

[0219] It should be noted that, in the medical field, the state recognition results obtained in the embodiments of this application are only a basis for assisting doctors in identifying diseases. For example, in the scenario of identifying whether cancer cells will metastasize, doctors can combine the state recognition results of the target object, other cancer-related information, and the patient's physical condition to determine the risk of cancer cell metastasis.

[0220] In the above embodiments, after obtaining multiple encoded features, the computer device further obtains multiple decoded features through steps 205 to 207, and then determines the state recognition result of the target object based on the obtained multiple decoded features. In some embodiments, after obtaining multiple encoded features, the computer device can directly determine the state recognition result of the target object based on the multiple encoded features. That is, the above steps 205 to 208 can be replaced by the following step: the computer device determines the state recognition result based on the multiple encoded features corresponding to the multiple sub-image features respectively.

[0221] The technical solution provided in this application divides a medical image containing a target object into multiple sub-images. Considering the combined influence of multiple sub-images on the state recognition result of the target object, for each sub-image feature, multiple sub-image features are encoded into the sub-image feature, thereby achieving effective integration of sub-image features and obtaining a more accurate feature representation. Based on the more accurate feature representation, a more accurate state recognition result can be obtained, thus improving the accuracy of state recognition.

[0222] Furthermore, by performing convolution processing on the sub-image features, the number of channels of the sub-image features can be reduced, which can not only improve the efficiency of subsequent feature processing, but also retain effective features that have a positive impact on the accuracy of state recognition results, thus ensuring the accuracy of state recognition.

[0223] Furthermore, location information is integrated to represent the features of sub-images. In subsequent processing of these features, the positional correlation between features can be combined for state recognition, further improving the accuracy of state recognition. In processing ultra-large medical images, state recognition can also be performed by combining the positional correlation between features, without losing the interactions between long-distance sub-images, thus ensuring the accuracy of state recognition.

[0224] Furthermore, based on the correlation between locations, a small number of key sub-image features are identified, and feature encoding is performed based on these key sub-image features. This reduces the amount of data for feature encoding, lowers computational complexity, and improves feature processing efficiency.

[0225] Furthermore, after obtaining the encoded features, the algorithm transcends both the encoding and decoding stages, using the obtained decoded features as a reference to process multiple encoded features and obtain the next decoded feature. This fully integrates the correlation between features, performs feature representation, and further improves the accuracy of feature representation. Then, based on the obtained multiple decoded features, the state recognition result is determined, further improving the accuracy of state recognition.

[0226] The above embodiment is illustrated by taking the example of a computer device directly processing a medical image containing a target object to obtain a state recognition result. In another embodiment, the computer device calls an image processing model to process the medical image to obtain a state recognition result. Figure 4 This is a schematic diagram of the structure of an image processing model provided in an embodiment of this application. See also... Figure 4 The image processing model includes a feature extraction network 41 and a feature processing network 42. The feature extraction network 41 is connected to the feature processing network 42. The feature extraction network 41 is used to extract features from the medical image, and the feature processing network 42 is used to process the features extracted by the feature extraction network 41. The feature processing network 42 includes an encoder 421 and a classifier 422. In some embodiments, the encoder 421 is connected to the feature extraction network 41, and the encoder 421 is also connected to the classifier 422. The encoder 421 is used to encode the features extracted by the feature extraction network 41, and the classifier 422 is used to determine the state recognition result based on the features encoded by the encoder 421.

[0227] In some embodiments, the encoder 421 includes a first self-attention unit 4211, which is used to perform feature encoding based on a self-attention mechanism. In some embodiments, the encoder 421 further includes a first feature update unit 4212, which is used to update the encoded features.

[0228] In some embodiments, the feature processing network 42 includes multiple encoders connected sequentially, with the first encoder connected to the feature extraction network 41 and the last encoder connected to the classifier 422. Each encoder after the first encoder updates the encoded features obtained by the previous encoder.

[0229] In some embodiments, the feature processing network 42 further includes a feature compressor 423, which is connected to the feature extraction network 41 and to the encoder 421. The feature compressor 423 is used to compress the features extracted by the feature extraction network 41, and the encoder 421 is used to encode the features compressed by the feature compressor 423.

[0230] The following is a detailed explanation of the process by which computer devices call the above image processing model to process medical images. Figure 5 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method is executed by a computer device and includes the following steps:

[0231] 501. The computer device calls the feature extraction network to extract features from multiple sub-images, thereby obtaining features from multiple sub-images.

[0232] Optionally, the feature extraction network is a convolutional neural network with a network structure of EfficientNet-B0, SqueezeNet, ResNet (Residual Network), DenseNet, InceptionNet (Deeper Network), or VGG (Visual Geometry Group).

[0233] In some embodiments, the feature extraction network is used to compress the size of the initial feature map of each sub-image until a feature map of the target size is obtained, and then compress the feature map of the target size into a sub-image feature of size 1×1.

[0234] 502. The computer device calls the feature compressor to perform convolution processing on each sub-image feature to obtain the convolution-processed sub-image features.

[0235] In this design, the number of channels in the sub-image features before convolution is a first number, and the number of channels in the sub-image features after convolution is a second number, where the second number is less than the first number. In some embodiments, the feature compressor is a 1×1 convolution, which is used to compress features with the first number of channels into features with the second number of channels. The number of input channels corresponding to the 1×1 convolution is the first number, and the number of output channels corresponding to the 1×1 convolution is the second number.

[0236] The above embodiments illustrate the use of a computer device calling a feature compressor to perform convolution processing on each sub-image feature. In some embodiments, the computer device calls a feature compressor to perform convolution processing on the feature map of a medical image to obtain a convolution-processed feature map, which includes multiple convolution-processed sub-image features.

[0237] In some embodiments, after executing step 501, the computer device may skip step 502 and directly execute step 503.

[0238] 503. The computer device calls the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features.

[0239] In some embodiments, the feature processing network includes an encoder, which is invoked by a computer device to determine the encoded features corresponding to a first sub-image feature. In some embodiments, the feature processing network includes multiple encoders connected sequentially. The computer device invokes a first encoder to determine the encoded features corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features; then invokes a second encoder connected after the first encoder to determine the updated encoded features corresponding to the first sub-image feature based on the encoded features corresponding to the first sub-image feature and multiple encoded features corresponding to the multiple second sub-image features. Similarly, a third encoder connected after the second encoder is invoked to determine the updated encoded features corresponding to the first sub-image feature based on the updated encoded features corresponding to the first sub-image feature and the updated encoded features corresponding to the multiple second sub-image features, and so on.

[0240] The process of the computer device calling each encoder to perform feature encoding is the same. The following description uses one encoder as an example to illustrate the feature encoding process. In some embodiments, the encoder includes a first self-attention unit. The computer device calls the first self-attention unit to perform cross-processing on the query features of the first sub-image features and the key and value features of each second sub-image feature to obtain the encoded features corresponding to the first sub-image features. The process of the computer device calling the first self-attention unit to determine the encoded features corresponding to the first sub-image features is the same as step 204 and will not be repeated here.

[0241] In some embodiments, the first self-attention unit has a special deformable sampling design. That is, for a given sub-image feature, instead of encoding all sub-image features into that sub-image feature, it dynamically samples and encodes a small number of key related sub-image features into that sub-image feature. In other words, interaction analysis is performed only on a small number of related sub-image features. The association between a given sub-image feature and the small number of key related sub-image features is learnable and is obtained through iterative optimization during model training.

[0242] The process by which a computer device calls a deformable first self-attention unit to obtain the encoded features of a sub-image is shown in the following formula four:

[0243] Formula 4:

[0244] Among them, MDSA(f q r q P i ) represents the sub-image feature P i The corresponding encoding feature, i is a positive integer or 0; f q Representing sub-image features P i Query characteristics; rq Representing sub-image features P i The corresponding position information; m represents the m-th attention head in the multi-head attention mechanism, m is not less than 0 and not greater than M; M represents the total number of attention heads in the multi-head attention mechanism, M is a positive integer; W m A represents the parameter matrix corresponding to the m-th attention head; mqk A represents the attention weight of the k-th sub-image feature corresponding to the m-th attention head. mqk A value not less than 0 and not greater than 1; K represents the total number of sub-image features involved in encoding, and K is a positive integer; Δr mqk This represents the k-th sub-image feature relative to sub-image feature P. i Position offset, k is a positive integer; W' m W' represents the sub-parameter matrix used to determine the value features corresponding to the m-th attention head; m P i (r q +Δr mqk ) represents the value feature of the k-th sub-image.

[0245] In some embodiments, the encoder further includes a first feature update unit. After the computer device calls the first self-attention unit to obtain the encoded features corresponding to the first sub-image features, it also calls the first feature update unit to update the encoded features corresponding to the first sub-image features, thereby obtaining the updated encoded features corresponding to the first sub-image features. In some embodiments, the network structure of the first feature update unit is a feedforward neural network.

[0246] In some embodiments, the encoder includes a residual network structure. The residual network structure includes a first self-attention unit and residual connections. The residual connection means that the input to the first self-attention unit skips the first self-attention unit and is connected to its output. Multiple sub-image features are processed by this residual network structure to obtain multiple encoded features. One encoded feature is a feature obtained by superimposing a sub-image feature with the encoded feature obtained by processing that sub-image feature through the first self-attention unit. In some embodiments, the encoder further includes a layer normalization unit. The layer normalization unit is connected to the residual network structure including the first self-attention unit and is used to perform layer normalization on the features. Accordingly, the process of a computer device calling the encoder to perform feature encoding is represented as H = LN(P i-1 +MDSA(P i-1 )); where P i-1 Represents the sub-image features, where i is a positive integer; MDSA(·) indicates encoding using the first self-attention unit; P i-1 +MDSA(P i-1) represents the above residual network structure; LN(·) represents layer normalization; H represents P i-1 The corresponding coding features.

[0247] In some embodiments, the residual network structure includes a first feature update unit and a residual connection. This residual connection means that the input to the first feature update unit skips the connection between the first feature update unit and its output. When an encoded feature is processed by this residual network structure, the output is the feature obtained by superimposing the original encoded feature with the updated version of the original encoded feature. In some embodiments, the encoder further includes a layer normalization unit. The layer normalization unit is connected to the residual network structure including the first feature update unit and is used to perform layer normalization on the features. Accordingly, the process of updating the encoded features by the computer device is represented as EB(P) i ) = LN(H + FFN(H)); where H represents P i-1 The corresponding encoded features, where i is a positive integer; FFN(·) indicates processing by a feedforward neural network; LN(·) indicates layer normalization; EB(P i ) represents the updated coding feature.

[0248] 504. The computer device calls the classifier and determines the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features.

[0249] In some embodiments, the classifier is a fully connected network that processes multiple encoded features to obtain a state recognition result for the target object. Optionally, the fully connected network includes multiple processing layers connected sequentially. The first processing layer processes the multiple encoded features to obtain a first processed feature, the second processing layer processes the first processed feature to obtain a second processed feature, and so on. The last processing layer processes the processed feature obtained from the previous processing layer to obtain a state recognition result.

[0250] The technical solution provided in this application calls an encoder based on a self-attention mechanism for feature encoding. It takes into account the combined influence of multiple sub-images on the state recognition result of the target object, realizes the effective integration of sub-image features, obtains a more accurate feature representation, and thus, based on the more accurate feature representation, can obtain a more accurate state recognition result, thereby improving the accuracy of state recognition.

[0251] Furthermore, the encoder's self-attention unit is implemented based on a multi-head self-attention mechanism, which integrates feature representations from different aspects, further improving the accuracy of feature representation.

[0252] Furthermore, the encoder's self-attention unit adopts a deformable sampling design, focusing only on the interaction between a small number of features, which reduces computational complexity and improves model computational efficiency.

[0253] Furthermore, encoders can increase model complexity through combination and stacking, thereby enabling them to process large-scale complex data, complete large-scale complex data prediction tasks, and improve model usability.

[0254] The above embodiments are illustrated using the example of a feature processing network 42 mainly comprising an encoder 421 and a classifier 422. In some embodiments, the feature processing network 42 further includes a decoder 424. See also Figure 6 The encoder 421 is connected to the feature extraction network 41, the encoder 421 is connected to the decoder 424, and the decoder 424 is connected to the classifier 422. The decoder 424 is used to decode the features encoded by the encoder 421, and the classifier 422 is used to determine the state recognition result based on the features decoded by the decoder 424. The decoder 424 includes a second self-attention unit 4241 and a decoding unit 4242. The second self-attention unit 4241 is used to determine the reference query features based on the obtained decoded features; the decoding unit 4242 is used to decode the features encoded by the encoder 421 based on the reference query features.

[0255] In some embodiments, the decoder 424 further includes a second feature update unit 4243, which is used to update the features decoded by the decoder 424.

[0256] In some embodiments, the feature processing network 42 includes multiple decoders connected sequentially, with the first decoder connected to the encoder 421 and the last decoder connected to the classifier 422. Each decoder after the first decoder updates the decoded features obtained by the previous decoder.

[0257] The following is a detailed explanation of the process by which computer devices call the above image processing model to process medical images. Figure 7 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method is executed by a computer device and includes the following steps:

[0258] 701. The computer device calls the feature extraction network to extract features from multiple sub-images, thereby obtaining features from multiple sub-images.

[0259] This step is the same as step 501.

[0260] 702. The computer device calls the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features.

[0261] This step is the same as step 502.

[0262] 703. The computer device invokes the second self-attention unit of the decoder to determine the first reference query feature based on the initial identification feature.

[0263] This step is the same as step 205.

[0264] In some embodiments, the decoder includes a residual network structure. The residual network structure includes a second self-attention unit and residual connections. The residual connection refers to the connection where the input of the second self-attention unit jumps to the second self-attention unit and is connected to its output. The first reference query feature is a feature obtained by superimposing an initial identifier feature with a first reference query feature obtained based on the second self-attention unit. In some embodiments, the decoder further includes a layer normalization unit, which is connected to the residual network structure including the second self-attention unit.

[0265] 704. The computer device calls the decoding unit of the decoder to perform cross-processing on the first reference query feature and the key and value features of multiple encoded features to obtain the decoding feature. The initial identification feature is used to indicate the start of decoding.

[0266] This step is the same as step 206.

[0267] In some embodiments, the decoder further includes a residual network structure consisting of a decoding unit and residual connections. In some embodiments, the decoder further includes a layer normalization unit connected to the residual network structure.

[0268] In some embodiments, the decoder further includes a second feature update unit connected to the decoding unit. Each time a decoded feature is obtained, the computer device invokes the second feature update unit to update the decoded feature, thus obtaining the updated decoded feature. In some embodiments, the decoder further includes a residual network structure composed of the second feature update unit and residual connections. In some embodiments, the decoder further includes a layer normalization unit connected to the residual network structure.

[0269] 705. The computer device calls the second self-attention unit to determine the second reference query feature based on the initial identification feature and the obtained decoding feature; calls the decoding unit to perform cross-processing on the second reference query feature and the key and value features of multiple encoded features to obtain the next decoding feature; until the obtained decoding feature indicates the end of decoding.

[0270] This step is the same as step 207.

[0271] The above embodiments are illustrated using a feature processing network that includes a single decoder. In some embodiments, the feature processing network includes multiple decoders. Each time the computer device calls the first decoder to obtain the decoded features, it calls the second decoder connected to the first decoder. Based on the obtained decoded features, a third reference query feature is determined. The third reference query feature and the key and value features of multiple encoded features are cross-processed to obtain the updated decoded features.

[0272] 706. The computer device calls the classifier and determines the state recognition result of the target object based on the multiple decoded features obtained.

[0273] In some embodiments, the classifier is a multi-layer fully connected network used to process multiple encoded features to obtain the state recognition result of the target object.

[0274] The above embodiments are illustrated using a feature processing network including an encoder and a decoder as an example. In some embodiments, the encoder can be replaced with a multi-layer fully connected network. In some embodiments, the encoder and decoder can be replaced entirely with a convolutional neural network.

[0275] The technical solution provided in this application utilizes an encoder and decoder that cross the attention mechanism to encode and decode sub-image features, thereby obtaining a more accurate feature representation. Based on the more accurate feature representation, a more accurate state recognition result can be obtained, thus improving the accuracy of state recognition.

[0276] It should be noted that the image processing model invoked by the computer device is a pre-trained image processing model. The feature extraction network within this model is trained on the ImageNet dataset or other datasets. The feature processing network is trained based on the pre-trained feature extraction network. The training process of the feature processing network includes: acquiring multiple sub-image features of the sample medical image and corresponding annotation information (the annotation information represents the state of the target object); invoking the feature processing network to process the multiple sub-image features of the sample medical image to obtain the sample recognition result; and training the feature processing network based on the sample recognition result and the annotation information. The multiple sub-image features are obtained by invoking the pre-trained feature extraction network.

[0277] To make the above image processing process clearer, the following will combine... Figure 8 Please provide an explanation. See also... Figure 8 Image processing mainly includes the following three processes: 801, dimensionality reduction while preserving location information; 802, feature encoding and decoding; 803, classification.

[0278] The dimensionality reduction process while preserving location information includes: dividing the medical image at 20x magnification into R rows and C columns of sub-images, each sub-image being 512×512 pixels; using the trained EfficientNet-B0 as the feature extraction network, extracting features from the 512×512×3 sub-images to obtain 7×7×1280 features, which are then transformed into one-dimensional features of length 1280. These 1280-dimensional features are the sub-image features; arranging the sub-image features according to their positions within the medical image, a feature map of the medical image is obtained, with a size of R×C×D, where D is 1280. Here, R and C are positive integers, with R positively correlated with the height of the medical image and C positively correlated with the width of the medical image.

[0279] The feature encoding-decoding process includes: A feature map of size R×C×D is processed by a 1×1 convolution to obtain a compressed feature map of size R×C×d, where d is less than D, for example, d is 512; each sub-image feature in the compressed feature map is input into the encoder to obtain the encoded features corresponding to each sub-image feature; the classification marker and the encoded features corresponding to each sub-image feature are input into the decoder to obtain the decoded features. Here, the encoder is the encoder in the Transformer architecture; optionally, the encoder is a deformable encoder used to encode a small number of key related sub-image features; the classification marker, also known as the initial identifier feature, is used to indicate the start of decoding; and the decoder is the decoder in the Transformer architecture.

[0280] The classification process includes inputting the decoded features from the decoder's output into a fully connected network to obtain the object's state identification result. In some embodiments, the state identification result is 0 or 1. This process can be applied to anomaly detection or prediction tasks. In anomaly detection tasks, 0 indicates the target object is not in an abnormal state, and 1 indicates the target object is in an abnormal state. In prediction tasks, 0 indicates that a target object in an abnormal state will not cause other objects to become abnormal, and 1 indicates that a target object in an abnormal state will cause other objects to become abnormal.

[0281] The feature processing network provided in this application combines location information and the correlation between sub-image features for encoding, which can effectively learn the interaction between sub-images at the feature level. When applied to the processing of ultra-large medical images, it will not lose the interaction between features that are far apart. It can more efficiently and organically integrate information in medical images to obtain more accurate prediction results, assist in providing more accurate medical solutions, and improve patients' quality of life.

[0282] Especially in the scenario of predicting the probability of abnormal metastasis, the tissue microenvironment characterized by medical images (including fibroblasts, immune cells, and blood vessels in medical images) has important implications for invasion and prognosis. The prediction of the probability of abnormal metastasis depends more on the changes in the overall microenvironment than on local morphological features. Therefore, the technical solution provided in this application integrates the interaction between features to characterize the tissue microenvironment as a whole, so as to predict the probability of abnormal metastasis based on the features that effectively characterize the tissue microenvironment, thereby improving the accuracy of the prediction.

[0283] In one example, an image processing model for predicting the probability of abnormal metastases was trained using the BREAST-LNM (BREAST-Lymph Node Metastases) dataset. The medical images in the BREAST-LNM dataset were obtained by 20x magnification scanning of 3957 preoperative pathological slides from breast patients stained with h and e. Lymph node metastasis was jointly annotated by two pathologists. The BREAST-LNM dataset contains 1840 positive samples and 2117 negative samples. Before training the image processing model, the dataset was divided into training, validation, and test sets at a ratio of 60%, 20%, and 20%, respectively. The image processing model was trained using the training set, the final model parameters were selected using the validation set, and the performance of the image processing model was validated using the test set.

[0284] After the image processing models were trained, the performance of several image processing models on the test set was compared. The comparison results are shown in Table 1.

[0285] Table 1

[0286]

[0287] The performance comparison includes four aspects: AUC (Area Under Curve, ROC (Receiver Operating Characteristic) curve), F1-score, Precision, and Recall. The units for AUC, F1-score, Precision, and Recall are all percentages (%). Model 1 represents an attention-weighted multi-instance model. Model 1 predicts anomalous transition probabilities based on multiple sub-images, and then performs attention-weighted summation on the anomalous transition probabilities corresponding to multiple sub-images to obtain the final anomalous transition probability. Model 2 represents an image processing model with an RNN (Recurrent Neural Network) network structure. Model 3 represents an image processing model with a convolutional neural network for feature processing. Model 4 represents an image processing model whose feature processing network includes an encoder that encodes features from multiple sub-images into a single encoded feature. Model 5 represents an image processing model whose feature processing network includes the encoder from the above embodiments. Model 6 represents an image processing model whose feature processing network includes both the encoder and decoder from the above embodiments.

[0288] As shown in Table 1 above, compared to the scheme of simply linearly adding the anomalous transition probabilities corresponding to each sub-image, the technical solution provided in this application embodiment considers the common influence of sub-images on the prediction result in terms of feature representation, effectively integrates the features of sub-images, and predicts the anomalous transition probability based on the integrated features, thus improving the accuracy of prediction. Furthermore, compared to the RNN model which only considers a sequential, unidirectional order, the technical solution provided in this application embodiment performs feature processing based on the correlation between any two sub-images, and is not limited to a sequential, unidirectional processing order. This allows it to capture the correlation between sub-images with a long distance between them and incorporates the influence of spatial location, further improving accuracy.

[0289] In one example, an image processing model for identifying tumor and normal tissues was trained using the CPTAC-LUAD (Clinical Proteomic Tumor Analysis Consortium Lung Adenocarcinoma) dataset. The CPTAC-LUAD dataset contains 1065 medical images, each labeled with its identification result. Before training the image processing model, the dataset was divided into training, validation, and test sets at 60%, 20%, and 20% respectively. The training set was used to train the image processing model, the validation set was used to select the final model parameters, and the test set was used to validate the model's performance.

[0290] After the image processing models were trained, the performance of several image processing models on the test set was compared. The comparison results are shown in Table 2 below.

[0291] Table 2

[0292]

[0293] In Table 2, models 1-6 are the same as those in Table 1. Since the task of identifying tumor tissue or normal tissue is relatively simple, all models have achieved good results in this task, and the technical solution provided in this application embodiment demonstrates even more outstanding performance in this task.

[0294] The image processing method provided in this application can be applied to various medical scenarios, such as predicting the probability of abnormal metastasis. The following is a combination of... Figure 9 The application of this image processing method in this prediction scenario is explained:

[0295] 901. A computer device acquires a medical image of lymph node tissue, divides the medical image into multiple sub-images, extracts features from the multiple sub-images, and obtains the features of the multiple sub-images.

[0296] 902. Computer equipment determines multiple coded features corresponding to multiple sub-image features.

[0297] 903. The computer device determines the state recognition result based on multiple coding features. This state recognition result is used to represent the probability of diseased cells metastasizing through lymph nodes.

[0298] Figure 10 This is a block diagram of an image processing apparatus provided in an embodiment of this application. See also... Figure 10 The device includes:

[0299] The sub-image feature extraction module 1001 is used to divide a medical image containing a target object into multiple sub-images, extract features from each of the multiple sub-images, and obtain multiple sub-image features.

[0300] The coding feature determination module 1002 is used to determine the coding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features. The first sub-image feature is any one of the multiple sub-image features, and the multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature.

[0301] The state recognition module 1003 is used to determine the state recognition result of the target object based on multiple encoded features corresponding to multiple sub-image features.

[0302] The image processing apparatus provided in this application divides a medical image containing a target object into multiple sub-images. Considering the combined influence of multiple sub-images on the state recognition result of the target object, for each sub-image feature, multiple sub-image features are encoded into the sub-image feature, thereby achieving effective integration of sub-image features and obtaining a more accurate feature representation. Based on the more accurate feature representation, a more accurate state recognition result can be obtained, thus improving the accuracy of state recognition.

[0303] In one possible implementation, the encoding feature determination module 1002 is used for:

[0304] The query features of the first sub-image feature and the key and value features of each second sub-image feature are cross-processed to obtain the encoded features corresponding to the first sub-image feature.

[0305] In one possible implementation, the encoding feature determination module 1002 is used for:

[0306] Based on the query features of the first sub-image features and the key features of each second sub-image feature, determine the correlation parameters between the first sub-image features and each second sub-image feature;

[0307] Based on the multiple correlation parameters corresponding to the multiple second sub-image features, the value features of the multiple second sub-image features are weighted and fused to obtain the encoded features corresponding to the first sub-image features.

[0308] In one possible implementation, the query features, key features, and value features of the sub-image features are obtained based on the first parameter matrix, and the apparatus further includes:

[0309] The first coding feature update module is used to concatenate and compress multiple coding features corresponding to the first sub-image features obtained from the multiple first parameter matrices respectively, in the presence of multiple first parameter matrices, to obtain the updated coding features corresponding to the first sub-image features.

[0310] In one possible implementation, the device further includes:

[0311] The associated location determination module is used to determine a second location associated with the first location based on the first location of the first sub-image in the medical image, wherein the features of the first sub-image are obtained by feature extraction of the first sub-image;

[0312] The associated sub-image feature determination module is used to determine the sub-image features corresponding to the second sub-image located at the second position as associated sub-image features.

[0313] In one possible implementation, the device further includes:

[0314] The second coding feature update module is used to determine the updated coding feature corresponding to the first sub-image feature based on the coding feature corresponding to the first sub-image feature and the multiple coding features corresponding to the multiple second sub-image features respectively.

[0315] In one possible implementation, the state recognition module 1003 includes:

[0316] The first decoding submodule is used to determine the first reference query feature based on the initial identifier feature, and to perform cross-processing on the key features and value features of the first reference query feature and multiple encoded features to obtain the decoding feature. The initial identifier feature is used to indicate the start of decoding.

[0317] The second decoding submodule is used to determine the second reference query feature based on the initial identifier feature and the obtained decoding feature, and to perform cross-processing on the key features and value features of the second reference query feature and multiple encoded features to obtain the next decoding feature, until the obtained decoding feature indicates the end of decoding;

[0318] The state recognition submodule is used to determine the state recognition result of the target object based on multiple decoded features obtained.

[0319] In one possible implementation, the key and value features of the encoded features are obtained based on a second parameter matrix, and the device further includes:

[0320] The first decoding feature update module is used to concatenate and compress multiple decoding features obtained based on multiple second parameter matrices to obtain updated decoding features when multiple second parameter matrices exist.

[0321] In one possible implementation, the second decoding submodule is used for:

[0322] Based on the latest obtained decoding features and the key features of the initial identifier features, determine the relevance parameters corresponding to the initial identifier features; and based on the latest obtained decoding features and the key features of each obtained decoding feature, determine the relevance parameters corresponding to each obtained decoding feature.

[0323] Based on the relevance parameters corresponding to the initial identifier features and the relevance parameters corresponding to each obtained decoded feature, the value features of the initial identifier features and the value features of each obtained decoded feature are weighted and fused to obtain the second reference query feature.

[0324] In one possible implementation, the query features, key features, and value features of the decoded features, as well as the query features, key features, and value features of the initial identifier features, are obtained based on a third parameter matrix. The apparatus further includes:

[0325] The query feature update module is used to concatenate and compress multiple second reference query features obtained from the multiple third parameter matrices to obtain updated second reference query features when multiple third parameter matrices exist.

[0326] In one possible implementation, the device further includes:

[0327] The second decoding feature update module is used to determine the third reference query feature based on the obtained decoding features after each decoding feature is obtained; and to perform cross-processing on the third reference query feature and the key features and value features of multiple encoded features to obtain the updated decoding features.

[0328] In one possible implementation, the number of channels for each sub-image feature is a first quantity, and the device further includes:

[0329] The feature compression module is used to perform convolution processing on each sub-image feature to obtain the convolution-processed sub-image feature. The number of channels of the convolution-processed sub-image feature is a second number, which is less than the first number.

[0330] The coding feature determination module 1002 is used to determine the coding features corresponding to the first sub-image features based on the first sub-image features after convolution processing and multiple second sub-image features after convolution processing.

[0331] In one possible implementation, the device further includes:

[0332] The location encoding module is used to encode the location information of each of the multiple sub-images to obtain location features, and then embed the location features into the sub-image features of the sub-image. The location information is used to represent the location of the sub-image in the medical image.

[0333] In one possible implementation, the sub-image feature extraction module 1001 is used to compress the size of the initial feature map of each sub-image until a feature map of the target size is obtained; and to compress the feature map of the target size into a sub-image feature of size 1×1.

[0334] In one possible implementation, multiple sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image, forming a feature map of the medical image, wherein the number of channels in the feature map is a first quantity; the device further includes:

[0335] The feature map compression module is used to perform convolution processing on the feature maps of medical images to obtain convolution-processed feature maps. The number of channels in the convolution-processed feature maps is a second number, which is less than the first number.

[0336] The coding feature determination module 1002 is used to determine the coding feature corresponding to the first sub-image feature based on the first sub-image feature in the convolutional feature map and multiple second sub-image features in the convolutional feature map.

[0337] In one possible implementation, the image processing model includes a feature extraction network and a feature processing network, wherein the feature processing network includes an encoder and a classifier;

[0338] The sub-image feature extraction module 1001 is used to call the feature extraction network to extract features from multiple sub-images respectively, and obtain multiple sub-image features;

[0339] The coding feature determination module 1002 is used to call the encoder to determine the coding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features;

[0340] The state recognition module 1003 is used to call the classifier and determine the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features.

[0341] In one possible implementation, the encoder includes a first self-attention unit; and an encoding feature determination module 1002, which calls the first self-attention unit to perform cross-processing on the query features of the first sub-image features and the key features and value features of each second sub-image feature to obtain the encoding features corresponding to the first sub-image features.

[0342] In one possible implementation, the encoder also includes a first feature update unit;

[0343] The device also includes:

[0344] The third coding feature update module is used to call the first feature update unit to update the coding features corresponding to the first sub-image features, and obtain the updated coding features corresponding to the first sub-image features.

[0345] In one possible implementation, the feature processing network includes multiple encoders;

[0346] The encoding feature determination module 1002 is used to call the first encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features; and to call the second encoder connected after the first encoder to determine the updated encoding feature corresponding to the first sub-image feature based on the encoding feature corresponding to the first sub-image feature and multiple encoding features corresponding to the multiple second sub-image features respectively.

[0347] In one possible implementation, the feature processing network further includes a decoder, which comprises a second self-attention unit and a decoding unit;

[0348] The status recognition module 1003 includes:

[0349] The first decoding submodule is used to call the second self-attention unit to determine the first reference query feature based on the initial identifier feature; and to call the decoding unit to perform cross-processing on the first reference query feature and the key and value features of multiple encoded features to obtain the decoding feature. The initial identifier feature is used to indicate the start of decoding.

[0350] The second decoding submodule is used to call the second self-attention unit to determine the second reference query feature based on the initial identifier feature and the obtained decoding feature; call the decoding unit to perform cross-processing on the second reference query feature and the key and value features of multiple encoded features to obtain the next decoding feature; until the obtained decoding feature indicates the end of decoding;

[0351] The state recognition submodule is used to call the classifier and determine the state recognition result of the target object based on multiple decoded features.

[0352] In one possible implementation, the decoder further includes a second feature update unit, and the device also includes:

[0353] The third decoding feature update module is used to call the second feature update unit to update the decoding features after each decoding feature is obtained, and obtain the updated decoding features.

[0354] In one possible implementation, the device further includes:

[0355] The second decoding feature update module is used to call the second decoder connected after the first decoder after each call to obtain the decoding features. Based on the obtained decoding features, the module determines the third reference query feature and performs cross-processing on the key and value features of the third reference query feature and multiple encoded features to obtain the updated decoding features.

[0356] In one possible implementation, the number of channels for each sub-image feature is a first quantity, and the feature processing network also includes a feature compressor;

[0357] The device also includes:

[0358] The feature compression module is used to call the feature compressor to perform convolution processing on each sub-image feature to obtain the convolution-processed sub-image feature. The number of channels of the convolution-processed sub-image feature is the second number, which is less than the first number.

[0359] The coding feature determination module 1002 is used to call the encoder to determine the coding features corresponding to the first sub-image features based on the first sub-image features after convolution and multiple second sub-image features after convolution.

[0360] In one possible implementation, multiple sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image to form a feature map of the medical image, and the number of channels in the feature map is a first quantity; the feature processing network also includes a feature compressor.

[0361] The device also includes:

[0362] The feature map compression module is used to call the feature compressor to perform convolution processing on the feature map of the medical image to obtain the convolution-processed feature map. The number of channels in the convolution-processed feature map is the second number, which is less than the first number.

[0363] The encoding feature determination module 1002 is used to call the encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature in the convolutional feature map and multiple second sub-image features in the convolutional feature map.

[0364] In one possible implementation, the sub-image feature extraction module 1001 is used to call the feature extraction network to compress the size of the initial feature map of each sub-image until a feature map of the target size is obtained; and to compress the feature map of the target size into a sub-image feature of size 1×1.

[0365] In one possible implementation, the training process of the feature processing network includes:

[0366] The features of multiple sub-images of the sample medical image and the corresponding annotation information of the sample medical image are obtained. The annotation information is used to represent the state of the target object.

[0367] The feature processing network is invoked to process multiple sub-image features of the sample medical image to obtain the sample recognition result;

[0368] Based on the sample recognition results and annotation information, a feature processing network is trained.

[0369] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when processing images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the image processing apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0370] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0371] Figure 11This is a block diagram of a computer device 1100 provided in an embodiment of this application. The computer device 1100 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The memories 1102 store computer programs, which are loaded and executed by the processors 1101 to implement the image processing methods provided in the various method embodiments described above. Of course, the computer device may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device may also include other components for implementing device functions, which will not be elaborated upon here.

[0372] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program that can be executed by a processor of a computer device to perform the image processing method described above. For example, the computer-readable storage medium may be a ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, or optical data storage device, etc.

[0373] This application also provides a computer program product comprising a computer program that, when executed by a processor, implements the image processing methods described in the various method embodiments above.

[0374] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0375] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The medical image containing the target object is divided into multiple sub-images, and feature extraction is performed on each of the multiple sub-images to obtain multiple sub-image features; Based on a first sub-image feature and multiple second sub-image features, a coding feature corresponding to the first sub-image feature is determined. The first sub-image feature is any one of the multiple sub-image features. The multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature. Based on a similar process to determining the coding features corresponding to the first sub-image features, the coding features corresponding to each of the plurality of sub-image features are determined; Based on the multiple encoded features corresponding to the multiple sub-image features, the state recognition result of the target object is determined; The step of determining the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes: The query features of the first sub-image feature and the key and value features of each second sub-image feature are cross-processed to obtain the encoded features corresponding to the first sub-image feature; The determination of the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features includes: Based on the aforementioned multiple encoding features, a cross-attention mechanism is employed to obtain multiple decoding features; Based on the obtained multiple decoding features, the state recognition result of the target object is determined.

2. The method according to claim 1, characterized in that, The process of cross-processing the query features of the first sub-image features and the key and value features of each second sub-image feature to obtain the encoded features corresponding to the first sub-image features includes: Based on the query features of the first sub-image features and the key features of each second sub-image feature, the correlation parameters between the first sub-image features and each second sub-image feature are determined. According to the multiple correlation parameters corresponding to the multiple second sub-image features, the value features of the multiple second sub-image features are weighted and fused to obtain the encoded features corresponding to the first sub-image features.

3. The method according to claim 1, characterized in that, The query features, key features, and value features of the sub-image features are obtained based on the first parameter matrix, and the method further includes: In the presence of multiple first parameter matrices, the multiple encoded features corresponding to the first sub-image features obtained based on the multiple first parameter matrices are concatenated and compressed to obtain the updated encoded features corresponding to the first sub-image features.

4. The method according to claim 1, characterized in that, The plurality of second sub-image features include the first sub-image features and associated sub-image features of the first sub-image features. Before determining the encoded features corresponding to the first sub-image features based on the first sub-image features and the plurality of second sub-image features, the method further includes: Based on a first position of a first sub-image in the medical image, a second position associated with the first position is determined, wherein the features of the first sub-image are obtained by feature extraction from the first sub-image; The sub-image features corresponding to the second sub-image located at the second position are determined as the associated sub-image features.

5. The method according to claim 1, characterized in that, Before determining the encoded feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features, the method further includes: For each of the plurality of sub-images, the position information of the sub-image is encoded to obtain position features, and the position features are embedded into the sub-image features of the sub-image. The position information is used to represent the position of the sub-image in the medical image.

6. The method according to claim 1, characterized in that, Based on the multiple encoded features, a cross-attention mechanism is used to obtain multiple decoded features, including: Based on the initial identifier feature, a first reference query feature is determined. The key and value features of the first reference query feature and the plurality of encoded features are cross-processed to obtain the decoding feature. The initial identifier feature is used to indicate the start of decoding. Based on the initial identifier feature and the obtained decoding feature, a second reference query feature is determined. The second reference query feature and the key and value features of the multiple encoded features are cross-processed to obtain the next decoding feature, until the obtained decoding feature indicates the end of decoding.

7. The method according to claim 6, characterized in that, The step of determining the second reference query feature based on the initial identifier feature and the obtained decoded feature includes: Based on the latest obtained decoding feature query feature and the key feature of the initial identifier feature, determine the relevance parameter corresponding to the initial identifier feature; and based on the latest obtained decoding feature query feature and the key feature of each obtained decoding feature, determine the relevance parameter corresponding to each obtained decoding feature. The value features of the initial identifier feature and the value features of each obtained decoded feature are weighted and fused according to the relevance parameters corresponding to the initial identifier feature and the relevance parameters corresponding to each obtained decoded feature to obtain the second reference query feature.

8. The method according to claim 1, characterized in that, The step of extracting features from the multiple sub-images to obtain multiple sub-image features includes: Compress the initial feature map size of each of the sub-images until a feature map of the target size is obtained; The feature map of the target size is compressed into a sub-image feature of size 1×1.

9. The method according to claim 1, characterized in that, The number of channels for each sub-image feature is a first quantity. Before determining the encoded feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features, the method further includes: Each of the sub-image features is subjected to convolution processing to obtain convolution-processed sub-image features, wherein the number of channels of the convolution-processed sub-image features is a second number, which is less than the first number; The step of determining the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes: Based on the first sub-image features after convolution processing and the second sub-image features after multiple convolution processing, the coding features corresponding to the first sub-image features are determined.

10. The method according to claim 1, characterized in that, The multiple sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image to form a feature map of the medical image, wherein the number of channels of the feature map is a first quantity; Before determining the encoded feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features, the method further includes: The feature map of the medical image is subjected to convolution processing to obtain a convolution-processed feature map. The number of channels in the convolution-processed feature map is a second number, which is less than the first number. The step of determining the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes: Based on the first sub-image feature in the convolutional feature map and multiple second sub-image features in the convolutional feature map, the encoding feature corresponding to the first sub-image feature is determined.

11. The method according to claim 1, characterized in that, The image processing model includes a feature extraction network and a feature processing network, wherein the feature processing network includes an encoder and a classifier; The step of extracting features from the multiple sub-images to obtain multiple sub-image features includes: calling the feature extraction network to extract features from the multiple sub-images to obtain multiple sub-image features; The step of determining the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features includes: calling the encoder to determine the encoding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features; The step of determining the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features includes: calling the classifier, using a cross-attention mechanism to obtain multiple decoded features based on the multiple encoded features corresponding to the multiple sub-image features; and determining the state recognition result of the target object based on the obtained multiple decoded features.

12. The method according to claim 11, characterized in that, The feature processing network further includes a decoder, which includes a second self-attention unit and a decoding unit; Based on the multiple encoded features corresponding to the multiple sub-image features, a cross-attention mechanism is used to obtain multiple decoded features; Based on the obtained multiple decoding features, the state recognition result of the target object is determined, including: The second self-attention unit is invoked to determine the first reference query feature based on the initial identifier features; The decoding unit is invoked to perform cross-processing on the key and value features of the first reference query feature and the plurality of encoded features to obtain the decoding feature, wherein the initial identifier feature is used to indicate the start of decoding; The second self-attention unit is invoked to determine the second reference query feature based on the initial identifier feature and the obtained decoded feature; the decoding unit is invoked to perform cross-processing on the second reference query feature and the key and value features of the multiple encoded features to obtain the next decoded feature; the decoding is completed until the obtained decoded feature indicates that the decoding is finished. The classifier is invoked to determine the state recognition result of the target object based on the obtained multiple decoded features.

13. The method according to claim 11, characterized in that, The number of channels for each sub-image feature is a first quantity, and the feature processing network further includes a feature compressor; Before invoking the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features, the method further includes: The feature compressor is invoked to perform convolution processing on each of the sub-image features to obtain convolution-processed sub-image features. The number of channels of the convolution-processed sub-image features is a second number, which is less than the first number. The step of invoking the encoder to determine the encoding features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes: The encoder is invoked to determine the encoded features corresponding to the first sub-image features based on the first sub-image features after convolution and multiple second sub-image features after convolution.

14. The method according to claim 11, characterized in that, The multiple sub-image features are arranged according to the position of the sub-image corresponding to each sub-image feature in the medical image to form a feature map of the medical image, and the number of channels of the feature map is a first quantity; the feature processing network also includes a feature compressor; Before invoking the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features, the method further includes: The feature compressor is invoked to perform convolution processing on the feature map of the medical image to obtain a convolution-processed feature map. The number of channels in the convolution-processed feature map is a second number, which is less than the first number. The step of invoking the encoder to determine the encoded features corresponding to the first sub-image features based on the first sub-image features and multiple second sub-image features includes: The encoder is invoked to determine the encoded features corresponding to the first sub-image features based on the first sub-image features in the convolutional feature map and multiple second sub-image features in the convolutional feature map.

15. The method according to any one of claims 11-14, characterized in that, The training process of the feature processing network includes: Multiple sub-image features of a sample medical image and corresponding annotation information of the sample medical image are obtained, wherein the annotation information is used to represent the state of the target object; The feature processing network is invoked to process multiple sub-image features of the sample medical image to obtain the sample recognition result; The feature processing network is trained based on the sample recognition results and the annotation information.

16. An image processing apparatus, characterized in that, The device includes: The sub-image feature extraction module is used to divide a medical image containing a target object into multiple sub-images, and extract features from each of the multiple sub-images to obtain multiple sub-image features; The coding feature determination module is used to determine the coding feature corresponding to the first sub-image feature based on the first sub-image feature and multiple second sub-image features. The first sub-image feature is any one of the multiple sub-image features, and the multiple second sub-image features include the first sub-image feature and at least one other sub-image feature besides the first sub-image feature. The coding feature determination module is further configured to determine the coding feature corresponding to each of the plurality of sub-image features based on a process similar to that used to determine the coding feature corresponding to the first sub-image feature; The state recognition module is used to determine the state recognition result of the target object based on the multiple encoded features corresponding to the multiple sub-image features respectively; The encoding feature determination module is used to perform cross-processing on the query features of the first sub-image features and the key features and value features of each second sub-image feature to obtain the encoding features corresponding to the first sub-image features. The state recognition module is used to obtain multiple decoded features based on the multiple encoded features and by employing a cross-attention mechanism; and to determine the state recognition result of the target object based on the obtained multiple decoded features.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, which is loaded and executed by the processor to implement the image processing method as described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1-15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1-15.

Images