Image processing method, apparatus, device, storage medium and program product

Abstract

The application provides an image processing method, device and equipment, a storage medium and a program product; relates to artificial intelligence technology; the method comprises the following steps: dividing a target image to obtain a plurality of subgraphs included in the target image; performing feature extraction on each subgraph to obtain subgraph features of each subgraph; performing first classification prediction on each subgraph based on the subgraph features of each subgraph to obtain subgraph classification results of each subgraph; for each subgraph, determining the weight of the subgraph based on the subgraph classification result of the subgraph, and determining the weighted subgraph feature of the subgraph based on the subgraph feature and the weight of the subgraph, wherein the weight is used to indicate the influence degree of the subgraph on the image classification result of the target image; performing second classification prediction on the target image based on the weighted subgraph features of the plurality of subgraphs to obtain the image classification result of the target image; through the application, the accuracy of image classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive field of computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0003] Image classification is also a key application area of ​​artificial intelligence. Related technologies often directly predict the image's classification based on its overall features to determine its category. However, due to the complexity and multi-category nature of some images, the accuracy of such predictions based solely on the overall image features is limited. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of image classification.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides an image processing method, including:

[0007] Dividing the target image to obtain a plurality of sub-images included in the target image;

[0008] Performing feature extraction on each of the subgraphs to obtain subgraph features of each of the subgraphs;

[0009] Based on the sub-graph features of each of the sub-graphs, a first classification prediction is performed on each of the sub-graphs to obtain a sub-graph classification result for each of the sub-graphs;

[0010] For each of the sub-graphs, determining a weight of the sub-graph based on a sub-graph classification result of the sub-graph, and determining a weighted sub-graph feature of the sub-graph based on the sub-graph feature and the weight, wherein the weight is used to indicate a degree of influence of the sub-graph on an image classification result of the target image;

[0011] Based on the weighted sub-graph features of the multiple sub-graphs, a second classification prediction is performed on the target image to obtain an image classification result of the target image.

[0012] The present application also provides an image processing device, including:

[0013] A division module, configured to divide a target image into multiple sub-images included in the target image;

[0014] A feature extraction module, configured to extract features from each of the subgraphs to obtain subgraph features of each of the subgraphs;

[0015] A first prediction module is configured to perform a first classification prediction on each of the subgraphs based on the subgraph features of the subgraphs to obtain a subgraph classification result for each of the subgraphs;

[0016] a determination module, configured to determine, for each of the sub-graphs, a weight of the sub-graph based on a sub-graph classification result of the sub-graph, and determine a weighted sub-graph feature of the sub-graph based on the sub-graph feature and the weight, wherein the weight is used to indicate a degree of influence of the sub-graph on the image classification result of the target image;

[0017] The second prediction module is used to perform a second classification prediction on the target image based on the weighted sub-image features of the multiple sub-images to obtain an image classification result of the target image.

[0018] In the above scheme, the division module is also used to determine a sliding window with a target size corresponding to the target image and a sliding step of the sliding window; by sliding the sliding window according to the sliding step, the target image is divided to obtain multiple sub-images included in the target image.

[0019] In the above scheme, the feature extraction module is also used to perform feature extraction of the first dimension on each of the sub-images to obtain the intermediate sub-image features of the first dimension of each of the sub-images; splice the intermediate sub-image features of the first dimension of each of the multiple sub-images to obtain the intermediate sub-image feature matrix of the target image; perform feature extraction of the second dimension on the intermediate sub-image feature matrix to obtain the sub-image feature matrix of the target image, the sub-image feature matrix including the sub-image features of the second dimension of each of the sub-images, where the second dimension is lower than the first dimension.

[0020] In the above scheme, the determination module is also used to obtain the number of classification categories predicted by the first classification and determine the uniform distribution of the number of classification categories; determine the difference between the sub-graph classification result of the sub-graph and the uniform distribution; and use the difference as the weight of the sub-graph.

[0021] In the above solution, the determining module is further configured to multiply the subgraph feature of the subgraph by the weight of the subgraph to obtain the weighted subgraph feature of the subgraph.

[0022] In the above scheme, the second prediction module is also used to perform feature aggregation processing on the weighted sub-graph features of the multiple sub-graphs to obtain aggregated sub-graph features; based on the aggregated sub-graph features, perform a second classification prediction on the target image to obtain an image classification result of the target image.

[0023] In the above scheme, the first classification prediction is implemented by the instance-level classification layer of the image classification model, and the second classification prediction is implemented by the bag-level classification layer of the image classification model; the device also includes a model training module; the model training module is used to obtain an image sample set for training the image classification model, the image sample set includes multiple image samples, the image samples are annotated with image labels, and the image samples include multiple sub-image samples, and the sub-image samples are annotated with sub-image labels; through the instance-level classification layer, a first classification prediction is performed on each sub-image sample in each of the image samples to obtain an instance classification result of each sub-image sample; for each of the image samples, the following processing is performed: combined with the sub-image label of each sub-image sample in the image sample, a second classification prediction is performed on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample; based on the difference between the instance classification result and the sub-image label of each sub-image sample, and the difference between the sample classification result and the image label of the image sample, a model loss value of the image classification model is determined; based on the model loss value, the model parameters of the image classification model are updated to train the image classification model.

[0024] In the above scheme, the model training module is also used to determine the intermediate instance-level loss of the image classification model for each sub-image sample based on the difference between the instance classification result of the sub-image sample and the sub-image label; add the intermediate instance-level losses of the image classification model to obtain the instance-level loss of the image classification model; determine the bag-level loss of the image classification model based on the difference between the sample classification result of the image sample and the image label; based on the first loss weight of the instance-level loss and the second loss weight of the bag-level loss, perform weighted sum processing on the instance-level loss and the bag-level loss to obtain the model loss value of the image classification model.

[0025] In the above scheme, the model training module is also used to obtain the sub-image sample features of each sub-image sample in each image sample; through the instance-level classification layer, based on the sub-image sample features of each sub-image sample, a first classification prediction is performed on each sub-image sample to obtain an instance classification result of each sub-image sample; the model training module is also used to determine the sub-image weight of each sub-image sample based on the sub-image label of each sub-image sample in the image sample, and determine the weighted sub-image sample features of each sub-image sample based on the sub-image sample features and sub-image weight of each sub-image sample; based on the weighted sub-image sample features of multiple sub-image samples in the image sample, a second classification prediction is performed on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample.

[0026] In the above scheme, the model training module is also used to perform the following processing for each sub-image sample in the image sample: obtaining the number of classification categories predicted by the first classification and determining the uniform distribution of the number of classification categories; determining the difference between the sub-image label of the sub-image sample and the uniform distribution; and using the difference as the sub-image weight of the sub-image sample.

[0027] In the above scheme, the model training module is also used to determine the key subgraphs of each category obtained by the first classification prediction from multiple subgraph samples in the image sample set based on the instance classification results of each subgraph sample in the image sample set before performing the following processing on each of the image samples respectively; for each of the categories, obtain the prototype features and corresponding prototype labels of the category, and use the key subgraphs of the category to momentum update the prototype features to obtain new prototype features of the category; for each of the subgraph samples, determine the similarity between the subgraph features of the subgraph sample and the new prototype features of each category, and use the prototype label corresponding to the target new prototype feature with the highest similarity as the subgraph prototype label of the subgraph sample; for each of the subgraph samples, use the subgraph prototype label of the subgraph sample to momentum update the subgraph label of the subgraph sample to obtain the new subgraph label of the subgraph sample.

[0028] In the above scheme, the model training module is also used to combine the new sub-image label of each sub-image sample in the image sample, perform a second classification prediction on the image sample through the bag-level classification layer, and obtain the sample classification result of the image sample; the model training module is also used to determine the model loss value of the image classification model based on the difference between the instance classification result of each sub-image sample and the new sub-image label, and the difference between the sample classification result of the image sample and the image label.

[0029] In the above scheme, the instance classification result includes the instance prediction probability of the sub-image sample belonging to each of the categories; the model training module is also used to perform the following processing for each of the categories: from multiple sub-image samples in the image sample set, determine the target number of target sub-image samples belonging to the category and sorted in descending order of instance prediction probability; use the target number of target sub-image samples as the target number of key sub-images of the category.

[0030] In the above scheme, the number of the key subgraphs is N, where N is an integer greater than 1. The model training module is further used to adopt the first key subgraph of the category, momentum update the prototype feature, and obtain the first intermediate prototype feature of the category; adopt the i-th key subgraph of the category, momentum update the (i-1)-th intermediate prototype feature of the category, and obtain the i-th intermediate prototype feature of the category, where i is greater than 0 and not greater than N; traverse the i to obtain the N-th intermediate prototype feature of the category, and use the N-th intermediate prototype feature of the category as the new prototype feature of the category.

[0031] In the above scheme, there are multiple key subgraphs, and the model training module is also used to obtain the key subgraph features of each key subgraph of the category, and determine the average subgraph features of the multiple key subgraph features; obtain the first feature weight of the average subgraph feature, and the second feature weight of the prototype feature; based on the first feature weight and the second feature weight, perform weighted summation processing on the average subgraph feature and the prototype feature to obtain a new prototype feature of the category.

[0032] In the above scheme, the model training module is also used to obtain a first label weight of the sub-graph prototype label and a second label weight of the sub-graph label; based on the first label weight and the second label weight, the sub-graph prototype label and the sub-graph label are weighted summed to obtain a new sub-graph label for the sub-graph sample.

[0033] An embodiment of the present application further provides an electronic device, including:

[0034] a memory for storing computer-executable instructions;

[0035] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0036] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the image processing method provided in the embodiment of the present application is implemented.

[0037] An embodiment of the present application further provides a computer program product, comprising computer-executable instructions or a computer program, which, when executed by a processor, implements the image processing method provided in the embodiment of the present application.

[0038] The embodiments of the present application have the following beneficial effects:

[0039] By applying the above embodiment of the present application, when performing classification prediction on a target image, the target image is first divided to obtain a plurality of sub-images included in the target image, and then feature extraction is performed on each sub-image to obtain the sub-image features of each sub-image, and based on the sub-image features of each sub-image, a first classification prediction is performed on each sub-image to obtain the sub-image classification results of each sub-image; thereby, based on the sub-image classification results of each sub-image, the weight of each sub-image is determined, and based on the sub-image features and weights of each sub-image, the sub-image features are weighted to obtain the weighted sub-image features of each sub-image, and finally, based on the weighted sub-image features of each sub-image, a second classification prediction is performed on the target image to obtain the image classification result of the target image. Since the weight of each sub-image feature indicates the degree of influence of the sub-image on the image classification result of the target image, the weighted sub-image features obtained by weighting can better characterize the target image, so that the image classification result obtained based on the classification prediction of each weighted sub-image feature is more accurate, thereby improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 1 is a schematic diagram of the architecture of an image processing system 100 provided in an embodiment of the present application;

[0041] Figure 2 is a structural diagram of an electronic device 500 for implementing an image processing method provided in an embodiment of the present application;

[0042] Figure 3 Schematic diagram of the image processing method provided in the embodiment of the present application;

[0043] Figure 4 Schematic diagram of the training process of the image classification model provided in the embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the training process of the image classification model provided in the embodiment of the present application;

[0045] Figure 6 This is a schematic diagram of the training process of the image classification model provided in the embodiment of the present application;

[0046] Figure 7 Schematic diagram of the momentum update process of the prototype feature provided in the embodiment of the present application;

[0047] Figure 8is a momentum updating process schematic diagram of a prototype feature provided by an embodiment of the present application;

[0048] Figure 9 is a momentum updating process schematic diagram of a subgraph label provided by an embodiment of the present application;

[0049] Figure 10 is a training process schematic diagram of an image classification model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0051] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0052] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0054] Before the embodiments of the present application are described in further detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0055] 1) Client, an application program running in a terminal for providing various services, such as a client supporting image processing.

[0056] 2) In response to, used to indicate the condition or state on which the operation is performed, when the dependent condition or state is met, the performed one or more operations can be real-time or have a set delay; in the absence of a specific description, there is no limitation on the execution order of the performed multiple operations.

[0057] 3) Computer vision (CV) is the science of making machines "see." This involves using cameras and computers to replace the human eye in identifying and measuring objects, and further processing the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0058] 4) Whole slide images (WSIs), namely pathological images WSI.

[0059] 5) Multiple Instance Learning (MIL), which belongs to the supervised learning framework. In multiple instance learning, the training set consists of "bags (or bags)" carrying bag-level labels, and each "bag" has multiple instances that do not carry labels. In the case of multi-instance binary classification, if all instances in the "bag" are negative, the label of the "bag" can be marked as negative; if there is at least one instance in the "bag" that is positive, the label of the "bag" can be marked as positive. In the embodiment of the present application, "bag" refers to an image, "instance" refers to a sub-graph Patch obtained by dividing the image, the bag-level label refers to the label of the image, and the instance-level label refers to the instance soft label of the instance.

[0060] 6) Prototype vectors, also known as prototype features, can be understood as the overall features of a category. For example, for the category "schoolbags," the prototype features could be features such as fabric, color, texture, size, or implicit features learned by the neural network. Each category can have multiple prototype features.

[0061] The embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of image classification.

[0062] It should be noted that the collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0063] The following describes the image processing system provided by the embodiment of the present application. Figure 1 , Figure 1 This is an architectural diagram of the image processing system 100 provided in an embodiment of the present application. To support an exemplary application, a terminal (terminal 400-1 is shown as an example) is connected to a server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using wireless or wired links.

[0064] The terminal (e.g., 400 - 1 , which may be provided with a client supporting image processing) is configured to send an image classification request for the target image to the server 200 in response to an image classification instruction for the target image;

[0065] The server 200 is configured to receive an image classification request for a target image sent by a terminal; in response to the image classification request, obtain the target image, and divide the target image to obtain a plurality of sub-graphs included in the target image; perform feature extraction on each sub-graph to obtain a sub-graph feature of each sub-graph; perform a first classification prediction on each sub-graph based on the sub-graph feature of each sub-graph to obtain a sub-graph classification result of each sub-graph; for each sub-graph, determine a weight of the sub-graph based on the sub-graph classification result of the sub-graph, and determine a weighted sub-graph feature of the sub-graph based on the sub-graph feature and weight of the sub-graph, wherein the weight is used to indicate the degree of influence of the sub-graph on the image classification result of the target image; perform a second classification prediction on the target image based on the weighted sub-graph features of the plurality of sub-graphs to obtain an image classification result of the target image; and return the image classification result of the target image to the terminal.

[0066] The terminal (eg, 400 - 1 ) is further configured to receive the image classification result of the target image returned by the server 200 ; and display the image classification result.

[0067] In some embodiments, the image processing method provided in the embodiments of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. The image processing method provided in the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, CV technology, AI medical care, etc.

[0068] In some embodiments, the electronic device for implementing the image processing method provided in the embodiments of the present application may be various types of terminals or servers. Among them, the server (such as server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal (such as terminal 400-1) may be a laptop, a tablet computer, a desktop computer, a smart phone, an intelligent voice interaction device (such as a smart speaker), a smart home appliance (such as a smart TV), a smart watch, a car terminal, a wearable device, a virtual reality (VR) device, etc., but is not limited to this. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application do not limit this.

[0069] In some embodiments, the image processing method provided in the embodiments of the present application can be implemented with the aid of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model application. It can form a resource pool that is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing resources and storage resources. As an example, the server (such as server 200) can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0070] In some embodiments, multiple servers may form a blockchain, with the servers being nodes on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes via these connections. Data related to the image processing methods provided in the embodiments of the present application (e.g., image classification models, training sample sets, etc.) may be stored on the blockchain.

[0071] In some embodiments, the terminal or server can implement the image processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0072] The electronic device for implementing the image processing method provided by the embodiment of the present application is described below. Figure 2 , Figure 2 : is a structural diagram of an electronic device 500 for implementing an image processing method provided in an embodiment of the present application. The electronic device 500 provided in an embodiment of the present application may be a terminal or a server. The electronic device 500 provided in an embodiment of the present application includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0073] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0075] The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may include one or more storage devices physically remote from the processor 510. The memory 550 includes volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0076] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0077] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0078] A network communication module 552 for reaching other electronic devices via one or more (wired or wireless) network interfaces 520 , exemplary network interfaces 520 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0079] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0080] The input processing module 554 is configured to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

[0081] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 An image processing device 555 stored in a memory 550 is shown, which can be software in the form of a program and plug-in, etc., including the following software modules: a division module 5551, a feature extraction module 5552, a first prediction module 5553, a determination module 5554, a second prediction module 5555 and a model training module 5556. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0082] The image processing method provided by the embodiment of the present application is described below. In some embodiments, the image processing method provided by the embodiment of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, or by a server alone, or by a terminal and a server in collaboration. Taking the terminal implementation as an example, see Figure 3 , Figure 3 : is a flow chart of an image processing method provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application includes:

[0083] Step 101: The terminal divides a target image to obtain a plurality of sub-images included in the target image.

[0084] Here, the terminal may be provided with a client supporting image processing. The terminal runs the client in response to an instruction to run the client. When the terminal receives an instruction to classify a target image, the terminal acquires the target image. For example, the client may be a client supporting pathological image classification. When a pathologist performs a pathological diagnosis based on a pathological image, the pathologist may run the client on the terminal and, through the client, execute the image processing method provided in the embodiments of the present application to classify the pathological image and achieve the purpose of pathological diagnosis.

[0085] In step 101, after acquiring the target image to be processed, the terminal divides the target image to obtain a plurality of sub-images included in the target image. It should be noted that the target image is a "bag" and the sub-image is an instance (i.e., Patch) in the "bag". Here, when dividing the target image, an image division method for the target image can be pre-set, so that the target image is divided based on the image division method to obtain a plurality of sub-images included in the target image. In actual applications, the image division method may include the size of each sub-image to be divided, the total number of sub-images to be divided, whether the image is divided by a sliding window, whether the image is divided by image segmentation, and the like.

[0086] In some embodiments, the terminal may divide the target image to obtain multiple sub-images included in the target image in the following manner: determine a sliding window with a target size corresponding to the target image, and a sliding step of the sliding window; and divide the target image by sliding the sliding window according to the sliding step to obtain multiple sub-images included in the target image.

[0087] Here, the target image is divided by a sliding window. In actual application, first, a sliding window with a target size corresponding to the target image is determined, which can be pre-set or determined according to the size of the target image; and a sliding step of the sliding window is determined. Then, the target image is divided by sliding the sliding window on the target image to obtain a plurality of sub-images included in the target image. In actual implementation, the starting position, sliding direction and sliding step of the sliding window can be pre-set, which can be pre-set or determined according to the size of the target image. For example, when the target image is a pathological image, since the pathological image has a huge size of terapixel level and heterogeneity, the pathological image can be placed under 20 times for image division, and the size of the sliding window can be 1120*1120, so as to obtain a plurality of sub-images with the size of 1120*1120.

[0088] Step 102: feature extraction is performed on each sub-image to obtain sub-image features of each sub-image.

[0089] In step 102, for the plurality of sub-images included in the target image, feature extraction is performed on each sub-image to obtain sub-image features of each sub-image. In some embodiments, the terminal can perform feature extraction on each sub-image to obtain sub-image features of each sub-image in the following manner: first dimension feature extraction is performed on each sub-image to obtain intermediate sub-image features of each sub-image in the first dimension; the intermediate sub-image features of each sub-image in the first dimension are spliced to obtain an intermediate sub-image feature matrix of the target image; second dimension feature extraction is performed on the intermediate sub-image feature matrix to obtain a sub-image feature matrix of the target image, the sub-image feature matrix including sub-image features of each sub-image in the second dimension, the second dimension being lower than the first dimension.

[0090] Here, we can first perform feature extraction of the first dimension on each sub-graph to obtain the intermediate sub-graph features of the first dimension for each sub-graph. For example, the feature extraction of the first dimension can be implemented using the pre-trained EfficientNet-B0 model to improve the accuracy of feature extraction. Then, the intermediate sub-graph features of the first dimension of each of the multiple sub-graphs are spliced ​​to obtain the intermediate sub-graph feature matrix of the target image. The intermediate sub-graph feature matrix is ​​then subjected to feature extraction of the second dimension to obtain the sub-graph feature matrix of the target image. The sub-graph feature matrix includes the sub-graph features of the second dimension of each sub-graph, and the second dimension is lower than the first dimension. In practical applications, the feature extraction of the second dimension can be implemented through a pre-trained backbone network. For the backbone network, a Transformer network with a residual structure can be used, which can not only achieve feature dimensionality reduction, but also realize feature interaction between the sub-graphs in the target image, thereby improving the accuracy of feature extraction, so that the obtained sub-graph features can better characterize the corresponding sub-graphs.

[0091] In this way, the extracted intermediate sub-image features are first spliced ​​to obtain the intermediate sub-image feature matrix of the target image, and then feature extraction is performed on the intermediate sub-image feature matrix. This can realize feature interaction between the sub-images in the target image, thereby improving the accuracy of feature extraction and reducing the feature dimension, thereby increasing the calculation speed and reducing the occupation of computing resources.

[0092] Step 103: Based on the sub-graph features of each sub-graph, a first classification prediction is performed on each sub-graph to obtain a sub-graph classification result of each sub-graph.

[0093] In step 103, after obtaining the sub-graph features of each sub-graph, a first classification prediction can be performed on each sub-graph based on the sub-graph features of each sub-graph to obtain a sub-graph classification result for each sub-graph. The sub-graph classification result is used to indicate the category to which the corresponding sub-graph belongs, and the sub-graph classification result can also include the instance prediction probability of the sub-graph belonging to each category. Here, the first classification prediction can be used for classification prediction of C categories. In practical applications, the first classification prediction can be implemented by the instance-level classification layer of the image classification model. The image classification model can be pre-trained, and the image classification model can be based on a neural network architecture, such as a convolutional neural network, a deep neural network, etc.

[0094] Step 104: for each subgraph, based on the subgraph classification result of the subgraph, determine the weight of the subgraph, and based on the subgraph feature and the weight of the subgraph, determine the weighted subgraph feature of the subgraph.

[0095] In step 104, after obtaining the sub-image classification results for each sub-image, the following processing is performed on each sub-image: based on the sub-image classification results of the sub-image, the weight of the sub-image is determined, and based on the sub-image features and weights of the sub-image, the weighted sub-image features of the sub-image are determined. It should be noted that the weight indicates the degree of influence of the sub-image (or the sub-image features of the sub-image) on the image classification results of the target image. In this way, the sub-image features of the sub-image can be weighted according to the weight, thereby improving the accuracy of the classification prediction of the target image based on the sub-image features of the sub-image.

[0096] In some embodiments, the terminal may determine the weight of the sub-graph based on the sub-graph classification result of the sub-graph in the following manner: obtain the number of classification categories predicted by the first classification, and determine the uniform distribution of the number of classification categories; determine the difference between the sub-graph classification result of the sub-graph and the uniform distribution; and use the difference as the weight of the sub-graph.

[0097] Here, we can first obtain the number of classification categories predicted by the first classification, such as 2 or 3 categories. Then, we can determine the uniform distribution corresponding to the number of classification categories. For example, when the number of classification categories is 3, the uniform distribution is [1 / 3, 1 / 3, 1 / 3], and when the number of classification categories is 2, the uniform distribution is [1 / 2, 1 / 2]. Then, we can determine the difference between the subgraph classification result and the uniform distribution, and use the difference as the weight of the subgraph. For example, we can determine the KL (Kullback-Leibler) divergence between the subgraph classification result and the uniform distribution, and then use the KL divergence as the weight of the subgraph.

[0098] In some embodiments, the terminal may determine the weighted subgraph feature of the subgraph based on the subgraph feature and weight of the subgraph in the following manner: multiplying the subgraph feature of the subgraph by the weight of the subgraph to obtain the weighted subgraph feature of the subgraph.

[0099] Step 105: Based on the weighted sub-image features of the multiple sub-images, a second classification prediction is performed on the target image to obtain an image classification result of the target image.

[0100] In step 105, after obtaining the weighted sub-graph features of the multiple sub-graphs in the target image, a second classification prediction is performed on the target image based on the weighted sub-graph features of each of the multiple sub-graphs in the target image to obtain an image classification result of the target image. The image classification result is used to indicate the category to which the target image belongs, and the image classification result may also include the bag-level prediction probability that the target image belongs to each category. In practical applications, the number of classification categories that can be achieved by the second classification prediction may be the same as the number of classification categories that can be achieved by the first classification prediction. The second classification prediction can be achieved by the bag-level classification layer of the image classification model, which may be pre-trained, and the image classification model may be based on a neural network architecture, such as a convolutional neural network, a deep neural network, etc.

[0101] Next, the training of the image classification model is described. In some embodiments, see Figure 4 , Figure 4 203, based on the difference between the instance classification result and the sub-image label of each sub-image sample, the model parameters of the image classification model are updated to train the image classification model.

[0102] Here, in step 201, an image sample set for training an image classification model is first obtained. The image sample set includes multiple image samples, each of which includes multiple sub-image samples. The sub-image samples are obtained by dividing the image samples. For example, for each image sample, a sliding window of a target size corresponding to the image sample and a sliding step size of the sliding window are determined; the image sample is divided by sliding the sliding window according to the sliding step size to obtain multiple sub-image samples included in the image sample. Each image sample is annotated with an image label, and each sub-image sample is annotated with a sub-image label. In practical applications, when initializing the sub-image label of a sub-image sample, the image label of the image sample to which the sub-image sample belongs can be used as the sub-image label of the sub-image sample. However, since sub-image samples may exist in the image sample that are unrelated to the image label, the sub-image label of the sub-image sample can be considered a soft label.

[0103] In step 202, the instance-level classification layer of the image classification model performs a first classification prediction on each sub-image sample in each image sample to obtain an instance classification result for each sub-image sample. In some embodiments, the terminal may obtain the instance classification result for each sub-image sample by obtaining sub-image sample features of each sub-image sample in each image sample; and performing a first classification prediction on each sub-image sample based on the sub-image sample features of each sub-image sample using the instance-level classification layer to obtain an instance classification result for each sub-image sample.

[0104] Here, the following processing can be performed for each image sample: the sub-image sample features of each sub-image sample in the image sample are obtained, for example, feature extraction is performed on each sub-image sample to obtain the sub-image sample features of each sub-image sample. Here, the feature extraction process can be the same as the feature extraction process of the above-mentioned sub-image features, and will not be described in detail here. Therefore, based on the sub-image sample features of each sub-image sample in the image sample, a first classification prediction is performed on each sub-image sample in each image sample to obtain an instance classification result for each sub-image sample. The instance classification result is used to indicate the category to which the corresponding sub-image sample belongs, and the instance classification result can also indicate the instance prediction probability that the corresponding sub-image sample belongs to each category.

[0105] In step 203, the following processing is performed for each image sample:

[0106] In step 2031, a second classification prediction is performed on the image sample using a bag-level classification layer in combination with the sub-image labels of each sub-image sample in the image sample to obtain a sample classification result for the image sample. In some embodiments, the terminal may obtain the sample classification result for the image sample by: determining a sub-image weight for each sub-image sample based on the sub-image label of each sub-image sample in the image sample, and determining a weighted sub-image sample feature for each sub-image sample based on the sub-image sample features and sub-image weight of each sub-image sample; and performing a second classification prediction on the image sample using a bag-level classification layer based on the weighted sub-image sample features of multiple sub-image samples in the image sample to obtain a sample classification result for the image sample.

[0107] Here, for each sub-image sample, a sub-image weight of the sub-image sample can be determined based on the sub-image label and instance classification result of the sub-image sample. The sub-image weight indicates the degree of influence of the sub-image sample on the sample classification result of the image sample. In some embodiments, the terminal can determine the sub-image weight of each sub-image sample based on the sub-image label of each sub-image sample in the image sample in the following manner: for each sub-image sample in the image sample, respectively perform the following processing: obtain the number of classification categories predicted by the first classification and determine the uniform distribution of the number of classification categories; determine the difference between the sub-image label of the sub-image sample and the uniform distribution; and use the difference as the sub-image weight of the sub-image sample.

[0108] Here, when calculating the subgraph weight of a subgraph sample, the number of classification categories predicted by the first classification can be obtained first, such as 2 categories, 3 categories, etc. Then, the uniform distribution corresponding to the number of classification categories is determined. For example, when the number of classification categories is 3, the uniform distribution is [1 / 3, 1 / 3, 1 / 3], and when the number of classification categories is 2, the uniform distribution is [1 / 2, 1 / 2]. Then, the difference between the instance classification result of the subgraph sample and the uniform distribution is determined, and the difference is used as the subgraph weight of the subgraph sample. For example, the KL (Kullback-Leibler) divergence between the instance classification result and the uniform distribution can be determined, and then the KL divergence can be used as the subgraph weight of the subgraph sample.

[0109] Continuing, the sub-image sample features of the sub-image samples are weighted based on the sub-image weights (i.e., the sub-image weights and the sub-image sample features are multiplied together) to obtain weighted sub-image sample features. Furthermore, based on the weighted sub-image sample features of the multiple sub-image samples in the image sample, a second classification prediction is performed on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample. The sample classification result is used to indicate the category to which the image sample belongs, and the sample classification result can also indicate the bag-level prediction probability that the image sample belongs to each category.

[0110] Step 2032: Determine the model loss value of the image classification model based on the difference between the instance classification results of each sub-image sample and the sub-image label, and the difference between the sample classification results of the image sample and the image label. Figure 5 , Figure 5 This is a schematic diagram of the training process of the image classification model provided in the embodiment of the present application. Figure 5 Shown Figure 4 Step 2032 in the figure can be implemented through steps 301 to 304: Step 301, for each sub-image sample, based on the difference between the instance classification result of the sub-image sample and the sub-image label, determine the intermediate instance-level loss of the image classification model; Step 302, add up the intermediate instance-level losses of the image classification model to obtain the instance-level loss of the image classification model; Step 303, based on the difference between the sample classification result of the image sample and the image label, determine the bag-level loss of the image classification model; Step 304, based on the first loss weight of the instance-level loss and the second loss weight of the bag-level loss, perform weighted summation on the instance-level loss and the bag-level loss to obtain the model loss value of the image classification model.

[0111] Here, in step 301, the loss function of the intermediate instance-level loss may be a cross-entropy loss function, an exponential loss function, etc. In step 303, the loss function of the bag-level loss may also be a cross-entropy loss function, an exponential loss function, etc. In step 304, the first loss weight and the second loss weight may be preset based on experience.

[0112] Step 2033: Based on the model loss value, the model parameters of the image classification model are updated to train the image classification model. Here, when the model loss value is greater than the loss threshold, an error signal of the image classification model can be determined based on the loss function. The error signal is then backpropagated through the image classification model, and the model parameters of each layer in the image classification model are updated during the propagation process. This achieves training of the image classification model.

[0113] It should be noted that since the sub-image labels of the sub-image samples are soft labels, there may be noise labels in the sub-image labels to affect the model training effect. Based on this, in the embodiment of the present application, the sub-image labels of the sub-image samples can also be updated during the training process to achieve label disambiguation (or label denoising), thereby improving the training effect of the image classification model. The first classification prediction is used to perform classification prediction of C (C is an integer greater than 1) categories for illustration, see Figure 6 , Figure 6 This is a schematic diagram of the training process of the image classification model provided in the embodiment of the present application. Figure 6 Shown in Figure 4 Before step 203 shown in the figure, and after step 202, the following steps are also included:

[0114] Step 401: Based on the instance classification results of each sub-image sample in the image sample set, determine the key sub-images of each category obtained by the first classification prediction from multiple sub-image samples in the image sample set; Step 402: For each category, obtain the prototype features and corresponding prototype labels of the category, and use the key sub-image of the category to momentum update the prototype features to obtain new prototype features of the category; Step 403: For each sub-image sample, determine the similarity between the sub-image features of the sub-image sample and the new prototype features of each category, and use the prototype label corresponding to the target new prototype feature with the highest similarity as the sub-image prototype label of the sub-image sample; Step 404: For each sub-image sample, use the sub-image prototype label of the sub-image sample to momentum update the sub-image label of the sub-image sample to obtain a new sub-image label of the sub-image sample.

[0115] After obtaining the instance classification results of each sub-image sample in the image sample set in step 202, in step 401, based on the instance classification results of each sub-image sample in the image sample set, key sub-images of each category are determined from the multiple sub-image samples in the image sample set. Here, the instance classification results include the instance prediction probabilities of the sub-image samples belonging to each category. Therefore, the terminal can determine the key sub-images of each category from the multiple sub-image samples in the image sample set in the following manner: for each category, the following processing is performed separately: from the multiple sub-image samples in the image sample set, a target number of target sub-image samples belonging to the category and ranked in descending order of instance prediction probabilities are determined; and the target number of target sub-image samples are used as the target number of key sub-images of the category. Here, the target number is less than the number of sub-image samples.

[0116] In step 402, for each category, the key subgraph of the category can be used to momentum update the prototype features of the category to obtain a new prototype feature of the category. Here, for each category, the prototype features and prototype labels of the category can be pre-initialized. The initialized prototype features can be an all-zero vector, and the category of the prototype label can correspond to the classification category that can be achieved by the image classification model (i.e., the classification category that can be achieved by the second classification prediction).

[0117] In some embodiments, see Figure 7 , Figure 7 It is a schematic diagram of the momentum update process of the prototype feature provided in an embodiment of the present application. Figure 7 Shown Figure 6 Step 402 in the figure can be implemented by the following steps: perform the following processing for each category: Step 4021a, use the 1st key subgraph of the category, momentum update the prototype feature, and obtain the 1st intermediate prototype feature of the category; Step 4022a, use the i-th key subgraph of the category, momentum update the (i-1)-th intermediate prototype feature of the category, and obtain the i-th intermediate prototype feature of the category, where i is greater than 0 and not greater than N; Step 4023a, traverse i, obtain the N-th intermediate prototype feature of the category, and use the N-th intermediate prototype feature of the category as the new prototype feature of the category.

[0118] In steps 4021a to 4023a, the prototype features are updated once through each key subgraph respectively, so as to obtain the final new prototype features. It should be noted that, taking "using the first key subgraph of the category, momentum updating the prototype features, and obtaining the first intermediate prototype features of the category" as an example, the specific implementation of each momentum update is explained: obtaining the key subgraph features of the first key subgraph; obtaining the third feature weight of the key subgraph features and the fourth feature weight of the prototype features; based on the third feature weight and the fourth feature weight, performing weighted summation processing on the key subgraph features and the prototype features to obtain the first intermediate prototype features of the category. Here, the third feature weight and the fourth feature weight can be set according to empirical values.

[0119] In some embodiments, see Figure 8 , Figure 8 It is a schematic diagram of the momentum update process of the prototype feature provided in an embodiment of the present application. Figure 8 Shown Figure 6 Step 402 in the above example can also be implemented by performing the following steps: performing the following processing for each category: Step 4021b, obtaining the key subgraph features of each key subgraph of the category, and determining the average subgraph features of the multiple key subgraph features; Step 4022b, obtaining the first feature weight of the average subgraph feature and the second feature weight of the prototype feature; Step 4023b, based on the first feature weight and the second feature weight, performing weighted summation processing on the average subgraph feature and the prototype feature to obtain a new prototype feature of the category. Here, the prototype feature is updated with only one momentum based on the average subgraph feature of the key subgraph features of the multiple key subgraphs to obtain the new prototype feature. The first feature weight and the second feature weight can also be set based on empirical values.

[0120] In step 403, the following processing can be performed on each sub-image sample: first, the similarity between the sub-image feature of the sub-image sample and the new prototype features of each category is determined, and then the prototype label corresponding to the target new prototype feature with the highest similarity is used as the sub-image prototype label of the sub-image sample.

[0121] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of the momentum update process of the sub-graph label provided in an embodiment of the present application. Figure 9 Shown Figure 4Step 404 can be implemented as follows: For each sub-graph sample, the following processing is performed: Step 4041: Obtain a first label weight for the sub-graph prototype label and a second label weight for the sub-graph label; Step 4042: Based on the first label weight and the second label weight, perform a weighted summation of the sub-graph prototype label and the sub-graph label to obtain a new sub-graph label for the sub-graph sample. Here, the sub-graph prototype label can be used to perform a momentum update on the sub-graph label. The first label weight and the second label weight can be set based on empirical values.

[0122] In some embodiments, after the sub-image labels are updated through steps 401 to 404, the updated sub-image labels are used for all sub-image labels required in step 203. That is, in step 2031, the bag-level classification layer is used to perform a second classification prediction on the image sample in combination with the new sub-image labels of each sub-image sample in the image sample to obtain a sample classification result for the image sample; in step 2032, the model loss value of the image classification model is determined based on the difference between the instance classification result of each sub-image sample and the new sub-image label, as well as the difference between the sample classification result of the image sample and the image label.

[0123] For example, if the target image is a pathology image, the image classification model can be used to classify the pathology image and determine the category to which the pathology image belongs. Here, the category can be "lesion present" or "lesion absent," or it can be the grade (e.g., grade 1, grade 2, etc.) and type (e.g., benign lesion, malignant lesion, early stage lesion, mid-stage lesion, late stage lesion, etc.) of the lesion.

[0124] By applying the above embodiment of the present application, when performing classification prediction on a target image, the target image is first divided to obtain a plurality of sub-images included in the target image, and then feature extraction is performed on each sub-image to obtain the sub-image features of each sub-image, and based on the sub-image features of each sub-image, a first classification prediction is performed on each sub-image to obtain the sub-image classification results of each sub-image; thereby, based on the sub-image classification results of each sub-image, the weight of each sub-image is determined, and based on the sub-image features and weights of each sub-image, the sub-image features are weighted to obtain the weighted sub-image features of each sub-image, and finally, based on the weighted sub-image features of each sub-image, a second classification prediction is performed on the target image to obtain the image classification result of the target image. Since the weight of each sub-image feature indicates the degree of influence of the sub-image on the image classification result of the target image, the weighted sub-image features obtained by weighting can better characterize the target image, so that the image classification result obtained based on the classification prediction of each weighted sub-image feature is more accurate, thereby improving the accuracy of image classification.

[0125] The following will take the image as a pathological image as an example to illustrate the exemplary application of the embodiment of the present application in an actual application scenario. In the medical scenario, the doctor obtains highly suspected lesion tissue by puncture, and processes it into a pathological slice image (i.e., a pathological image) in the pathology department. The traditional pathological image analysis method requires specially trained pathologists to find the region of interest (ROI) one by one under a microscope, and then analyze and diagnose whether there is a lesion area, the lesion grade, etc. based on professional knowledge. However, the traditional pathological image analysis method will bring a huge workload to the physician; and the accuracy of the pathological diagnosis is directly related to the experience of the pathologist, and the analysis results are subjective and difficult to replicate. Therefore, digital pathology plays an important role in the diagnosis and interpretation of diseases and is receiving more and more attention.

[0126] Since the pathological slice image WSI has a huge gigapixel size and heterogeneity, in an embodiment of the present application, a multi-instance learning (MIL) scheme can be used to perform pathological image analysis. The embodiment of the present application provides an image processing method based on MIL, including: 1) to overcome the weak supervision nature of MIL, the instance-level MIL is formalized as a noisy label learning task, and an instance-level supervision based on label disambiguation is designed for MIL, that is, prototype vectors and confidence libraries are used to reduce the influence of noisy labels; 2) the confidence library in the instance-level supervision is used to perform pre-attention weighting on the bag-level supervision to achieve interaction between the two supervisions, thereby utilizing the joint interactive supervision of the instance-level and bag-level to improve the performance of MIL.

[0127] In practical applications, the embodiments of the present application can be used for pathological image analysis, automatically learning effective features from pathological images to provide accurate and consistent auxiliary diagnosis suggestions, providing reference opinions for preoperative decision-making, and reducing the workload of pathologists, allowing them to focus on difficult or controversial cases. The embodiments of the present application can be applied to a variety of clinical classification scenarios based on pathological images. For example, it can be used to solve tasks such as cancer diagnosis, cancer typing, survival prediction, lymph node metastasis prediction, and gene mutation prediction for patients in the clinic, assisting doctors in making more accurate and efficient diagnoses.

[0128] Next, the image processing method provided by the embodiment of the present application is described in detail. Figure 10 , Figure 10 Schematic diagram of the training process of the image classification model provided in the embodiment of the present application. The image processing method provided in the embodiment of the present application includes the following steps:

[0129] (1) Collect multiple pathological images (such as Figure 10 S1, S2, ..., S shown N), and perform label determination of clinical targets on each pathological image to obtain bag-level labels for each pathological image (i.e., the category to which it belongs, the number of categories is C). For each pathological image, the pathological image is divided into multiple sub-images (i.e., patches, which can also be called instances). For example, for a pathological image at 20x magnification, a sliding window of 1120×1120 can be used to slide within the pathological image with a step size of 1120 to obtain non-overlapping patches at different positions, that is, the pathological image is divided into multiple patches of 1120×1120. It should be noted that the multiple collected pathological images will be used as the training sample set required for training the image classification model (including pathological images of C categories), and the trained image classification model is used to perform classification predictions of C categories.

[0130] (2) For each pathological image, the image feature extractor is used to extract features from each patch obtained by segmenting the pathological image, and obtains a feature vector of each patch (for example, the dimension of the feature vector can be 1280). In practical applications, the image feature extractor can use the EfficientNet-B0 model pre-trained on the ImageNet dataset (such as Figure 10 for transfer learning.

[0131] (3) For each pathological image, the feature vectors of all patches in the pathological image are spliced ​​to obtain the feature matrix Instance Embedding of the pathological image (such as Figure 10 As shown in e1,....,e i ,....,e N For example, if the dimension of the feature vector of the patch is 1280, then the feature matrix can be an M×1280 matrix. M is a hyperparameter representing the number of patches obtained by randomly partitioning a single pathology image. If the total number of patches in the current pathology image is less than M, the remaining positions in the M×1280 matrix can be filled with zeros.

[0132] (4) For each pathological image, the feature matrix of the pathological image is input into the backbone network (such as Figure 10 As shown in ), feature extraction is performed through the Backbone network to obtain the instance feature matrix of the pathological image (as shown in Figure 10 The x1,....,x shown i ,....,x N), the instance feature matrix includes the instance features of each instance (i.e., Patch) in the pathological image. For example, the feature matrix of M×1280 can be input into the Backbone network, and the matrix of M×64 (i.e., instance feature matrix) can be output through the Backbone network. Here, for the Backbone network, a Transformer network with a residual structure can be used, which can not only achieve dimensionality reduction of features (for example, from dimension 1280 to dimension 64), but also achieve feature interaction between each patch in the pathological image. It should be noted that the instance feature matrix of the pathological image will subsequently undergo two-way processing. One way is the instance-level supervision network (Instance-level Supervision, such as Figure 10 The other is the bag-level supervision network (Bag-level Supervision, as shown in Figure 10 shown) processing.

[0133] (5) Figure 10 As shown in Figure 1, the instance-level supervision network includes an instance-level classifier and a label disambiguation module. The instance-level classifier can be composed of a LayerNorm layer and a Linear layer, and the output is an M×C matrix. The instance-level loss can be the cross-entropy loss calculated by the instance classification result (i.e., the predicted probability of each instance belonging to each category) of each instance (i.e., patch) predicted by the instance-level classifier and the instance soft label of each instance generated by the label disambiguation module.

[0134] (5.1) The label disambiguation unit includes a key feature selection unit, a prototype unit, and a confidence library unit. The label disambiguation unit is used to: first, select key instance features of each category through the key feature selection unit; then, update the prototype vector of the prototype unit using the key instance features and determine the prototype label of each instance; finally, update the instance soft label of each instance using the momentum of the prototype label of each instance through the confidence library unit.

[0135] (5.2) The input of the key feature selection unit is: "instance feature matrix (such as M×64 matrix)" and "instance prediction probability (such as M×C matrix)"; the output of the key feature selection unit is: K key instance features (such as K×64 matrix). The key feature selection unit is used to: for each of the C categories to which multiple pathological images belong, for example, category c, from the instances represented by the instance classification results belonging to category c, select the instance features of the K instances with the highest instance prediction probability in descending order as the K key instance features of category c. In practical applications, the number of key instance features for each category can be K. Of course, the value of K can also be dynamically adjusted according to the number of pathological image samples in each category.

[0136] (5.3) The prototype unit is used to maintain the prototype vector (which can also be an M×64 matrix) and the prototype label (which can also be an M×C matrix). Specifically, the prototype unit is used to momentum update the prototype vector of each category (which can be a 1×64 matrix) based on the K key instance features of each category. Taking the momentum update of the prototype vector of category c based on the K key instance features of category c as an example, it includes: 1) Momentum update the prototype vector once through each key instance feature to obtain the final new prototype vector. Momentum update refers to: obtaining a new prototype vector by weighted summing the prototype vector and the key instance feature. The weights of the prototype vector and the key instance feature can be set in advance based on experience. 2) Momentum update the prototype vector once through the average instance feature of the K key instance features to obtain a new prototype vector. Momentum update refers to: obtaining a new prototype vector by weighted summing the prototype vector and the average instance feature. The weights of the prototype vector and the average instance feature can be set in advance based on experience.

[0137] Here, the prototype vector maintained by the prototype unit is initialized with all zeros. After the prototype vector is updated based on the momentum of each key instance feature, the instance prototype label of each instance is further determined based on the similarity between the instance features of each instance and the updated new prototype vector of each category. Specifically, the prototype label of the new prototype vector with the highest similarity can be used as the instance prototype label of the corresponding instance. Here, the prototype label and the instance prototype label can be in the form of one-hot encoding; the similarity can be calculated using cosine similarity or other similarity calculation methods; the prototype label of each category is 0 or 1, or it can be in the form of a soft label.

[0138] (5.4) The confidence library unit is used to maintain the confidence library. The confidence library records the instance soft labels of all instance patches in the training sample set. The bag-level label of the pathological image can be used as the instance soft label of each instance included in the pathological image to initialize the instance soft label of each instance. During the training process, the instance prototype label of each instance can be used to momentum update the instance soft label of each instance. Specifically, the new instance soft label can be obtained by weighted summing the instance prototype label and the instance soft label to achieve momentum update. Here, the weights of the instance prototype label and the instance soft label can be set in advance based on experience. In this way, the instance-level loss can be calculated based on the instance soft label of each instance.

[0139] (6) Figure 10 As shown in Figure 1, the bag-level supervision network includes a pre-attention unit, a feature aggregator, and a bag-level classifier. The bag-level loss can be calculated by comparing the bag-level classification results (i.e., the bag-level predicted probabilities of pathological images belonging to each category) output by the bag-level classifier with the cross-entropy loss calculated between the bag-level labels.

[0140] (6.1) In practical applications, the feature aggregator can use a Transformer Decoder, and the bag-level classifier can use a Linear layer. The input of the bag-level supervised network is the instance feature matrix of the pathology image (e.g., an M × 64 matrix), and the output is the bag-level classification result (e.g., a 1 × 2 matrix, where 2 is the number of categories in the bag-level classifier). Here, the number of categories C in the instance classifier can be the same as the number of categories in the bag-level classifier, or it can be different.

[0141] (6.2) The pre-attention unit is used to realize the interaction between the bag-level supervision network and the instance-level supervision network. Among them, 1) Interaction in the training process: The label disambiguation unit of the instance-level supervision network outputs the instance soft label of each instance in the pathological image, and then the pre-attention unit calculates the KL divergence of each instance soft label and the uniform distribution (for example, the uniform distribution of the three categories is [1 / 3, 1 / 3, 1 / 3]) to obtain the pre-weighted weight of each instance; then the pre-weighted weight of each instance is multiplied by the instance feature of each instance output by the backbone network to obtain the pre-weighted instance feature of each instance; finally, the pre-weighted instance feature of each instance is input into the feature aggregator and the bag-level classifier in sequence to obtain the bag-level classification result. 2) Interaction in the testing process: Since the instance soft label of each instance does not exist in the test set, the instance classification result of each instance can be used instead of the instance soft label for calculation. The other processes are consistent with the training process.

[0142] In practical applications, the interaction between the instance-level supervision network and the bag-level supervision network is in the form of pre-attention weighting of the instance features of each instance. The pre-weighted weight is calculated by calculating the KL divergence between the instance soft label and the uniform distribution. Of course, other calculation methods can also be used to calculate the pre-weighted weight (such as calculating the entropy between the instance soft label and the uniform distribution, etc.); the positions of the pre-attention unit and the feature aggregator can be interchanged, so that after passing through the feature aggregator, the instance features of each instance can be pre-attention weighted to realize the interaction between the instance-level supervision network and the bag-level supervision network.

[0143] (7) During the training phase of the image classification model, an end-to-end training method is adopted. The total loss of the model is obtained by weighted summation of bag-level loss and instance-level loss. In practical applications, the model is trained and evaluated using training sample sets and validation sample sets. The early termination method can be used to control the training time and select the model weights. That is, when the model performance on the validation sample set no longer improves after multiple iterations, the training is terminated and the model weights that perform best on the validation sample set are saved.

[0144] (8) After the image classification model training is completed, the optimal model parameters obtained through training can be loaded to predict the image classification task in order to test the image classification model.

[0145] Applying the above-mentioned embodiments of the present application, a multi-instance learning framework using bag-level and instance-level interactive supervision is proposed. Among them, on the one hand, a label disambiguation unit is introduced, and the instance labels are softened using prototype vectors and confidence libraries to reduce the label noise impact of instance-level supervision; on the other hand, bag-level supervision is pre-weighted by instance-level supervision to achieve joint interaction between the two types of supervision. In this way, all information in the image can be more efficiently and organically integrated, and more accurate classification results can be obtained, thereby improving the performance of MIL; thus, when the embodiments of the present application are applied to medical scenarios, medical solutions can be provided more accurately, improving the quality of life of patients.

[0146] The following continues to describe the exemplary structure of the image processing device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2As shown, the software modules stored in the image processing device 555 of the memory 550 may include: a division module 5551, used to divide the target image to obtain multiple sub-images included in the target image; a feature extraction module 5552, used to extract features of each sub-image to obtain sub-image features of each sub-image; a first prediction module 5553, used to perform a first classification prediction on each sub-image based on the sub-image features of each sub-image to obtain a sub-image classification result of each sub-image; a determination module 5554, used to determine the weight of each sub-image based on the sub-image classification result of the sub-image, and determine the weighted sub-image feature of the sub-image based on the sub-image feature and the weight of the sub-image, wherein the weight is used to indicate the degree of influence of the sub-image on the image classification result of the target image; a second prediction module 5555, used to perform a second classification prediction on the target image based on the weighted sub-image features of the multiple sub-images to obtain the image classification result of the target image.

[0147] In some embodiments, the division module 5551 is also used to determine a sliding window with a target size corresponding to the target image and a sliding step of the sliding window; by sliding the sliding window according to the sliding step, the target image is divided to obtain multiple sub-images included in the target image.

[0148] In some embodiments, the feature extraction module 5552 is further used to perform feature extraction of the first dimension on each of the sub-images to obtain intermediate sub-image features of the first dimension for each of the sub-images; to splice the intermediate sub-image features of the first dimension of each of the multiple sub-images to obtain an intermediate sub-image feature matrix of the target image; and to perform feature extraction of the second dimension on the intermediate sub-image feature matrix to obtain a sub-image feature matrix of the target image, wherein the sub-image feature matrix includes the sub-image features of the second dimension for each of the sub-images, and the second dimension is lower than the first dimension.

[0149] In some embodiments, the determination module 5554 is further used to obtain the number of classification categories predicted by the first classification, and determine the uniform distribution of the number of classification categories; determine the difference between the sub-graph classification result of the sub-graph and the uniform distribution; and use the difference as the weight of the sub-graph.

[0150] In some embodiments, the determining module 5554 is further configured to multiply the subgraph feature of the subgraph by the weight of the subgraph to obtain the weighted subgraph feature of the subgraph.

[0151] In some embodiments, the second prediction module 5555 is further used to perform feature aggregation processing on the weighted sub-graph features of the multiple sub-graphs to obtain aggregated sub-graph features; based on the aggregated sub-graph features, perform a second classification prediction on the target image to obtain an image classification result of the target image.

[0152] In some embodiments, the first classification prediction is implemented by an instance-level classification layer of an image classification model, and the second classification prediction is implemented by a bag-level classification layer of the image classification model; the apparatus further comprises: a model training module 5556; the model training module 5556 is configured to obtain an image sample set for training the image classification model, the image sample set comprising a plurality of image samples, the image samples being annotated with image labels, the image samples comprising a plurality of sub-image samples, the sub-image samples being annotated with sub-image labels; and performing a first classification prediction on each of the sub-image samples in each of the image samples through the instance-level classification layer, Obtain an instance classification result for each of the sub-image samples; perform the following processing for each of the image samples: combine the sub-image labels of each of the sub-image samples in the image sample, perform a second classification prediction on the image sample through the bag-level classification layer, and obtain a sample classification result for the image sample; based on the difference between the instance classification result and the sub-image label of each of the sub-image samples, and the difference between the sample classification result and the image label of the image sample, determine the model loss value of the image classification model; based on the model loss value, update the model parameters of the image classification model to train the image classification model.

[0153] In some embodiments, the model training module 5556 is further used to determine, for each of the sub-image samples, the intermediate instance-level loss of the image classification model based on the difference between the instance classification result of the sub-image sample and the sub-image label; add the intermediate instance-level losses of the image classification model to obtain the instance-level loss of the image classification model; determine the bag-level loss of the image classification model based on the difference between the sample classification result of the image sample and the image label; and perform weighted summation processing on the instance-level loss and the bag-level loss based on the first loss weight of the instance-level loss and the second loss weight of the bag-level loss to obtain the model loss value of the image classification model.

[0154] In some embodiments, the model training module 5556 is further used to obtain the sub-image sample features of each of the sub-image samples in each of the image samples; through the instance-level classification layer, based on the sub-image sample features of each of the sub-image samples, a first classification prediction is performed on each of the sub-image samples to obtain an instance classification result of each of the sub-image samples; the model training module 5556 is further used to determine the sub-image weight of each of the sub-image samples based on the sub-image label of each of the sub-image samples in the image sample, and determine the weighted sub-image sample features of each of the sub-image samples based on the sub-image sample features and sub-image weight of each of the sub-image samples; based on the weighted sub-image sample features of multiple sub-image samples in the image sample, a second classification prediction is performed on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample.

[0155] In some embodiments, the model training module 5556 is further used to perform the following processing for each of the sub-image samples in the image sample: obtaining the number of classification categories predicted by the first classification, and determining the uniform distribution of the number of classification categories; determining the difference between the sub-image label of the sub-image sample and the uniform distribution; and using the difference as the sub-image weight of the sub-image sample.

[0156] In some embodiments, the model training module 5556 is also used to determine, before performing the following processing on each of the image samples, based on the instance classification results of each of the sub-image samples in the image sample set, the key sub-images of each category obtained by the first classification prediction from multiple sub-image samples in the image sample set; for each of the categories, obtain the prototype features and corresponding prototype labels of the category, and use the key sub-images of the category to momentum update the prototype features to obtain new prototype features of the category; for each of the sub-image samples, determine the similarity between the sub-image features of the sub-image sample and the new prototype features of each of the categories, and use the prototype label corresponding to the target new prototype feature with the highest similarity as the sub-image prototype label of the sub-image sample; for each of the sub-image samples, use the sub-image prototype label of the sub-image sample to momentum update the sub-image label of the sub-image sample to obtain a new sub-image label of the sub-image sample.

[0157] In some embodiments, the model training module 5556 is further used to combine the new sub-image label of each sub-image sample in the image sample, perform a second classification prediction on the image sample through the bag-level classification layer, and obtain the sample classification result of the image sample; the model training module 5556 is also used to determine the model loss value of the image classification model based on the difference between the instance classification result of each sub-image sample and the new sub-image label, and the difference between the sample classification result of the image sample and the image label.

[0158] In some embodiments, the instance classification result includes the instance prediction probability of the sub-image sample belonging to each of the categories; the model training module 5556 is also used to perform the following processing for each of the categories: from multiple sub-image samples in the image sample set, determine the target number of target sub-image samples belonging to the category and sorted in descending order of instance prediction probability; use the target number of target sub-image samples as the target number of key sub-images of the category.

[0159] In some embodiments, the number of the key subgraphs is N, where N is an integer greater than 1. The model training module 5556 is further used to adopt the 1st key subgraph of the category, momentum update the prototype feature, and obtain the 1st intermediate prototype feature of the category; adopt the i-th key subgraph of the category, momentum update the (i-1)th intermediate prototype feature of the category, and obtain the i-th intermediate prototype feature of the category, where i is greater than 0 and not greater than N; traverse i to obtain the Nth intermediate prototype feature of the category, and use the Nth intermediate prototype feature of the category as the new prototype feature of the category.

[0160] In some embodiments, there are multiple key subgraphs, and the model training module 5556 is further used to obtain the key subgraph features of each key subgraph of the category, and determine the average subgraph feature of the multiple key subgraph features; obtain the first feature weight of the average subgraph feature, and the second feature weight of the prototype feature; based on the first feature weight and the second feature weight, perform weighted summation processing on the average subgraph feature and the prototype feature to obtain a new prototype feature of the category.

[0161] In some embodiments, the model training module 5556 is further used to obtain a first label weight of the sub-graph prototype label and a second label weight of the sub-graph label; based on the first label weight and the second label weight, the sub-graph prototype label and the sub-graph label are weighted summed to obtain a new sub-graph label for the sub-graph sample.

[0162] By applying the above embodiment of the present application, when performing classification prediction on a target image, the target image is first divided to obtain a plurality of sub-images included in the target image, and then feature extraction is performed on each sub-image to obtain the sub-image features of each sub-image, and based on the sub-image features of each sub-image, a first classification prediction is performed on each sub-image to obtain the sub-image classification results of each sub-image; thereby, based on the sub-image classification results of each sub-image, the weight of each sub-image is determined, and based on the sub-image features and weights of each sub-image, the sub-image features are weighted to obtain the weighted sub-image features of each sub-image, and finally, based on the weighted sub-image features of each sub-image, a second classification prediction is performed on the target image to obtain the image classification result of the target image. Since the weight of each sub-image feature indicates the degree of influence of the sub-image on the image classification result of the target image, the weighted sub-image features obtained by weighting can better characterize the target image, so that the image classification result obtained based on the classification prediction of each weighted sub-image feature is more accurate, thereby improving the accuracy of image classification.

[0163] The present application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or the computer program from the computer-readable storage medium and executes the computer-executable instructions or the computer program, causing the electronic device to perform the image processing method provided in the present application.

[0164] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image processing method provided in the embodiment of the present application.

[0165] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0166] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0167] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0168] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0169] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Dividing the target image to obtain a plurality of sub-images included in the target image; Performing feature extraction on each of the subgraphs to obtain subgraph features of each of the subgraphs; Based on the sub-graph features of each of the sub-graphs, a first classification prediction is performed on each of the sub-graphs to obtain a sub-graph classification result for each of the sub-graphs; For each of the sub-graphs, determining a weight of the sub-graph based on a sub-graph classification result of the sub-graph, and determining a weighted sub-graph feature of the sub-graph based on the sub-graph feature and the weight, wherein the weight is used to indicate a degree of influence of the sub-graph on an image classification result of the target image; Based on the weighted sub-graph features of the multiple sub-graphs, a second classification prediction is performed on the target image to obtain an image classification result of the target image.

2. The method according to claim 1, wherein The target image is divided to obtain a plurality of sub-images of the target image, including: Determining a sliding window of a target size corresponding to the target image and a sliding step size of the sliding window; The target image is divided by sliding the sliding window according to the sliding step size to obtain a plurality of sub-images included in the target image.

3. The method according to claim 1, wherein The extracting features of each subgraph to obtain subgraph features of each subgraph includes: Performing feature extraction of the first dimension on each of the subgraphs to obtain intermediate subgraph features of the first dimension for each of the subgraphs; splicing the intermediate sub-image features of the first dimension of each of the multiple sub-images to obtain an intermediate sub-image feature matrix of the target image; Perform feature extraction of a second dimension on the intermediate sub-image feature matrix to obtain a sub-image feature matrix of the target image, wherein the sub-image feature matrix includes the sub-image features of a second dimension of each sub-image, where the second dimension is lower than the first dimension.

4. The method according to claim 1, wherein The determining the weight of the subgraph based on the subgraph classification result of the subgraph includes: Obtaining the number of classification categories predicted by the first classification, and determining a uniform distribution of the number of classification categories; determining a difference between a subgraph classification result of the subgraph and the uniform distribution; The difference is used as the weight of the subgraph.

5. The method according to claim 1, wherein The determining, based on the subgraph feature and the weight of the subgraph, a weighted subgraph feature of the subgraph includes: The subgraph feature of the subgraph and the weight of the subgraph are multiplied to obtain the weighted subgraph feature of the subgraph.

6. The method according to claim 1, wherein The performing a second classification prediction on the target image based on the weighted sub-graph features of the plurality of sub-graphs to obtain an image classification result of the target image includes: Performing feature aggregation processing on the weighted subgraph features of the multiple subgraphs to obtain aggregated subgraph features; Based on the aggregated subgraph features, a second classification prediction is performed on the target image to obtain an image classification result of the target image.

7. The method according to claim 1, wherein The first classification prediction is implemented by an instance-level classification layer of an image classification model, and the second classification prediction is implemented by a bag-level classification layer of the image classification model; the method further includes: Acquire an image sample set for training the image classification model, wherein the image sample set includes a plurality of image samples, the image samples are annotated with image labels, and the image samples include a plurality of sub-image samples, and the sub-image samples are annotated with sub-image labels; Performing a first classification prediction on each of the sub-image samples in each of the image samples through the instance-level classification layer to obtain an instance classification result for each of the sub-image samples; For each of the image samples, the following processing is performed respectively: Combining the sub-image labels of the sub-image samples in the image sample, performing a second classification prediction on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample; Determining a model loss value of the image classification model based on a difference between the instance classification result of each of the sub-image samples and the sub-image label, and a difference between the sample classification result of the image sample and the image label; Based on the model loss value, the model parameters of the image classification model are updated to train the image classification model.

8. The method according to claim 7, wherein Determining a model loss value of the image classification model based on a difference between the instance classification result of each sub-image sample and the sub-image label, and a difference between the sample classification result of the image sample and the image label, includes: For each of the sub-image samples, determining an intermediate instance-level loss of the image classification model based on a difference between the instance classification result of the sub-image sample and the sub-image label; Adding the intermediate instance-level losses of the image classification model to obtain the instance-level loss of the image classification model; Determining a bag-level loss of the image classification model based on a difference between the sample classification result of the image sample and the image label; Based on the first loss weight of the instance-level loss and the second loss weight of the bag-level loss, a weighted summation process is performed on the instance-level loss and the bag-level loss to obtain a model loss value of the image classification model.

9. The method according to claim 7, wherein The step of performing a first classification prediction on each of the sub-image samples in each of the image samples through the instance-level classification layer to obtain an instance classification result of each of the sub-image samples includes: Obtaining sub-image sample features of each of the sub-image samples in each of the image samples; By means of the instance-level classification layer, based on the sub-image sample features of each of the sub-image samples, a first classification prediction is performed on each of the sub-image samples to obtain an instance classification result of each of the sub-image samples; Combining the sub-image labels of the sub-image samples in the image sample, performing a second classification prediction on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample includes: determining a subimage weight of each subimage sample based on a subimage label of each subimage sample in the image sample, and determining a weighted subimage sample feature of each subimage sample based on a subimage sample feature and a subimage weight of each subimage sample; Based on the weighted sub-image sample features of the plurality of sub-image samples in the image sample, a second classification prediction is performed on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample.

10. The method according to claim 9, wherein The determining, based on the sub-image label of each sub-image sample in the image sample, a sub-image weight of each sub-image sample includes: For each sub-image sample in the image sample, the following processing is performed respectively: Obtaining the number of classification categories predicted by the first classification, and determining a uniform distribution of the number of classification categories; determining a difference between the sub-image labels of the sub-image samples and the uniform distribution; The difference is used as the sub-graph weight of the sub-graph sample.

11. The method according to claim 7, wherein Before performing the following processing on each of the image samples, the method further includes: Based on the instance classification results of each of the sub-image samples in the image sample set, determining the key sub-images of each category obtained by the first classification prediction from the multiple sub-image samples in the image sample set; For each of the categories, a prototype feature and a corresponding prototype label of the category are obtained, and a key subgraph of the category is used to momentum update the prototype feature to obtain a new prototype feature of the category; For each of the sub-image samples, determining the similarity between the sub-image feature of the sub-image sample and the new prototype feature of each category, and using the prototype label corresponding to the target new prototype feature with the highest similarity as the sub-image prototype label of the sub-image sample; For each of the sub-image samples, the sub-image prototype label of the sub-image sample is used to momentum update the sub-image label of the sub-image sample to obtain a new sub-image label of the sub-image sample.

12. The method according to claim 11, wherein Combining the sub-image labels of the sub-image samples in the image sample, performing a second classification prediction on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample includes: Combining the new sub-image labels of the sub-image samples in the image sample, performing a second classification prediction on the image sample through the bag-level classification layer to obtain a sample classification result of the image sample; Determining a model loss value of the image classification model based on a difference between the instance classification result of each sub-image sample and the sub-image label, and a difference between the sample classification result of the image sample and the image label, includes: A model loss value of the image classification model is determined based on a difference between the instance classification result of each of the sub-image samples and the new sub-image label, and a difference between the sample classification result of the image sample and the image label.

13. The method according to claim 11, wherein The instance classification result includes a predicted instance probability that the sub-image sample belongs to each of the categories; and determining, based on the instance classification result of each of the sub-image samples in the image sample set, a key sub-image of each category obtained by the first classification prediction from a plurality of sub-image samples in the image sample set, includes: For each of the above categories, the following processing is performed: Determining, from the plurality of sub-image samples in the image sample set, a target number of target sub-image samples belonging to the category and having a higher target number than the target number of target sub-image samples ranked in descending order of instance prediction probability; The target subgraph samples of the target quantity are used as the key subgraphs of the target quantity of the category.

14. The method according to claim 11, wherein The number of the key subgraphs is N, where N is an integer greater than 1. The key subgraphs of the category are used to momentum update the prototype features to obtain new prototype features of the category, including: Using the first key subgraph of the category, momentum updates the prototype feature to obtain the first intermediate prototype feature of the category; Using the i-th key subgraph of the category, momentum updating the (i-1)-th intermediate prototype feature of the category to obtain the i-th intermediate prototype feature of the category, where i is greater than 0 and not greater than N; Traverse the i to obtain the Nth intermediate prototype feature of the category, and use the Nth intermediate prototype feature of the category as the new prototype feature of the category.

15. The method according to claim 11, wherein There are multiple key subgraphs, and the key subgraphs of the category are used to momentum update the prototype features to obtain new prototype features of the category, including: Obtaining a key subgraph feature of each of the key subgraphs of the category, and determining an average subgraph feature of the plurality of key subgraph features; Obtaining a first feature weight of the average sub-graph feature and a second feature weight of the prototype feature; Based on the first feature weight and the second feature weight, a weighted summation process is performed on the average subgraph feature and the prototype feature to obtain a new prototype feature of the category.

16. The method according to claim 11, wherein The step of using the sub-image prototype label of the sub-image sample and momentum updating the sub-image label of the sub-image sample to obtain a new sub-image label of the sub-image sample includes: Obtaining a first label weight of the subgraph prototype label and a second label weight of the subgraph label; Based on the first label weight and the second label weight, a weighted sum process is performed on the subgraph prototype label and the subgraph label to obtain a new subgraph label for the subgraph sample.

17. An image processing device, characterized in that: The device comprises: A division module, configured to divide a target image into multiple sub-images included in the target image; A feature extraction module, configured to extract features from each of the subgraphs to obtain subgraph features of each of the subgraphs; A first prediction module is configured to perform a first classification prediction on each of the subgraphs based on the subgraph features of the subgraphs to obtain a subgraph classification result for each of the subgraphs; a determination module, configured to determine, for each of the sub-graphs, a weight of the sub-graph based on a sub-graph classification result of the sub-graph, and determine a weighted sub-graph feature of the sub-graph based on the sub-graph feature and the weight, wherein the weight is used to indicate a degree of influence of the sub-graph on the image classification result of the target image; The second prediction module is used to perform a second classification prediction on the target image based on the weighted sub-image features of the multiple sub-images to obtain an image classification result of the target image.

18. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 16 when executing the computer-executable instructions stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 16 is implemented.

Images