Image processing method, device, equipment and storage medium

Abstract

The present application relates to an image processing method, apparatus, device and storage medium, and relates to the field of artificial intelligence technology. The method includes: extracting at least one first image block from a target medical image; mapping the image features of at least one first image block to a first classification space, and obtaining a first classification probability distribution for each of at least one first image block; extracting at least one second image block from the target medical image based on the position of the target image block in the target medical image; mapping the image features of at least one second image block to a first classification space, and obtaining a first classification probability distribution for each of at least one second image block; and obtaining the image classification of the target medical image based on the first classification probability distribution for each of at least one second image block. This solution improves the accuracy of medical image classification.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the continuous development of the application of artificial intelligence (AI) in the medical field, medical images can now be classified through AI to assist medical personnel in making medical-related decisions.

[0003] In related technologies, medical images can be classified through deep neural networks; for example, a medical image recognition model can be pre-trained, the complete medical image can be identified by the model, and the classification result of the medical image can be directly output.

[0004] However, the image recognition model in the related art performs recognition processing on the entire image, and the recognition result is easily affected by useless information in the image, resulting in low accuracy in the classification of medical images. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium, which can improve the accuracy of medical image classification. The technical solution is as follows.

[0006] In one aspect, an image processing method is provided, the method comprising:

[0007] extracting at least one first image block from a target medical image;

[0008] Mapping the image features of at least one of the first image blocks to a first classification space to obtain a first classification probability distribution for each of the at least one first image block; the classifications in the first classification probability distribution include at least two valid classifications and at least one invalid classification in the medical field;

[0009] extracting at least one second image block from the target medical image based on a position of the target image block in the target medical image; a distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one of the first image blocks, and a first classification probability distribution of the target image block indicates that the classification of the target image block is the valid classification;

[0010] Mapping the image feature of at least one of the second image blocks to the first classification space to obtain a first classification probability distribution of the at least one of the second image blocks;

[0011] An image classification of the target medical image is obtained based on the first classification probability distribution of each of at least one of the second image blocks.

[0012] In another aspect, an image processing method is provided, the method comprising:

[0013] Processing the first sample image block using a first classification model to obtain a second classification probability distribution for the first sample image block; the classifications in the second classification probability distribution include at least two valid classifications in the medical field; and the first sample image block is an image block in a first sample medical image;

[0014] Setting a classification label for the first sample image block based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs; the classification label is used to indicate that the classification of the first sample image block is one of the at least two valid classifications and at least one invalid classification;

[0015] Processing the first sample image block through a second classification model to obtain a first classification probability distribution of the first sample image block; the classifications in the first classification probability distribution include at least two of the valid classifications and at least one of the invalid classifications;

[0016] Based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block, updating the parameters of the second classification model to obtain an image classification model;

[0017] The image classification model is used to perform the step of mapping the image features of the image block to the first classification space in the method described in the above aspect.

[0018] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0019] A first image block extraction module, configured to extract at least one first image block from a target medical image;

[0020] a first classification module, configured to map image features of at least one of the first image blocks to a first classification space to obtain a first classification probability distribution for each of the at least one first image block; wherein the classifications in the first classification probability distribution include at least two valid classifications and at least one invalid classification in the medical field;

[0021] a second image block extraction module configured to extract at least one second image block from the target medical image based on a position of the target image block in the target medical image; the distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one of the first image blocks, and the first classification probability distribution of the target image block indicates that the classification of the target image block is the valid classification;

[0022] a second classification module, configured to map image features of at least one of the second image blocks to the first classification space to obtain a first classification probability distribution of each of the at least one of the second image blocks;

[0023] The third classification module is used to obtain the image classification of the target medical image based on the first classification probability distribution of at least one of the second image blocks.

[0024] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0025] a first processing module, configured to process the first sample image block using a first classification model to obtain a second classification probability distribution for the first sample image block; the classifications in the second classification probability distribution include at least two valid classifications in the medical field; and the first sample image block is an image block in a first sample medical image;

[0026] a label setting module, configured to set a classification label for the first sample image block based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs; the classification label is used to indicate that the first sample image block is classified as one of at least two of the valid classifications and at least one invalid classification;

[0027] a second processing module, configured to process the first sample image block using a second classification model to obtain a first classification probability distribution of the first sample image block, wherein the classifications in the first classification probability distribution include at least two of the valid classifications and at least one of the invalid classifications;

[0028] a first parameter updating module, configured to update parameters of the second classification model based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block, so as to obtain an image classification model;

[0029] The image classification model is used to perform the step of mapping the image features of the image block to the first classification space in the method described in the above aspect.

[0030] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the above-mentioned image processing method.

[0031] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor to implement the above-mentioned image processing method.

[0032] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image processing method.

[0033] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0034] First, at least one first image block is extracted from the target medical image, the image features of the first image block are mapped to the first classification space, and based on the position of the first image block belonging to the valid classification, at least one second image block is extracted from the target medical image, and the image classification to which the target medical image belongs is determined based on the mapping result of the image features of the second image block in the first classification space; through the above-mentioned two-level classification scheme, image blocks at key positions can be extracted from the target medical image, and the image classification of the target medical image can be determined based on the classification results of the image blocks at key positions, which can reduce the influence of images at non-key positions on the classification results of the entire target medical image, thereby improving the accuracy of medical image classification.

[0035] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] Figure 1 is a system configuration diagram of an image classification system involved in various embodiments of the present application;

[0038] Figure 2 is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0039] Figure 3is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0040] Figure 4 is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0041] Figure 5 is a diagram showing a training and testing framework for an image classification model for medical images according to an exemplary embodiment;

[0042] Figure 6 yes Figure 5 The overall training flow chart involved in the illustrated embodiment;

[0043] Figure 7 yes Figure 5 A flow chart of the basic training rounds involved in the illustrated embodiment;

[0044] Figure 8 yes Figure 5 A flowchart of label resetting involved in the illustrated embodiment;

[0045] Figure 9 yes Figure 5 A schematic diagram of the intensive training process involved in the illustrated embodiment;

[0046] Figure 10 is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0047] Figure 11 is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0048] Figure 12 The figure is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0050] Please refer to Figure 1 , which shows a system structure diagram of an image classification system involved in various embodiments of this application. Figure 1 As shown, the system may include a medical image acquisition device 120 , a terminal 140 , and a server 160 ; optionally, the system may further include a database 180 .

[0051] The medical image acquisition device 120 may be a medical device for acquiring medical images, such as an endoscope, a microscope, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, etc. Accordingly, the medical images may include color images, microscope images, three-dimensional scanning images (such as brain CT images, brain MRI images), etc.

[0052] The medical image acquisition device 120 may include an image output interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, or an Ethernet interface, etc.; or, the above-mentioned image output interface may also be a wireless interface, such as a Wireless Local Area Network (WLAN) interface, a Bluetooth interface, etc.

[0053] Accordingly, depending on the type of the above-mentioned image output interface, the operator can also have multiple ways to export the brain scan image, for example, importing the brain scan image to the terminal 140 via a wired or short-distance wireless method, or importing the brain scan image to the terminal 140 or the server 160 via a local area network or the Internet.

[0054] The terminal 140 may be a terminal device with certain processing capabilities and interface display functions. For example, the terminal 140 may be a mobile phone, a tablet computer, an e-book reader, smart glasses, a laptop computer, a desktop computer, and the like.

[0055] The terminal 140 may include a terminal used by developers and a terminal used by medical personnel.

[0056] When the terminal 140 is implemented as a terminal used by developers, the developers can develop a machine learning model for classifying medical images through the terminal 140 and deploy the machine learning model to the server 160 or a terminal used by medical personnel.

[0057] When the terminal 140 is implemented as a terminal used by medical personnel, an application for classifying medical images and presenting classification results can be installed in the terminal 140. After the terminal 140 obtains the medical images acquired by the medical image acquisition device 120, the terminal 140 obtains the classification results by classifying and identifying the medical images through the above application, and presents the classification results so that doctors can perform pathological diagnosis, surgical navigation and other operations.

[0058] exist Figure 1 In the illustrated system, terminal 140 and medical image acquisition device 120 are physically separate devices. Alternatively, in another possible implementation, when terminal 140 is implemented as a terminal for use by medical personnel, terminal 140 and medical image acquisition device 120 may be integrated into a single device; for example, terminal 140 may be a terminal device with medical image acquisition functionality.

[0059] Among them, server 160 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0060] Among them, the above-mentioned server 160 can be a server that provides background services for the application installed in the terminal 140. The background server can be responsible for version management of the application, background classification processing of medical images obtained by the application and returning processing results, background training of machine learning models developed by developers, etc.

[0061] The database 180 may be a Redis database, or may be another type of database. The database 180 is used to store various types of data.

[0062] Optionally, the terminal 140 is connected to the server 160 via a communication network. Optionally, the medical image acquisition device 120 is connected to the server 160 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0063] Optionally, the system may further include a management device ( Figure 1 (not shown), the management device is connected to the server 160 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0064] Figure 2 This is a flowchart of an image processing method according to an exemplary embodiment. The method can be executed by a computer device, for example, the computer device can be a server, or the computer device can be a terminal, or the computer device can include a server and a terminal, wherein the server can be the above-mentioned Figure 1 In the embodiment shown, the server 160, the terminal may be the above-mentioned Figure 1 In the embodiment shown, the terminal 140 is used by medical personnel. The computer device can be implemented as a classification device for image classification. Figure 2 As shown, the image processing method may include the following steps.

[0065] Step 201: extract at least one first image block from a target medical image.

[0066] The first image block may be an image block of a specified shape extracted from the target medical image, for example, a square image block, a rectangular image block, a circular image block, etc.

[0067] The at least one image block mentioned above can constitute a part of the target medical image after being combined; or, the at least one image block mentioned above can constitute the entire target medical image after being combined.

[0068] In a possible implementation, there may be an overlapping portion between the at least one first image block; or, the at least one first image block does not overlap with each other.

[0069] Step 202 : Mapping image features of at least one first image block to a first classification space to obtain a first classification probability distribution of each of the at least one first image block; the classifications in the first classification probability distribution include at least two valid classifications and at least one invalid classification in the medical field.

[0070] The first classification space may be a label space (also referred to as an output space) corresponding to a plurality of pre-designed classification labels (also referred to as tags in some scenarios) in the field of machine learning, or in other words, the first classification space may be a set of classification labels of at least two valid classifications and at least one invalid classification in the medical field. Mapping the image features of the image block to the first classification space refers to obtaining, based on the image features of the image block, the probability that the image block belongs to the classification corresponding to each classification label (i.e., the first classification probability distribution) to achieve classification and recognition of the image block.

[0071] Among them, the above-mentioned first classification probability distribution includes probability values ​​corresponding to at least two valid classifications, and probability values ​​corresponding to at least one invalid classification, wherein the above-mentioned probability values ​​are used to indicate the probability that the image block for classification identification belongs to the classification to which the probability value belongs.

[0072] In the medical field, the above-mentioned effective classification refers to a classification that helps medical personnel make medical-related decisions; for example, effective classification is used to indicate whether the image to be classified and identified belongs to a normal human tissue image or an abnormal human tissue image (such as whether there is cancer); or, effective classification is used to indicate the degree of abnormality or abnormality level of human tissue in the image to be classified and identified (such as no cancer, early cancer, mid-stage cancer or late cancer), etc.

[0073] In the embodiment of the present application, the above-mentioned invalid classification means that the image block for classification recognition cannot be accurately identified as a valid classification; that is, the result of identifying the image block corresponding to the invalid classification will have a negative impact on the true recognition result of the entire medical image.

[0074] Step 203: extract at least one second image block from the target medical image based on the position of the target image block in the target medical image; the distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one first image block, and the first classification probability distribution of the target image block indicates that the classification of the target image block is a valid classification.

[0075] Step 204 : Map the image features of the at least one second image block to the first classification space to obtain a first classification probability distribution of the at least one second image block.

[0076] Step 205 : Obtain an image classification of the target medical image based on the first classification probability distribution of each of the at least one second image blocks.

[0077] In an embodiment of the present application, if an image block is classified into a valid classification, it means that the image data in the image block can play a positive role in accurately distinguishing the image classification of the entire image. Accordingly, the image block and the image area around the image block are very likely to be the key area for distinguishing the image classification of the entire image. Based on this, after the computer device classifies and identifies the first image block, for the target image block belonging to the valid classification, at least one second image block is further extracted near the target image block as the image block of the key area for image classification in the target medical image. Subsequently, classification and identification are performed based on the at least one second image block, and the identification result of the entire target medical image is determined based on the classification and identification result of the second image block, thereby realizing the automatic extraction of the image blocks that play a key role in image classification in the entire target medical image, and performing image classification based on these image blocks that play a key role in image classification.

[0078] To summarize, the scheme shown in the embodiment of the present application first extracts at least one first image block from the target medical image, and maps the image features of the first image block to a first classification space, and based on the position of the first image block belonging to a valid classification in the medical field, extracts at least one second image block from the target medical image, and determines the image classification to which the target medical image belongs based on the mapping result of the image features of the second image block in the first classification space; through the above-mentioned two-level classification scheme, image blocks at key positions can be extracted from the target medical image, and the image classification of the target medical image can be determined based on the classification results of the image blocks at key positions, which can reduce the influence of images at non-key positions on the classification results of the entire target medical image, thereby improving the accuracy of medical image classification.

[0079] In addition, the scheme shown in the embodiment of the present application can extract partial image blocks from the target medical image for classification. Compared with the recognition scheme that processes the entire image, it can reduce the amount of data processed during the image classification process and improve the efficiency of medical image classification.

[0080] This application Figure 2 The solution in the illustrated embodiment can be implemented based on AI. For example, the above-mentioned step of mapping the image features of the image block to the first classification space can be performed by an image classification model trained based on AI technology.

[0081] AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0082] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0083] This application Figure 2 The solution in the illustrated embodiment can be applied to the Medical Cloud. Among them. The Medical Cloud refers to the use of "cloud computing" to create a medical and health service cloud platform based on new technologies such as cloud computing, mobile technology, multimedia, wireless communication, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the application and combination of cloud computing technology, the Medical Cloud improves the efficiency of medical institutions and facilitates residents' medical treatment. For example, the current hospital appointment registration and medical insurance are the products of the combination of cloud computing and the medical field. The Medical Cloud also has the advantages of data security, information sharing, dynamic expansion, and global layout.

[0084] Figure 3 This is a flowchart of an image processing method according to an exemplary embodiment. The method can be executed by a computer device, for example, the computer device can be a server, or the computer device can be a terminal, or the computer device can include a server and a terminal, wherein the server can be the above-mentioned Figure 1 In the embodiment shown, the server 160, the terminal may be the above-mentioned Figure 1 In the embodiment shown, the terminal 140 is used by developers. The computer device can be implemented as a model training device for training image classification models. Figure 3 As shown, the image processing method may include the following steps.

[0085] Step 301: Process a first sample image block through a first classification model to obtain a second classification probability distribution of the first sample image block; the classification in the second classification probability distribution includes at least two valid classifications in the medical field; the first sample image block is an image block in the first sample medical image.

[0086] In this embodiment of the present application, the first classification model takes an image block as input and outputs a probability distribution of whether the image block belongs to at least two valid classifications. The first classification model can extract image features from the input image block and map the extracted image features to a second classification space to obtain the second classification probability distribution.

[0087] That is, the second classification probability distribution includes probability values ​​corresponding to at least two valid classifications, wherein the probability value is used to indicate the probability that the image block to be classified and identified belongs to the classification to which the probability value belongs.

[0088] In a possible implementation, the first sample image block may be an image block obtained by segmenting the first sample medical image; or, the first sample image block may be an image block directly extracted from the first sample medical image.

[0089] Step 302: Based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs, a classification label is set for the first sample image block; the classification label is used to indicate that the classification of the first sample image block is one of at least two valid classifications and at least one invalid classification.

[0090] In which, when the valid classification of the sample image block is known, the above-mentioned second classification probability distribution can indicate whether the image block input into the first classification model can be accurately classified into the valid classification to which it belongs. For example, when the probability value of the valid classification to which the sample image block actually belongs in the second classification probability distribution of a sample image block satisfies the confidence condition, for example, is greater than the probability threshold (for example, greater than 80%), then it can be considered that the first classification model can accurately classify the sample image block into the valid classification to which it belongs; and if the probability value of the valid classification to which the sample image block actually belongs in the second classification probability distribution of the sample image block does not satisfy the confidence condition, or the maximum probability in the second classification probability distribution of the sample image block satisfies the confidence condition, but the valid classification corresponding to the most probable probability is not the valid classification to which the sample image block actually belongs, then it can be considered that the first classification model cannot accurately classify the sample image block into the valid classification to which it belongs.

[0091] Based on the above reasons, in an embodiment of the present application, after the computer device classifies the first sample image block through the first classification model, it can reset the classification label of each first sample image block based on the obtained second classification probability distribution, and add at least one invalid classification in addition to at least two valid classifications to indicate whether the corresponding sample image block can be accurately classified by the machine learning model, thereby automatically labeling the sample image block as a valid classification that can be accurately identified, or an invalid classification that cannot be accurately identified.

[0092] For example, for a sample image block, if the valid classification indicated by the second classification probability distribution of the sample image block is the correct classification, the actual valid classification of the sample image block is set as the classification label of the sample image block; if the probability that meets the confidence condition in the second classification probability distribution of the sample image block corresponds to an incorrect valid classification, or if all the probabilities in the second classification probability distribution of the sample image block do not meet the confidence condition, the actual valid classification of the sample image block is set to an invalid classification.

[0093] Step 303 : Process the first sample image block using the second classification model to obtain a first classification probability distribution of the first sample image block; the categories in the first classification probability distribution include at least two of the above-mentioned valid categories and at least one invalid category.

[0094] The difference between the second classification model and the first classification model is that the probability distribution output by the second classification model includes, in addition to the probabilities indicating that the input sample image block belongs to each valid category, the probabilities indicating that the input sample image block belongs to each invalid category. In other words, the second classification model can extract image features from the input image block and map the extracted image features to the first classification space to obtain the first classification probability distribution.

[0095] Step 304 : Based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block, update the parameters of the second classification model to obtain an image classification model.

[0096] Image classification models are used in the above Figure 2 In the illustrated embodiment, a step of mapping image features of an image block to a first classification space is performed.

[0097] In the embodiment of the present application, the image classification model obtained by training the second classification model with the first sample image block with the reset classification label can be used to identify which image blocks in the input image blocks can be accurately classified and which image blocks cannot be accurately classified. Figure 2 The solution in the illustrated embodiment can exclude images in areas of the entire image that have a negative impact on image classification as much as possible through a two-step classification method, and select image blocks in key areas of the entire image that have a positive impact on image classification for classification.

[0098] In summary, the scheme shown in the embodiment of the present application first uses a first classification model that outputs at least two valid classifications to classify the sample image blocks, resets the classification labels of the sample image blocks according to the probability distribution of the first classification model, and expands the classification labels of the sample image blocks to at least two valid classifications and at least one invalid classification; and uses the sample image blocks after the reset classification labels to train to obtain an image classification model. Subsequently, in the process of classifying the target medical image, at least one first image block is first extracted from the target medical image through the image classification model, and the image features of the first image block are mapped to the first classification space. Based on the position of the first image block belonging to the valid classification, at least one second image block is extracted from the target medical image, and then the second image block is classified through the image classification model, and the image category to which the target medical image belongs is determined based on the mapping result of the image features of the second image block in the first classification space. Through the above-mentioned two-level classification scheme, image blocks at key positions can be extracted from the target medical image, and the image classification of the target medical image can be determined based on the classification results of the image blocks at key positions. This can reduce the influence of images at non-key positions on the classification results of the entire target medical image, thereby improving the accuracy of medical image classification.

[0099] In addition, the solution shown in the embodiment of the present application resets the classification labels of the sample image blocks through the classification probability distribution output by the first classification model, thereby realizing automatic labeling of the classification labels, improving the efficiency of sample generation, and further improving the efficiency of model training.

[0100] In addition, the scheme shown in the embodiment of the present application can extract partial image blocks from the target medical image for identification. Compared with the identification scheme that processes the entire image, it can reduce the amount of data processed during the image classification process and improve the efficiency of medical image classification.

[0101] The solution shown in the embodiments of the present application can be applied to various medical image classification scenarios. For example, the above-mentioned medical image classification scenarios include but are not limited to cervical image classification scenarios. For example, it can also be applied to fundus image classification scenarios, gastroscopy image / colonoscopy image classification scenarios, three-dimensional medical image classification scenarios, etc.

[0102] Taking the classification scenario of cervical images as an example, first, based on Figure 3In the solution shown, developers can collect several sample cervical images, extract sample image blocks from the sample cervical images, and label the sample image blocks with valid classification labels (such as normal labels or pathology labels). Then, they can classify the sample image blocks using the trained first classification model, outputting the probability distribution of the sample image blocks belonging to various valid classifications. Based on this probability distribution, they can reset the classification labels of the sample image blocks, expanding the labels of the sample image blocks to labels of valid classifications and labels of invalid classifications. Then, they can train the second classification model using the sample image blocks with reset classification labels to obtain an image classification model.

[0103] After the above image classification model training is completed, it can be deployed in the artificial intelligence electronic colposcopy assisted diagnosis system. Figure 2 In the illustrated scheme, after deploying the image classification model, when medical personnel perform diagnostic processing based on cervical images, the computer device in the artificial intelligence electronic colposcopy-assisted diagnosis system can extract each first image block from the cervical image and process each first image block through the image classification model. Based on the probability distribution of the classification output, the computer device selects a target image block that can be accurately classified from the first image block. Because the location of the target image block in the cervical image is likely to be a key location for accurate classification of the cervical image, the computer device further extracts a second image block corresponding to the key location near the first image block and classifies the second image block through the image classification model. Based on the classification result of the second image block, the classification result of the entire cervical image is determined. Subsequently, medical personnel can combine the classification results of the entire cervical image to make diagnostic decisions, such as determining the location and grade of suspicious lesions and providing real-time guidance on whether biopsy sampling is necessary and the sampling site.

[0104] Figure 4 This is a flowchart of an image processing method according to an exemplary embodiment. The method can be executed by a computer device, for example, the computer device can be a server, or the computer device can be a terminal, or the computer device can include a server and a terminal, wherein the server can be the above-mentioned Figure 1 In the embodiment shown, the server 160, the terminal may be the above-mentioned Figure 1 The terminal 140 in the embodiment shown. Figure 4 As shown, the image processing method may include the following steps.

[0105] Step 401: Process the second sample image block through the first classification model to obtain a second classification probability distribution of the second sample image block; the classifications in the second classification probability distribution include at least two valid classifications in the medical field; the first sample image block is an image block in the second sample medical image.

[0106] For example, using the solution described in the embodiments of this application as an example for a cervical image classification scenario, the second sample medical image can be a sample cervical image pre-labeled with the image classification. In other medical scenarios, the second sample medical image can be a medical image corresponding to the medical scenario. For example, in a fundus image classification scenario, the medical image can be a sample fundus image pre-labeled with the image classification.

[0107] Among them, the above-mentioned effective classification can be a classification that is effective for subsequent auxiliary decision-making in medical scenarios. For example, taking cervical images as an example, the effective classification can include normal classification and lesion classification (such as cervical cancer), or, the effective classification can include normal classification and at least two different lesion classifications (such as cervical cancer classifications with different degrees of severity, such as early cervical cancer classification, late cervical cancer classification, etc.). Subsequent medical personnel can make diagnostic decisions based on the effective classification.

[0108] In one possible implementation, developers can pre-collect second sample medical images that have been labeled with image classifications, and input the labeled second sample medical images into a computer device used for model training (i.e., a model training device). The model training device segments the second sample medical images to obtain a number of second sample image blocks of specified sizes. The initial classification label of each second sample image block can be inherited from the classification label of the second sample medical image. Optionally, each second sample image block can also be re-labeled with an initial classification label by the developer or medical staff. The second sample medical image, or the initial classification label of the second sample image block, is one of the valid classifications.

[0109] Afterwards, the model training device processes the second sample image blocks using the first classification model and outputs a second classification probability for each second sample image block, that is, outputs the probability that each second sample image block belongs to various valid classifications.

[0110] Step 402 : updating parameters of the first classification model based on the second classification probability distribution of the second sample image block and the valid classification to which the second sample image block belongs.

[0111] The valid category to which the second sample image block belongs is the valid category corresponding to the initial category label of the second sample image block.

[0112] In this embodiment of the present application, the model training device can calculate a loss function value based on the second classification probability distribution of the second sample image block and the initial classification label of the second sample image block, and update the parameters of the first classification model using the calculated loss function value. After the first classification model training is completed, the subsequent step 403 can be executed.

[0113] Step 403 : Process the first sample image block through the first classification model to obtain a second classification probability distribution of the first sample image block; the first sample image block is an image block in the first sample medical image.

[0114] The first sample medical image and the second sample medical image are medical images corresponding to the same medical scene. For example, when the first sample medical image is a cervical image, the second sample medical image is also a cervical image.

[0115] Optionally, the first sample medical image and the second sample medical image may be the same medical image or different medical images. Correspondingly, the first sample image block and the second sample image block may be the same sample image block or different sample image blocks.

[0116] Step 404: Based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs, a classification label is set for the first sample image block; the classification label is used to indicate that the classification of the first sample image block is one of at least two of the above-mentioned valid classifications and at least one invalid classification.

[0117] In the embodiment of the present application, the at least one invalid classification includes at least one of a meaningless classification and an interference classification;

[0118] Among them, meaningless classification can be used to indicate that the valid classification to which it belongs cannot be distinguished through the input image;

[0119] The interference classification is used to indicate that the input image is incorrectly distinguished from the valid classification.

[0120] In a possible implementation, setting a classification label for the first sample image block based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs includes:

[0121] In response to a maximum probability in the second classification probability distribution of the first sample image block being greater than a probability threshold, and the classification to which the maximum probability belongs matching the valid classification to which the first sample image block belongs, setting the classification label of the first sample image block to the valid classification to which the first sample image block belongs;

[0122] In response to a maximum probability in the second classification probability distribution of the first sample image block being greater than a probability threshold, and the classification to which the maximum probability belongs not matching the valid classification to which the first sample image block belongs, setting the classification label of the first sample image block to an interference classification;

[0123] In response to the maximum probability in the second classification probability distribution of the first sample image block being not greater than the probability threshold, the classification label of the first sample image block is set to a meaningless classification.

[0124] In an exemplary scheme of an embodiment of the present application, if the maximum probability in the second classification probability distribution of the first sample image block is greater than the probability threshold, the maximum probability can be considered to be credible. At this time, if the classification corresponding to the maximum probability is the valid classification corresponding to the initial classification label of the first sample image block, it is considered that the first classification model can accurately identify the valid classification to which the first sample image block belongs. At this time, the classification label of the first sample image block can be kept unchanged (that is, kept as the initial classification label); and if the classification corresponding to the maximum probability is not the valid classification corresponding to the initial classification label of the first sample image block, it is considered that the first classification model has incorrectly identified the valid classification to which the first sample image block belongs. At this time, it can be considered that the first sample image block will cause negative interference to the classification recognition of the entire image, and the model training device can reset the classification label of the first sample image block to an interference classification.

[0125] If the maximum probability in the second classification probability distribution of the first sample image block is not greater than the probability threshold, the maximum probability can be considered unreliable. At this time, it is considered that the first classification model cannot distinguish the valid classification to which the first sample image block belongs, which means that the first sample image block has no effect on the classification recognition of the entire image (whether positive or negative). At this time, the classification label of the first sample image block can be reset to a meaningless classification.

[0126] Step 405 : Process the first sample image block using a second classification model to obtain a first classification probability distribution of the first sample image block; the classifications in the classification probability distribution include at least two valid classifications and at least one invalid classification.

[0127] In the embodiment of the present application, the probability distribution output by the second classification model includes, in addition to the probabilities corresponding to valid classifications, the probabilities of invalid classifications.

[0128] In a possible implementation, before the first sample image block is processed by the second classification model, the method further includes:

[0129] A second classification model is constructed based on the feature extraction network; the feature extraction network is the model part of the first classification model except the output layer.

[0130] In an embodiment of the present application, the first classification model includes a feature extraction network and a first output layer, wherein the first output layer is used to extract image features from an input sample image block according to the feature extraction network, and output a first classification probability distribution of the sample image block. In order to increase the training speed of the second classification model and improve training efficiency, the solution shown in the embodiment of the present application can construct and train the second classification model based on the feature extraction network of the first classification model. Since the feature extraction network in the first classification model has been pre-trained, in the subsequent training of the second classification model, only fine-tuning of the parameters of the second classification model is required to achieve rapid convergence.

[0131] Optionally, when constructing a second classification model based on the feature extraction network, a second output layer can be added on the basis of the feature extraction network, wherein the second output layer is used to output the second classification probability distribution of the sample image block based on the image features obtained by feature extraction of the input sample image block according to the feature extraction network.

[0132] In another possible implementation, developers can also independently build a second classification model outside the first classification model; or, after building the second classification model based on the feature extraction network in the first classification model, reset the parameters of the second classification model for retraining.

[0133] Step 406 : Based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block, update the parameters of the second classification model to obtain an image classification model.

[0134] In an embodiment of the present application, the model training device can calculate a loss function value based on the first classification probability distribution of the first sample image block and the reset classification label of the first sample image block, and update the parameters of the second classification model using the calculated loss function value. After the second classification model training is completed, an image classification model can be obtained.

[0135] Image classification models are used in Figure 2 In the embodiment shown, the step of mapping the image features of the image block to the first classification space is performed. This process may refer to the subsequent steps.

[0136] Step 407: extract at least one first image block from the target medical image.

[0137] Among them, the above-mentioned target medical image is a medical image corresponding to the same medical scene as the above-mentioned first sample medical image and the second sample medical image. For example, when the above-mentioned first sample medical image and the second sample medical image are cervical images with labeled image classification, the above-mentioned target medical image can be an unlabeled cervical image.

[0138] In a possible implementation, the step of extracting at least one first image block from the target medical image may include:

[0139] randomly extracting at least one first image block from a target medical image;

[0140] Alternatively, at least one first image block is obtained by sampling from the target medical image according to a sampling distance interval.

[0141] In an embodiment of the present application, a computer device (e.g., a classification device) during the model application process may randomly extract a portion of image blocks from the target medical image as the first image block. For example, the classification device may divide the target medical image into multiple image blocks of equal size (e.g., 9×9 image blocks) and randomly extract 20 image blocks from the image blocks as the first image block.

[0142] Alternatively, the classification device may uniformly collect a number of image blocks from the target medical image at a certain sampling distance interval as the first image block. For example, the classification device divides the target medical image into multiple large image blocks of equal size (e.g., 3×3 large image blocks), extracts a small image block from the center of each large image block, and uses the extracted small image block as the first image block.

[0143] In another possible implementation, the classification device may also randomly acquire the first image block in combination with a sampling distance interval. For example, the classification device divides the target medical image into multiple large image blocks of equal size (e.g., 3×3 large image blocks), then divides each large image block into multiple small image blocks (e.g., each large image block into 3×3 small image blocks), and then randomly selects one or more small image blocks from each large image block as the first image block.

[0144] Step 408 : Map the image features of the at least one first image block to a first classification space to obtain a first classification probability distribution of the at least one first image block.

[0145] The classification device can classify and identify the at least one first image block using an image classification model. Specifically, the classification device can extract image features of the first image block using the image classification model and map the extracted image features to a first classification space to obtain a first classification probability distribution for each of the at least one first image block. The execution process of this step is similar to the process of classifying the first sample image block using the second classification model and will not be further described here.

[0146] Step 409: extract at least one second image block from the target medical image based on the position of the target image block in the target medical image; the distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one first image block.

[0147] In an embodiment of the present application, the classification device may take the target image block as the center of a key position in the target medical image, and select one or more image blocks within a certain range around the center point of the target image block from the target medical image as second image blocks, and the distance between each second image block and the target image block (for example, the distance may be the pixel distance or length distance between the centers of the two image blocks) is less than a distance threshold.

[0148] For example, the classification device can start from the center point of the target image block, select candidate points whose distance from the center point of the target image block is less than a distance threshold in one or more random directions (or one or more specified directions), and use the candidate points as the center of the second image block to extract one or more second image blocks of a specified size.

[0149] The above distance threshold can be set or adjusted by developers or medical personnel in combination with application scenarios.

[0150] In a possible implementation manner, the distance threshold is less than or equal to the side length of the first image block.

[0151] Step 410 : Map the image features of at least one second image block to the first classification space to obtain a first classification probability distribution of each of the at least one second image block.

[0152] The execution process of this step is similar to the above step 408 and will not be repeated here.

[0153] Step 411 : Obtain an image classification of the target medical image based on the first classification probability distribution of each of the at least one second image blocks.

[0154] In one possible implementation, obtaining an image classification of a target medical image based on the first classification probability distribution of each of the at least one second image blocks includes:

[0155] In response to the number of the at least one second image block being n, where n is an integer greater than or equal to 2, averaging the first classification probability distributions of the n second image blocks to obtain a predicted average probability distribution;

[0156] Based on the predicted mean probability distribution, an image classification of the target medical image is obtained.

[0157] In this embodiment of the present application, if there are multiple second image blocks, the classification device may average the first classification probability distributions of the multiple second image blocks. In other words, for the first classification probability distributions of the multiple second image blocks, the probability values ​​corresponding to the same classification in these first classification probability distributions are averaged to obtain a predicted average probability distribution.

[0158] In one possible implementation, obtaining an image classification of a target medical image based on the first classification probability distribution of each of the at least one second image blocks includes:

[0159] averaging the first classification probability distribution of each of the at least one second image block and the first classification probability distribution of the target image block to obtain a predicted average probability distribution;

[0160] Based on the predicted mean probability distribution, an image classification of the target medical image is obtained.

[0161] In an embodiment of the present application, the classification device may also combine the first classification probability distribution of the second image block and the first classification probability distribution of the target image block to further improve the classification accuracy of the entire image.

[0162] In one possible implementation, the step of obtaining the image classification of the target medical image based on the predicted average probability distribution may include:

[0163] In response to a maximum probability in the predicted average probability distribution being greater than a probability threshold, a classification corresponding to the maximum probability is obtained as an image classification of the target medical image.

[0164] In the embodiment of the present application, the classification device may detect whether the maximum probability of the predicted average probability distribution is greater than a probability threshold. If so, the predicted average probability distribution is deemed credible, and the classification corresponding to the maximum probability is used as the image classification of the target medical image. Alternatively, if the maximum probability is not greater than the probability threshold, the image classification model is deemed to have failed to distinguish the image classification of the target medical image.

[0165] In summary, the scheme shown in the embodiment of the present application first uses a first classification model that outputs at least two valid classifications to classify the sample image blocks, resets the classification labels of the sample image blocks according to the probability distribution of the first classification model, and expands the classification labels of the sample image blocks to at least two valid classifications and at least one invalid classification; and uses the sample image blocks after the reset classification labels to train to obtain an image classification model. Subsequently, in the process of classifying the target medical image, at least one first image block is first extracted from the target medical image through the image classification model, and the image features of the first image block are mapped to the first classification space. Based on the position of the first image block belonging to the valid classification, at least one second image block is extracted from the target medical image, and then the second image block is classified and identified through the image classification model, and the image category to which the target medical image belongs is determined based on the mapping result of the image features of the second image block in the first classification space. Through the above-mentioned two-level classification scheme, image blocks at key positions can be extracted from the target medical image, and the image classification of the target medical image can be determined based on the classification results of the image blocks at key positions. This can reduce the influence of images at non-key positions on the classification results of the entire target medical image, thereby improving the accuracy of medical image classification.

[0166] In addition, the solution shown in the embodiment of the present application resets the classification labels of the sample image blocks through the classification probability distribution output by the first classification model, thereby realizing automatic labeling of the classification labels, improving the efficiency of sample generation, and further improving the efficiency of model training.

[0167] In addition, the scheme shown in the embodiment of the present application can extract partial image blocks from the target medical image for identification. Compared with the identification scheme that processes the entire image, it can reduce the amount of data processed during the image classification process and improve the efficiency of medical image classification.

[0168] Please refer to Figure 5 , which is a diagram showing a training and testing framework for an image classification model for medical images according to an exemplary embodiment. Figure 5 As shown, taking the classification scenario of cervical images as an example, the training and testing process of the image classification model can be as follows:

[0169] First, before model training, the developer prepares a sample cervical image 51a and performs effective classification annotation and segmentation on the sample cervical image 51a to obtain a number of sample cervical image blocks 51c corresponding to initial classification labels 51b.

[0170] For example, please refer to Figure 6 , which shows the overall training flow chart involved in the embodiment of this application. Figure 6As shown, in the solution shown in the embodiments of the present application, the model training process is divided into a data collection process 61, a basic training round 62, and an enhanced training round 63.

[0171] In the model training process, first, the basic model 52 including the main network framework 52a is trained using the sample cervical image blocks 51c and the initial classification labels 51b.

[0172] For example, please refer to Figure 7 , which shows the flowchart of the basic training round involved in the embodiments of the present application. As Figure 7 shown, taking the effective classification including Class A and Class B as an example, in the basic training round, each sample cervical image block 51c inherits the label of the effective classification of the sample cervical image 51a, and the model training device sends each sample cervical image block into the basic model 52 including the main network framework 52a for training.

[0173] After the basic model 52 is trained, the model training device processes the sample cervical image blocks through the basic model 52 to obtain the effective classification probability distribution 53 of the sample cervical image blocks, and updates the classification label 51b of the sample cervical image blocks 51c through the effective classification probability distribution 53 to obtain the updated classification label 51d. In addition to the effective classification, the classification in the updated classification label 51d also includes invalid classifications.

[0174] Among them, the enhanced training round includes two steps: re-labeling the image blocks and fine-tuning training. For example, please refer to Figure 8 , which shows the flowchart of the label reset involved in the embodiments of the present application. As Figure 8 shown, the model training device sends the sample cervical image blocks into the trained basic model 52 for testing to obtain the probability distribution T of each sample cervical image block. The closer T is to 0, the more the sample cervical image block tends to Class A, and the closer T is to , the more the sample cervical image block tends to Class B. According to the probability values, the sample cervical image blocks can be divided into three categories:

[0175] A) The probability value is in the range t1 < T < t2. This image block is considered not suitable for clearly distinguishing the actual classes A / B and is identified as a meaningless area, and a new label Y is assigned.

[0176] Among them, the values of t1 and t2 can be set by developers or medical staff in combination with the application scenario. For example, the value of t1 can be 0.45, and the value of t2 can be 0.55.

[0177] B) The probability value T < t1 and the gold standard of the sample cervical image patch (i.e., the initial classification label) is class A, or T > t2 and the gold standard of the sample cervical image patch is class B. The classification of this sample cervical image patch is correct and is considered to cover the key information available for classification, and the label value remains unchanged.

[0178] C) The probability value T < t1 and the gold standard of the sample cervical image patch is class B, or T > t2 and the gold standard of the sample cervical image patch is class A. The classification of this image patch is incorrect and is identified as an interfering image patch, and a new label Z is assigned.

[0179] Then, the model training device further trains the image classification model 54 including the main network framework 52a through the sample cervical image patch 51c and the updated classification label 51d to obtain the trained image classification model 54. Among them, the initial main network framework 52a in the image classification model 54 is the main network framework 52a in the trained basic model 52. During the training process of the image classification model 54, the parameters of the main network framework 52a will be continuously adjusted and updated. Different from the basic model 52, the probability distribution output by the image classification model 54 includes both valid classification and invalid classification.

[0180] Please refer to Figure 9 , which shows a schematic diagram of the reinforcement training process involved in the embodiments of the present application. As Figure 9 shown, after each sample cervical image patch is labeled with a new label according to the above Figure 8 steps, the image classification model 54 constructed based on the main network framework 52a in the basic model is fine-tuned to obtain the finally strengthened image classification model 54.

[0181] During the training process of the image classification model 54, the model training device can test the image classification model 54, that is, randomly sample and extract image patches from the test images to obtain a series of sampled image patches, and then send all the sampled image patches into the image classification model 54 for testing to obtain the classification probability of each sampled image patch. Determine the classification according to the classification probability value. If the classification is Y (meaningless image patch) or Z (interfering image patch), discard it. For other classified image patches, resample near them to obtain k image patches; send the k image patches into the image classification model 54 for retesting to obtain k prediction result probabilities, and average the k probabilities to obtain the final test result.

[0182] The classification process of the above image classification model 54 after deployment is similar to the above test process and will not be elaborated here.

[0183] The solution shown in the embodiment of the present application proposes a solution for automatically identifying local key information and using it for enhanced training. In this solution, effective classification is taken as an example of binary classification. Optionally, this solution can also be extended to more effective classification scenarios.

[0184] The solutions shown in the embodiments of the present application may include but are not limited to the following four advantages:

[0185] 1) The training is divided into two rounds. Based on the results of the first round, the image interference / meaningless areas are reassigned and sent to the second round of training to strengthen key local information;

[0186] 2) During testing, the model can automatically extract multiple key local information for analysis and output the final judgment result;

[0187] 3) Due to the whole-image disassembly prediction, only local key image blocks need to be used, which reduces the amount of data to be analyzed and improves the algorithm efficiency;

[0188] 4) The combination of random sampling and key sampling testing can effectively reduce accidental errors in predictions.

[0189] The solutions shown in the above embodiments of the present application can be implemented or executed in conjunction with blockchain. For example, some or all of the steps in the above embodiments can be executed in a blockchain system; or, the data required or generated for executing the steps in the above embodiments can be stored in a blockchain system; for example, the training samples used for the above model training and the model input data such as medical images during the model application process can be obtained from the blockchain system by a computer device; for another example, the model parameters obtained after the above model training can be stored in the blockchain system.

[0190] Figure 10 This is a structural block diagram of an image processing device according to an exemplary embodiment. Figure 2 or Figure 4 All or part of the steps in the method provided in the illustrated embodiment, the image processing device includes:

[0191] A first image block extraction module 1001 is configured to extract at least one first image block from a target medical image;

[0192] A first classification module 1002 is configured to map image features of at least one of the first image blocks to a first classification space to obtain a first classification probability distribution for each of the at least one first image block; the classifications in the first classification probability distribution include at least two valid classifications and at least one invalid classification in the medical field;

[0193] A second image block extraction module 1003 is configured to extract at least one second image block from the target medical image based on a position of the target image block in the target medical image; the distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one of the first image blocks, and the first classification probability distribution of the target image block indicates that the classification of the target image block is the valid classification;

[0194] A second classification module 1004 is configured to map the image features of at least one of the second image blocks to the first classification space to obtain a first classification probability distribution of each of the at least one of the second image blocks;

[0195] The third classification module 1005 is configured to obtain an image classification of the target medical image based on the first classification probability distribution of at least one of the second image blocks.

[0196] In a possible implementation, at least one of the invalid classifications includes at least one of a meaningless classification and an interference classification;

[0197] The meaningless classification is used to indicate that the valid classification to which the input image belongs cannot be distinguished;

[0198] The interference category is used to indicate the valid category to which the input image is incorrectly distinguished.

[0199] In a possible implementation, the third classification module 1005 is configured to:

[0200] In response to the number of at least one second image block being n, where n is an integer greater than or equal to 2, averaging the first classification probability distributions of the n second image blocks to obtain a predicted average probability distribution;

[0201] Based on the predicted average probability distribution, an image classification of the target medical image is obtained.

[0202] In a possible implementation, the third classification module 1005 is configured to:

[0203] averaging the first classification probability distribution of at least one of the second image blocks and the first classification probability distribution of the target image block to obtain a predicted average probability distribution;

[0204] Based on the predicted average probability distribution, an image classification of the target medical image is obtained.

[0205] In a possible implementation, the third classification module 1005 is configured to:

[0206] In response to a maximum probability in the predicted average probability distribution being greater than a probability threshold, a classification corresponding to the maximum probability is obtained as the image classification of the target medical image.

[0207] In a possible implementation, the first image block extraction module 1001 is configured to:

[0208] randomly extracting at least one first image block from the target medical image;

[0209] Alternatively, at least one first image block is obtained by sampling from the target medical image according to a sampling distance interval.

[0210] In a possible implementation manner, the distance threshold is less than or equal to the side length of the first image block.

[0211] In a possible implementation, the target medical image is a cervical image.

[0212] To summarize, the scheme shown in the embodiment of the present application first extracts at least one first image block from the target medical image, maps the image features of the first image block to a first classification space, and based on the position of the first image block belonging to a valid classification in the medical field, extracts at least one second image block from the target medical image, and determines the image classification to which the target medical image belongs based on the mapping result of the image features of the second image block in the first classification space; through the above-mentioned two-level classification scheme, image blocks at key positions can be extracted from the target medical image, and the image classification of the target medical image can be determined based on the classification results of the image blocks at key positions, which can reduce the influence of images at non-key positions on the classification results of the entire target medical image, thereby improving the accuracy of medical image classification.

[0213] In addition, the scheme shown in the embodiment of the present application can extract partial image blocks from the target medical image for identification. Compared with the identification scheme that processes the entire image, it can reduce the amount of data processed during the image classification process and improve the efficiency of medical image classification.

[0214] Figure 11 This is a structural block diagram of an image processing device according to an exemplary embodiment. Figure 3 or Figure 4 All or part of the steps in the method provided in the illustrated embodiment, the image processing device includes:

[0215] A first processing module 1101 is configured to process a first sample image block using a first classification model to obtain a second classification probability distribution for the first sample image block, wherein the classifications in the second classification probability distribution include at least two valid classifications in the medical field; and the first sample image block is an image block in a first sample medical image.

[0216] a label setting module 1102, configured to set a classification label for the first sample image block based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs; the classification label being used to indicate whether the first sample image block is classified as one of at least two of the valid classifications and at least one invalid classification;

[0217] A second processing module 1103 is configured to process the first sample image block using a second classification model to obtain a first classification probability distribution of the first sample image block, wherein the categories in the first classification probability distribution include at least two valid categories and at least one invalid category;

[0218] A first parameter updating module 1104 is configured to update parameters of the second classification model based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block to obtain an image classification model;

[0219] The image classification model is used to perform Figure 2 or Figure 4 In the illustrated embodiment, the image features of the image block are mapped to the first classification space.

[0220] In a possible implementation, at least one of the invalid classifications includes at least one of a meaningless classification and an interference classification;

[0221] The meaningless classification is used to indicate that the valid classification to which the input image belongs cannot be distinguished;

[0222] The interference category is used to indicate the valid category to which the input image is incorrectly distinguished.

[0223] In a possible implementation, the label setting module 1102 is used to:

[0224] In response to a maximum probability in the second classification probability distribution of the first sample image block being greater than a probability threshold, and the classification to which the maximum probability belongs matching the valid classification to which the first sample image block belongs, setting the classification label of the first sample image block to the valid classification to which the first sample image block belongs;

[0225] In response to a maximum probability in the second classification probability distribution of the first sample image block being greater than a probability threshold, and the classification to which the maximum probability belongs not matching the valid classification to which the first sample image block belongs, setting the classification label of the first sample image block to the interference classification;

[0226] In response to the maximum probability in the second classification probability distribution of the first sample image block being not greater than the probability threshold, the classification label of the first sample image block is set to the meaningless classification.

[0227] In a possible implementation, the apparatus further includes:

[0228] The second parameter updating module is used to, before the first processing module 1101 processes the first sample image block through the first classification model,

[0229] Processing the second sample image block using the first classification model to obtain a second classification probability distribution of the second sample image block; the first sample image block is an image block in the second sample medical image;

[0230] Based on the second classification probability distribution of the second sample image block and the valid classification to which the second sample image block belongs, parameters of the first classification model are updated.

[0231] In a possible implementation, the apparatus further includes:

[0232] The model construction module is used to construct the second classification model based on the feature extraction network before the second processing module 1103 processes the first sample image block through the second classification model; the feature extraction network is the model part of the first classification model except the output layer.

[0233] In summary, the scheme shown in the embodiment of the present application first uses a first classification model that outputs at least two valid classifications to classify the sample image blocks, resets the classification labels of the sample image blocks according to the probability distribution of the first classification model, and expands the classification labels of the sample image blocks to at least two valid classifications and at least one invalid classification; and uses the sample image blocks after the reset classification labels to train to obtain an image classification model. Subsequently, in the process of classifying the target medical image, at least one first image block is first extracted from the target medical image through the image classification model, and the image features of the first image block are mapped to the first classification space. Based on the position of the first image block belonging to the valid classification, at least one second image block is extracted from the target medical image, and then the second image block is classified through the image classification model, and the image category to which the target medical image belongs is determined based on the mapping result of the image features of the second image block in the first classification space. Through the above-mentioned two-level classification scheme, image blocks at key positions can be extracted from the target medical image, and the image classification of the target medical image can be determined based on the classification results of the image blocks at key positions. This can reduce the influence of images at non-key positions on the classification results of the entire target medical image, thereby improving the accuracy of medical image classification.

[0234] In addition, the solution shown in the embodiment of the present application resets the classification labels of the sample image blocks through the classification probability distribution output by the first classification model, thereby realizing automatic labeling of the classification labels, improving the efficiency of sample generation, and further improving the efficiency of model training.

[0235] In addition, the scheme shown in the embodiment of the present application can extract partial image blocks from the target medical image for identification. Compared with the identification scheme that processes the entire image, it can reduce the amount of data processed during the image classification process and improve the efficiency of medical image classification.

[0236] Figure 121 is a schematic diagram of the structure of a computer device according to an exemplary embodiment. The computer device can be implemented as the computer device used to train the first image recognition model in the above-mentioned various method embodiments, or can be implemented as the computer device used to perform brain midline recognition using the second image recognition model in the above-mentioned various method embodiments. The computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a basic input / output system 1206 that helps transmit information between various components within the computer, and a large-capacity storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0237] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable media provide non-volatile storage for the computer device 1200. In other words, the mass storage device 1207 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0238] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the aforementioned types. The above-mentioned system memory 1204 and mass storage device 1207 may be collectively referred to as memory.

[0239] The computer device 1200 can be connected to the Internet or other network devices through a network interface unit 1211 connected to the system bus 1205 .

[0240] The memory also includes one or more programs, which are stored in the memory. The one or more programs contain computer instructions. The central processing unit 1201 implements the computer instructions in the one or more programs by executing them. Figure 2 、 Figure 3 or Figure 4 All or part of the steps of any of the methods shown.

[0241] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including a computer program (computer instructions), which can be executed by a processor of a computer device to perform the methods described in various embodiments of the present application. For example, the non-transitory computer-readable storage medium can be a read-only memory, a random access memory, a read-only optical disc, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0242] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various embodiments above.

[0243] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0244] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: extracting at least one first image block from a target medical image; Mapping image features of at least one of the first image blocks to a first classification space to obtain a first classification probability distribution for each of the at least one first image block; the classifications in the first classification probability distribution include at least two valid classifications and at least one invalid classification in the medical field; the first classification probability distribution includes probability values ​​corresponding to the at least two valid classifications, and a probability value corresponding to the at least one invalid classification, the probability values ​​being used to indicate the probability that the image block being classified and identified belongs to the classification to which the probability values ​​belong; the valid classification refers to a classification that is helpful in making medical-related decisions; the invalid classification refers to an image block being classified and identified that cannot be accurately identified as a valid classification; extracting at least one second image block from the target medical image based on a position of the target image block in the target medical image; a distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one of the first image blocks, and a first classification probability distribution of the target image block indicates that the classification of the target image block is the valid classification; Mapping the image feature of at least one of the second image blocks to the first classification space to obtain a first classification probability distribution of the at least one of the second image blocks; An image classification of the target medical image is obtained based on the first classification probability distribution of each of at least one of the second image blocks.

2. The method according to claim 1, characterized in that At least one of the invalid classifications includes at least one of a meaningless classification and an interference classification; The meaningless classification is used to indicate that the valid classification to which the input image belongs cannot be distinguished; The interference category is used to indicate the valid category to which the input image is incorrectly distinguished.

3. The method according to claim 1, characterized in that The obtaining of the image classification of the target medical image based on the first classification probability distribution of at least one second image block comprises: In response to the number of at least one second image block being n, where n is an integer greater than or equal to 2, averaging the first classification probability distributions of the n second image blocks to obtain a predicted average probability distribution; Based on the predicted average probability distribution, an image classification of the target medical image is obtained.

4. The method according to claim 1, wherein The obtaining of the image classification of the target medical image based on the first classification probability distribution of at least one second image block comprises: averaging the first classification probability distribution of at least one of the second image blocks and the first classification probability distribution of the target image block to obtain a predicted average probability distribution; Based on the predicted average probability distribution, an image classification of the target medical image is obtained.

5. The method according to claim 3 or 4, characterized in that The obtaining of the image classification of the target medical image based on the predicted average probability distribution includes: In response to a maximum probability in the predicted average probability distribution being greater than a probability threshold, a classification corresponding to the maximum probability is obtained as the image classification of the target medical image.

6. The method according to claim 1, characterized in that The step of extracting at least one first image block from the target medical image comprises: randomly extracting at least one first image block from the target medical image; or, At least one first image block is obtained by sampling from the target medical image according to a sampling distance interval.

7. The method according to claim 1, characterized in that The distance threshold is less than or equal to the side length of the first image block.

8. The method according to claim 1, characterized in that The target medical image is a cervical image.

9. An image processing method, characterized in that: The method comprises: Processing the first sample image block using a first classification model to obtain a second classification probability distribution for the first sample image block; the classifications in the second classification probability distribution include at least two valid classifications in the medical field; and the first sample image block is an image block in a first sample medical image; Based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs, setting a classification label for the first sample image block; the classification label is used to indicate that the first sample image block is classified as one of at least two valid classifications and at least one invalid classification; the second classification probability distribution includes probability values ​​corresponding to the at least two valid classifications, each of which indicates the probability that the image block to be classified and identified belongs to the classification to which the probability value belongs; the valid classification is a classification that is helpful for making medical-related decisions; the invalid classification is an image block to be classified and identified that cannot be accurately identified as a valid classification; Processing the first sample image block through a second classification model to obtain a first classification probability distribution of the first sample image block; the classifications in the first classification probability distribution include at least two of the valid classifications and at least one of the invalid classifications; Based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block, updating the parameters of the second classification model to obtain an image classification model; The image classification model is used to perform the step of mapping the image features of the image block to the first classification space in the method according to any one of claims 1 to 8.

10. The method according to claim 9, characterized in that At least one of the invalid classifications includes at least one of a meaningless classification and an interference classification; The meaningless classification is used to indicate that the valid classification to which the input image belongs cannot be distinguished; The interference category is used to indicate the valid category to which the input image is incorrectly distinguished.

11. The method according to claim 10, characterized in that The step of setting a classification label for the first sample image block based on the second classification probability distribution of the first sample image block and the valid classification to which the first sample image block belongs includes: In response to a maximum probability in the second classification probability distribution of the first sample image block being greater than a probability threshold, and the classification to which the maximum probability belongs matching the valid classification to which the first sample image block belongs, setting the classification label of the first sample image block to the valid classification to which the first sample image block belongs; In response to a maximum probability in the second classification probability distribution of the first sample image block being greater than a probability threshold, and the classification to which the maximum probability belongs not matching the valid classification to which the first sample image block belongs, setting the classification label of the first sample image block to the interference classification; In response to the maximum probability in the second classification probability distribution of the first sample image block being not greater than the probability threshold, the classification label of the first sample image block is set to the meaningless classification.

12. The method according to claim 9, characterized in that Before processing the first sample image block by the first classification model, the method further includes: Processing the second sample image block using the first classification model to obtain a second classification probability distribution of the second sample image block; the first sample image block is an image block in the second sample medical image; Based on the second classification probability distribution of the second sample image block and the valid classification to which the second sample image block belongs, parameters of the first classification model are updated.

13. The method according to claim 9, characterized in that Before processing the first sample image block by the second classification model, the method further includes: The second classification model is constructed based on a feature extraction network; the feature extraction network is the model part of the first classification model except the output layer.

14. An image processing device, characterized in that: The device comprises: A first image block extraction module, configured to extract at least one first image block from a target medical image; a first classification module, configured to map image features of at least one of the first image blocks to a first classification space to obtain a first classification probability distribution for each of the at least one first image block; wherein the classifications in the first classification probability distribution include at least two valid classifications and at least one invalid classification in the medical field; A second image block extraction module is configured to extract at least one second image block from the target medical image based on a position of the target image block in the target medical image; the distance between the second image block and the target image block is less than a distance threshold; the target image block belongs to at least one of the first image blocks, and the first classification probability distribution of the target image block indicates that the classification of the target image block is the valid classification; the first classification probability distribution includes probability values ​​corresponding to the at least two valid classifications, and probability values ​​corresponding to the at least one invalid classification, the probability values ​​being used to indicate the probability that the image block to be classified and identified belongs to the classification to which the probability values ​​belong; the valid classification refers to a classification that is helpful in making medical-related decisions; the invalid classification refers to an image block to be classified and identified that cannot be accurately identified as a valid classification; a second classification module, configured to map image features of at least one of the second image blocks to the first classification space to obtain a first classification probability distribution of each of the at least one of the second image blocks; The third classification module is used to obtain the image classification of the target medical image based on the first classification probability distribution of at least one of the second image blocks.

15. An image processing device, characterized in that: The device comprises: a first processing module, configured to process the first sample image block using a first classification model to obtain a second classification probability distribution for the first sample image block; the classifications in the second classification probability distribution include at least two valid classifications in the medical field; and the first sample image block is an image block in a first sample medical image; a label setting module, configured to set a classification label for the first sample image block based on a second classification probability distribution of the first sample image block and a valid classification to which the first sample image block belongs; the classification label being configured to indicate that the first sample image block is classified as one of at least two valid classifications and at least one invalid classification; the second classification probability distribution including probability values ​​corresponding to the at least two valid classifications, each of the probability values ​​indicating a probability that the image block being classified and identified belongs to the classification to which the probability value belongs; the valid classification being a classification that is helpful in making medical-related decisions; and the invalid classification being a classification that cannot be accurately identified as a valid classification for the image block being classified and identified; a second processing module, configured to process the first sample image block using a second classification model to obtain a first classification probability distribution of the first sample image block, wherein the classifications in the first classification probability distribution include at least two of the valid classifications and at least one of the invalid classifications; a first parameter updating module, configured to update parameters of the second classification model based on the first classification probability distribution of the first sample image block and the classification label of the first sample image block, so as to obtain an image classification model; The image classification model is used to perform the step of mapping the image features of the image block to the first classification space in the method according to any one of claims 1 to 8.

16. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that The storage medium stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 13.

18. A computer program product, characterized in that The computer program product includes computer instructions, which are read and executed by a processor of a computer device, so that the computer device executes the image processing method according to any one of claims 1 to 13.

Images