Image processing device, method, equipment, ultrasound system and readable storage medium

Abstract

The present invention discloses an image processing device, method, equipment, ultrasonic equipment, ultrasonic system and readable storage medium. The device comprises: an acquisition module, used for acquiring a liver ultrasonic image to be processed; a segmentation and classification module, used for classifying and segmenting the liver ultrasonic image to be processed, and acquiring a target classification result corresponding to the segmented image and the liver ultrasonic image to be processed; wherein the target classification result comprises a disease classification result of a lesion or a result that the liver has no lesion, and the disease classification result of the lesion comprises a classification result of a liver echinococcosis disease or a classification result of a non-liver echinococcosis disease; when the target classification result is a disease classification result of a lesion, the segmented image comprises a liver contour segmentation line and a lesion contour segmentation line; the present invention utilizes the segmentation and classification module to identify lesions of liver echinococcosis disease and other liver space-occupying diseases, thereby reducing the misdiagnosis rate and missed diagnosis rate; and the liver contour is segmented in the segmented image to assist in improving the accuracy of the lesion contour.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to an image processing device, method, equipment, ultrasound equipment, ultrasound system and readable storage medium. Background Art

[0002] With the vigorous development of deep learning technology in recent years, image recognition algorithms based on deep learning are gradually being applied to ultrasound image diagnosis, with the characteristics of high recognition rate and fast speed. Echinococcosis is a common zoonosis parasitic disease in pastoral areas, which has caused serious economic losses to the corresponding countries and regions for many years. At present, the screening and diagnosis of echinococcosis mainly relies on imaging examinations, among which ultrasound imaging technology has the advantages of being non-invasive, radiation-free and low-priced, and is the preferred technology for the diagnosis of echinococcosis.

[0003] In the existing technology, the deep learning-based hydatid disease image recognition system mainly uses algorithms such as target detection, which can only identify the lesions of hydatid disease. It has no ability to recognize the lesions of other liver space-occupying diseases with similar characteristics to hydatid disease, such as liver cysts, hepatocellular carcinoma, hilar cholangiocarcinoma, etc. The recognition ability is not high and it is very easy to cause misdiagnosis and missed diagnosis.

[0004] Therefore, how to more accurately identify the lesion contours of hydatid cysts and other liver space-occupying diseases in ultrasound images, improve the recognition ability of ultrasound images, reduce the misleading of medical staff, and reduce the problems of misdiagnosis and missed diagnosis is an issue that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide an image processing device, method, equipment, ultrasound equipment, ultrasound system and readable storage medium to improve the recognition ability of ultrasound images, reduce the problems of misdiagnosis and missed diagnosis, and assist in improving the accuracy of lesion contours.

[0006] In order to solve the above technical problems, the present invention provides an image processing device, comprising:

[0007] An acquisition module, used for acquiring an ultrasound image of the liver to be processed;

[0008] A segmentation and classification module is used to classify and segment the liver ultrasound image to be processed, and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed; wherein the target classification result includes the disease classification result of the lesion or the liver no lesion result, and the disease classification result of the lesion includes the classification result of the echinococcosis disease or the classification result of the non-echinococcosis disease; when the target classification result is the disease classification result of the lesion, the segmented image includes a liver contour segmentation line and a lesion contour segmentation line.

[0009] Optionally, the classification results of hepatic echinococcosis species include any one of 6 types of cystic hepatic echinococcosis species and 3 types of alveolar hepatic echinococcosis species; the classification results of non-hepatic echinococcosis species include any one of 8 types of liver space-occupying diseases other than the hepatic echinococcosis species.

[0010] Optionally, when the target classification result includes the disease classification result of the lesion, the target classification result also includes: a derivation probability corresponding to the disease classification result of the lesion.

[0011] Optionally, the device further comprises:

[0012] A display module is used to generate a display image according to the segmented image and the target classification result.

[0013] Optionally, the acquisition module includes:

[0014] An acquisition submodule, used for acquiring an original ultrasound image of the liver;

[0015] The preprocessing submodule is used to preprocess the original liver ultrasound image to obtain the liver ultrasound image to be processed.

[0016] Optionally, the acquisition module is further used to acquire clinical information corresponding to the liver ultrasound image to be processed;

[0017] Correspondingly, the segmentation and classification module is specifically used to utilize an integrated model of instance segmentation and classification to classify and segment the liver ultrasound image to be processed according to the clinical information, and obtain the segmented image and the target classification result.

[0018] Optionally, the device further comprises:

[0019] A model training module is used to input the liver training image and the training clinical information corresponding to each of the liver training images into the initial model corresponding to the instance segmentation and classification integrated model for iterative training, so as to obtain the trained instance segmentation and classification integrated model; wherein the liver training image includes the annotated target classification result and the liver contour segmentation line, and when the liver training image has a lesion, the liver training image also includes the annotated lesion contour segmentation line.

[0020] Optionally, the segmentation and classification module includes:

[0021] A feature extraction submodule, used to extract target features in the liver ultrasound image to be processed; wherein the target features include liver features or the liver features and lesion features;

[0022] A classification submodule, used for classifying the liver ultrasound image to be processed according to the target feature to obtain the target classification result;

[0023] The segmentation submodule is used to segment the liver ultrasound image to be processed according to the target feature to obtain the segmented image.

[0024] Optionally, the classification submodule includes:

[0025] A lesion classification unit, configured to classify the lesions of the liver ultrasound image to be processed according to the target feature, and obtain a lesion classification result; wherein the lesion classification result includes the presence of a lesion in the liver or the absence of a lesion in the liver;

[0026] The disease classification unit is used to classify the corresponding liver ultrasound image to be processed into a disease type according to the target feature when the lesion classification result is that a lesion exists in the liver, so as to obtain the disease classification result of the lesion.

[0027] Optionally, the acquisition module is specifically used to acquire an ultrasound image of the liver to be processed, and input the image of the liver to be processed into an instance segmentation and classification integrated model; wherein the instance segmentation and classification integrated model includes a feature extraction part, a segmentation branch, a lesion classification branch and a disease classification branch;

[0028] The feature extraction submodule is specifically used to extract the target features in the liver ultrasound image to be processed by using the feature extraction part;

[0029] The segmentation submodule is specifically used to segment the liver ultrasound image to be processed according to the target feature by using the segmentation branch to obtain the segmented image;

[0030] The lesion classification unit is specifically used to classify the liver ultrasound image to be processed according to the target feature by using the lesion classification branch to obtain the lesion classification result;

[0031] The disease classification unit is specifically used to utilize the disease classification branch to classify the corresponding liver ultrasound image to be processed into diseases according to the target feature, and obtain the disease classification result.

[0032] Optionally, the instance segmentation and classification integrated model adopts a deep learning network model; wherein the lesion classification branch includes a lesion convolutional layer and a lesion fully connected layer connected in sequence, and the disease type classification branch includes a ROI pooling layer, a disease type convolutional layer and a disease type fully connected layer connected in sequence.

[0033] Optionally, the lesion convolution layer includes 1 7*7 convolution layer and 1 1*1 convolution layer; the disease type convolution layer includes 3 3*3 convolution layers and 1 1*1 convolution layer.

[0034] Optionally, when the acquisition module is also used to acquire clinical information corresponding to the liver ultrasound image to be processed, the disease classification unit is specifically used to utilize the disease classification branch to classify the liver ultrasound image to be processed into a disease according to the target feature and the clinical information, and obtain the disease classification result.

[0035] The present invention also provides an ultrasound device, comprising: the image processing device as described above.

[0036] The present invention also provides an image processing method, comprising:

[0037] Acquire an ultrasound image of the liver to be processed;

[0038] The liver ultrasound image to be processed is classified and segmented to obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed; wherein the target classification result includes the disease classification result of the lesion or the result that the liver has no lesion, and the disease classification result of the lesion includes the classification result of the echinococcosis type of the liver or the classification result of the non-echinococcosis type of the liver; when the target classification result is the disease classification result of the lesion, the segmented image includes a liver contour segmentation line and a lesion contour segmentation line.

[0039] The present invention also provides an image processing device, comprising:

[0040] Memory for storing computer programs;

[0041] A processor is used to implement the steps of the image processing method as described above when executing the computer program.

[0042] The present invention also provides an ultrasound system, comprising: the image processing device as described above.

[0043] In addition, the present invention also provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image processing method as described above are implemented.

[0044] An image processing device provided by the present invention comprises: an acquisition module, used for acquiring an ultrasound image of the liver to be processed; a segmentation and classification module, used for classifying and segmenting the ultrasound image of the liver to be processed, and acquiring a target classification result corresponding to the segmented image and the ultrasound image of the liver to be processed; wherein the target classification result comprises a disease classification result of a lesion or a result that the liver has no lesion, and the disease classification result of the lesion comprises a classification result of a liver echinococcosis disease or a classification result of a non-liver echinococcosis disease; when the target classification result is a disease classification result of a lesion, the segmented image comprises a liver contour segmentation line and a lesion contour segmentation line;

[0045] It can be seen that the present invention uses the segmentation and classification module to classify and segment the liver ultrasound image to be processed, which can identify the types of liver echinococcosis and other types of liver space-occupying diseases, and segment the corresponding lesion contours in the segmented image, which can identify more types of lesions of liver space-occupying diseases, reducing the misdiagnosis rate and missed diagnosis rate; and can segment the liver contour in the segmented image, so that medical staff can more conveniently understand the relative position of the liver and the lesion, and help improve the accuracy of the lesion contour. In addition, the present invention also provides an image processing method, device, ultrasound device, ultrasound system and readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A structural block diagram of an image processing device provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram showing the display of classification and segmentation results of an image processing device provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a processing process of another image processing device provided by an embodiment of the present invention;

[0050] Figure 4 A structural block diagram of another image processing device provided by an embodiment of the present invention;

[0051] Figure 5 A simplified schematic diagram of an instance segmentation and classification integrated model provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the structure of an integrated model for instance segmentation and classification provided by an embodiment of the present invention;

[0053] Figure 7 A flowchart of an image processing method provided by an embodiment of the present invention;

[0054] Figure 8 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention;

[0055] Fig. 9 A schematic diagram of the specific structure of an image processing device provided in this embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Please refer to Figure 1 , Figure 1 This is a structural block diagram of an image processing device provided by an embodiment of the present invention. The device may include:

[0058] An acquisition module 10 is used to acquire an ultrasound image of the liver to be processed;

[0059] The segmentation and classification module 20 is used to classify and segment the liver ultrasound image to be processed, and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed; wherein the target classification result includes the disease classification result of the lesion or the result that the liver has no lesion, and the disease classification result of the lesion includes the classification result of the echinococcosis type or the classification result of the non-echinococcosis type; when the target classification result is the classification result of the echinococcosis type, the segmented image includes the liver contour segmentation line and the lesion contour segmentation line.

[0060] It can be understood that, in this embodiment, the ultrasonic image of the liver to be processed acquired by the acquisition module 10 can be an image corresponding to the ultrasonic image of the liver position collected by the ultrasonic probe in the ultrasonic device (i.e., the original ultrasonic image of the liver); for example, the ultrasonic image of the liver to be processed can be the original ultrasonic image of the liver, i.e., the segmentation and classification module 20 can directly classify and segment the original ultrasonic image of the liver; the ultrasonic image of the liver to be processed can also be the image after the original ultrasonic image of the liver is preprocessed, i.e., the segmentation and classification module 20 can classify and segment the image obtained after the original ultrasonic image of the liver is preprocessed. In other words, the acquisition module 10 can include: an acquisition submodule, used to acquire the original ultrasonic image of the liver; and a preprocessing submodule, used to preprocess the original ultrasonic image of the liver to obtain the ultrasonic image of the liver to be processed.

[0061] Specifically, the specific method for the acquisition module 10 in this embodiment to acquire the liver ultrasound image to be processed can be set by the designer according to practical scenarios and user needs. For example, when the liver ultrasound image to be processed is an original liver ultrasound image, the acquisition module 10 can directly receive the original liver ultrasound image acquired by the ultrasound probe in the ultrasound device. When the liver ultrasound image to be processed is an image preprocessed from the original liver ultrasound image, the acquisition module 10 can receive the original liver ultrasound image acquired by the ultrasound probe in the ultrasound device, and preprocess the original liver ultrasound image to obtain the liver ultrasound image to be processed.

[0062] It should be noted that, in the present embodiment, the target classification results corresponding to the liver ultrasound image to be processed acquired by the segmentation and classification module 20 may include the results corresponding to the presence of identifiable lesions in the liver ultrasound image to be processed (i.e., the disease classification results of the lesions) or the results corresponding to the absence of identifiable lesions (i.e., the result that there are no lesions in the liver); wherein, the disease classification results of the lesions may include the classification results of echinococcosis and the classification results of non-echinococcosis, that is, the identifiable lesions in the present embodiment may include lesions of echinococcosis and lesions of non-echinococcosis.

[0063] Specifically, the classification results of echinococcosis in this embodiment may include 6 types of cystic echinococcosis (such as CL, CE1, CE2, CE3, CE4 and CE5) and 3 types of alveolar echinococcosis (such as AE1, AE2 and AE3), that is, the identifiable lesions may include 9 types of specific echinococcosis lesions; the classification results of non-echinococcosis in this embodiment may include other liver space-occupying diseases other than echinococcosis, such as 8 types of liver space-occupying diseases other than echinococcosis (such as liver cysts, bacterial liver abscesses, liver hemangiomas, hepatocellular carcinomas, gallbladder cancer, hilar cholangiocarcinoma, liver cystic solid lesions and FNH), that is, the identifiable lesions may include 8 types of non-echinococcosis lesions. That is to say, the target classification result corresponding to the liver ultrasound image to be processed obtained by the segmentation classification module 20 in this embodiment may be the disease classification result of any one of the 17 types of diseases (i.e., 9 types of echinococcosis and 8 types of non-echinococcosis).

[0064] Correspondingly, the segmentation and classification module 20 in this embodiment can segment the liver contour and the lesion contour in the liver ultrasound image to be processed to obtain a segmented image, so that the liver contour segmentation line and the lesion contour segmentation line can appear in the segmented image. In other words, when there is an identifiable lesion in the liver ultrasound image to be processed, that is, when the target classification result corresponding to the liver ultrasound image to be processed is the disease classification result of the lesion, the segmented image may include a liver contour segmentation line and a lesion contour segmentation line (that is, the lesion contour segmentation line corresponding to the disease classification result of the lesion); when there is no identifiable lesion in the liver ultrasound image to be processed, that is, when the target classification result corresponding to the liver ultrasound image to be processed is a liver lesion-free result, the segmented image may include a liver contour segmentation line but not a lesion contour segmentation line.

[0065] Specifically, the specific method for the segmentation and classification module 20 in this embodiment to classify and segment the liver ultrasound image to be processed and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed can be set by the designer according to practical scenarios and user needs. For example, the segmentation and classification module 20 can use one or more models to classify and segment the liver ultrasound image to be processed and obtain the target classification result corresponding to the segmented image and the liver ultrasound image to be processed, such as using an existing model of segmentation first and then classification or classification first and then segmentation or multiple models (such as a feature extraction model, a classification model and a segmentation model) to classify and segment the liver ultrasound image to be processed and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed. Furthermore, the segmentation and classification module 20 can also use the instance segmentation and classification integrated model to classify and segment the liver ultrasound image to be processed, and obtain the target classification results corresponding to the segmented image and the liver ultrasound image to be processed; compared with the existing model of first segmentation and then classification or first classification and then segmentation, the classification and segmentation of the instance segmentation and classification integrated model can share the feature extraction part, and due to the correlation between multiple tasks, the algorithm performance can be improved, and then the efficiency of classification and segmentation can be improved. And because it is an integrated model, it can be more convenient to deploy on ultrasound products, which improves the deployment efficiency. This embodiment does not impose any restrictions on this.

[0066] Similarly, the specific process of the segmentation and classification module 20 in this embodiment classifying and segmenting the liver ultrasound image to be processed, and obtaining the segmented image and the target classification result corresponding to the liver ultrasound image to be processed, can be set by the designer. For example, the segmentation and classification module 20 can first extract the target features in the liver ultrasound image to be processed; wherein the target features include liver features or liver features and lesion features; and then classify and segment the liver ultrasound image to be processed according to the target features, respectively, to obtain the target classification result and segmented image corresponding to the liver ultrasound image to be processed. That is to say, the segmentation and classification module 20 can include: a feature extraction submodule, for extracting the target features in the liver ultrasound image to be processed; wherein the target features include liver features or liver features and lesion features; a classification submodule, for classifying the liver ultrasound image to be processed according to the target features, and obtaining the target classification result; and a segmentation submodule, for segmenting the liver ultrasound image to be processed according to the target features, and obtaining the segmented image.

[0067] It can be understood that in order to improve the accuracy of classification and segmentation of the segmentation and classification module 20, the segmentation and classification module 20 in this embodiment can use the clinical information of the patient corresponding to the liver ultrasound image to be processed to classify and segment the liver ultrasound image to be processed, and obtain the target classification results corresponding to the segmented image and the liver ultrasound image to be processed, thereby utilizing the patient's clinical information to improve the ability to identify lesions of easily confused diseases; for example, the classification submodule in the segmentation and classification module 20 can classify the liver ultrasound image to be processed according to the clinical information corresponding to the liver ultrasound image to be processed and the target features in the liver ultrasound image to be processed extracted by the feature extraction submodule, and obtain the target classification results, thereby improving the accuracy of the obtained target classification results.

[0068] Furthermore, when the target classification result corresponding to the liver ultrasound image to be processed obtained by the segmentation classification module 20 in this embodiment is the disease classification result of the lesion, the target classification result may also include: the derivation probability corresponding to the disease classification result of the lesion, so that medical staff can better understand the possibility of the lesion identified by the image processing device provided by this embodiment (i.e., the derivation probability), and further avoid misdiagnosis. Correspondingly, the acquisition module 10 can also be used to obtain clinical information corresponding to the liver ultrasound image to be processed.

[0069] Correspondingly, the image processing device provided in this embodiment may further include: a display module, which is used to generate a display image according to the segmented image and the target classification result. That is, the display module can add the target classification result to the segmented image to generate a display image, such as Figure 2 As shown, when the target classification result is the disease classification result of the lesion, the displayed image may include the disease classification result of the lesion corresponding to the lesion contour segmentation line (such as CE2) and the corresponding derivation probability (such as 0.771).

[0070] It should be noted that this embodiment is demonstrated by taking the classification and segmentation of a liver ultrasound image to be processed by the image processing device as an example. For the classification and segmentation of multiple liver ultrasound images to be processed by the image processing device, the same or similar method as the device provided in this embodiment can be used to implement it, and this embodiment does not impose any restrictions on this. Similarly, the image processing device provided in this embodiment can be applied to ultrasound equipment, such as Figure 3As shown, the ultrasound imaging system in the ultrasound device can directly retrieve the ultrasound image that needs to be analyzed and processed (i.e., the original ultrasound image of the liver) from the ultrasound system data stream, and implement the classification and segmentation of the ultrasound image through the device provided in this embodiment; the image processing device provided in this embodiment can also be applied to a computer with storage and computing functions connected to the ultrasound device, such as a host connected to the ultrasound device in the ultrasound system. For example, the computer can obtain the original ultrasound image of the liver collected by the ultrasound probe in the ultrasound device through communication or connection with a data cloud, and this embodiment also does not impose any restrictions on this.

[0071] In this embodiment, the embodiment of the present invention utilizes the segmentation and classification module 20 to classify and segment the liver ultrasound image to be processed, and can identify types of echinococcosis and other types of liver space-occupying diseases, and segment the corresponding lesion contours in the segmented image, and can identify more types of lesions of liver space-occupying diseases, thereby reducing the misdiagnosis rate and missed diagnosis rate; and can segment the liver contour in the segmented image, so that medical staff can more conveniently understand the relative position of the liver and the lesion, and help improve the accuracy of the lesion contour.

[0072] Please refer to Figure 4 , Figure 4 This is a structural block diagram of another image processing device provided by an embodiment of the present invention. The device may include:

[0073] An acquisition submodule 11 is used to acquire an original ultrasound image of the liver and clinical information corresponding to the original ultrasound image of the liver;

[0074] A preprocessing submodule 12 is used to preprocess the original liver ultrasound image to obtain a liver ultrasound image to be processed;

[0075] The segmentation and classification module 20 is used to classify and segment the liver ultrasound image to be processed according to clinical information using an integrated model of instance segmentation and classification, and obtain a segmented image and a target classification result; wherein the target classification result includes a disease classification result of the lesion or a result that the liver has no lesions, and the disease classification result of the lesion includes a classification result of echinococcosis or a classification result of non-echinococcosis; when the target classification result is a disease classification result of the lesion, the segmented image includes a liver contour segmentation line and a lesion contour segmentation line, and the target classification result also includes a derived probability corresponding to the disease classification result of the lesion; the classification result of echinococcosis includes 6 types of cystic echinococcosis and 3 types of alveolar echinococcosis; the classification result of non-echinococcosis includes 8 types of liver space-occupying diseases other than echinococcosis;

[0076] The display module 30 is used to generate a display image according to the segmented image and the target classification result.

[0077] Specifically, in this embodiment, the original ultrasound image of the liver and the corresponding clinical information acquired by the acquisition submodule 11 may be an ultrasound image of the patient's liver position acquired by an ultrasound probe in an ultrasound device and the patient's clinical information (such as epidemiological history and past medical history, etc.). For example, the acquisition submodule 11 may directly receive the original ultrasound image of the patient's liver acquired by an ultrasound probe in an ultrasound device, and receive the clinical information of the patient input or retrieved by medical staff.

[0078] It is understandable that the specific manner in which the preprocessing submodule 12 in this embodiment preprocesses the original ultrasound image of the liver to obtain the ultrasound image of the liver to be processed can be set by the designer. For example, the preprocessing submodule 12 can scale the original ultrasound image of the liver to an image of a preset size, and perform grayscale conversion on the image to obtain the ultrasound image of the liver to be processed; that is, the preprocessing submodule 12 can include a scaling unit for scaling the original ultrasound image of the liver to obtain a scaled image of a preset size; and a grayscale unit for grayscale conversion on the scaled image to obtain the ultrasound image of the liver to be processed. This embodiment does not impose any restrictions on this.

[0079] Specifically, in this embodiment, the segmentation and classification module 20 can utilize an instance segmentation and classification integrated model to perform classification and segmentation of the liver ultrasound image to be processed by inputting the liver ultrasound image to be processed and clinical information into the instance segmentation and classification integrated model, thereby obtaining a segmented image and target classification result output by the instance segmentation and classification integrated model. When the target classification result includes the disease classification result of the lesion, the disease classification result of the lesion can be any one of 9 types of hydatid disease (i.e., 6 types of cystic hydatid disease and 3 types of alveolar hydatid disease) and 8 types of non-hydatid disease (such as liver cyst, hepatocellular carcinoma and hilar cholangiocarcinoma, etc.); that is, the segmentation and classification module 20 can use the instance segmentation and classification integrated model to detect the lesions in the liver ultrasound image to be processed according to the liver ultrasound image to be processed and clinical information. If an identifiable lesion is detected, the parent type category (such as hydatid disease and non-hydatid disease such as liver cyst and hepatocellular carcinoma) is determined; if the determined parent type category is hydatid disease, the subtype classification result (i.e., 9 types of hydatid disease) is determined and output; if the determined parent type category is the other 8 types of non-hydatid disease, it can be directly output.

[0080] Correspondingly, the specific process of the segmentation and classification module 20 in this embodiment using the instance segmentation and classification integrated model to classify and segment the liver ultrasound image to be processed, that is, the specific network structure of the instance segmentation and classification integrated model, can be set by the designer according to practical scenarios and user needs, such as Figure 4As shown, the segmentation and classification module 20 may include: a feature extraction submodule 21, used to extract target features in the liver ultrasound image to be processed; wherein the target features include liver features or liver features and lesion features; a classification submodule 22, used to classify the liver ultrasound image to be processed according to the target features and clinical information, and obtain a target classification result; a segmentation submodule 23, used to segment the liver ultrasound image to be processed according to the target features, and obtain a segmented image.

[0081] Specifically, Figure 5 and Figure 6 As shown, the instance segmentation and classification integrated model adopted by the segmentation and classification module 20 can adopt a deep learning network model, based on the Mask R-CNN (an instance segmentation algorithm) network structure, and can include: the feature extraction part adopted in the feature extraction process of the feature extraction submodule 21, such as the backbone (backbone network), RPN (region proposal network) and RoI Align (a regional feature aggregation method) layer; the lesion classification branch and the disease classification branch adopted in the classification process of the classification submodule 22; and the regression (i.e., positioning) branch and the segmentation branch adopted in the segmentation process of the segmentation submodule 23.

[0082] Among them, the backbone may include two parts: a feature extraction network (such as ResNet 101, resnet50 or vgg16, etc.) and FPN (feature pyramid network); FPN is a multi-scale feature pyramid structure constructed to solve the problem of poor extraction of small features; RPN is used to generate a proposed region; the RoI Align layer maps the proposed region into a fixed-size feature map (i.e., the target feature). In other words, the feature extraction submodule 21 may include: a backbone extraction unit, which is used to use the backbone ResNet 101 and FPN to perform feature extraction on the liver ultrasound image to be processed and obtain a feature map; a proposed region unit, which is used to use the RPN network to generate a proposed region corresponding to the feature map; a region fusion unit, which is used to use the RoI Align layer to map the proposed region into a feature map of a preset size, and use the feature map as the target feature.

[0083] Correspondingly, the lesion classification branch in the integrated instance segmentation and classification model may include convolutional layers (i.e., lesion convolutional layers) and fully connected layers (FC layers, i.e., lesion fully connected layers), and the lesion classification branch may output two categories: liver and lesions (i.e., the presence or absence of lesions); the disease classification branch in the integrated instance segmentation and classification model may be a new branch added on the basis of the existing Mask R-CNN, and may include ROI pooling layers (i.e., disease type ROI pooling layers), convolutional layers (i.e., disease type convolutional layers) and fully connected layers (i.e., disease type fully connected layers), and ROI The pooling layer can significantly accelerate training and detection and improve the accuracy of detection. The disease classification branch can use target features and clinical information to further subdivide the lesions and output the disease classification results. For example, when the lesion classification branch only obtains liver output, the disease classification result output by the disease classification branch can be empty (that is, indicating that there is no lesion in the liver); when the lesion classification branch obtains liver output and lesion output, the disease classification result output by the disease classification branch can be any one of the 6 types of cystic echinococcosis, 3 types of alveolar echinococcosis and 8 types of liver space-occupying diseases. That is, if the parent type classification result is echinococcosis, the classification output is the subtype classification result (that is, 6 types of cystic echinococcosis and 3 types of alveolar echinococcosis). If the parent type classification result belongs to the other 8 types of liver space-occupying diseases, it is directly output as the disease classification result. That is to say, the classification submodule 22 may include: a lesion classification unit, which is used to classify the corresponding liver ultrasound images to be processed according to the target features, using the lesion convolution layer and the lesion fully connected layer, to obtain the lesion classification results; wherein the lesion classification results include the presence of lesions in the liver (i.e., liver output and lesion output) or the absence of lesions in the liver (i.e., liver output); a disease classification unit, which is used to classify the liver ultrasound images to be processed according to the target features and clinical information, using the disease ROI pooling layer, the disease convolution layer and the disease fully connected layer, to obtain the disease classification results of the lesions; wherein the disease convolution layer can be three 3*3 convolution layers and one 1*1 convolution layer set between the disease ROI pooling layer and the disease fully connected layer.

[0084] Correspondingly, the segmentation branch in the instance segmentation and classification integrated model may include a ROI pooling layer (i.e., a segmentation ROI pooling layer) and a fully convolutional network, which are used to implement and output the segmentation results of the liver and lesions (i.e., contour segmentation lines), and can output pixel-level segmentation masks. The regression branch in the instance segmentation and classification integrated model may include a convolutional layer (i.e., a regression convolutional layer) and a fully connected layer (FC layers, i.e., a regression fully connected layer), which are used to locate and output the liver position and the lesion position, so that the contour segmentation line output by the segmentation branch can be placed in the corresponding position; among them, Figure 6As shown, the regression branch can share the same convolution layer and part of the fully connected layer with the lesion classification branch, that is, the regression convolution layer of the regression branch and the lesion convolution layer of the lesion classification branch can be the same convolution layer, and a part of the regression fully connected layer of the regression branch and a part of the lesion fully connected layer of the lesion classification branch are shared parts of the fully connected layer. In other words, the segmentation submodule 23 can include a segmentation unit for generating a liver contour segmentation line and a lesion contour segmentation line corresponding to the disease classification result of the lesion according to the target features by using the segmentation ROI pooling layer and the fully convolutional network; a regression unit for obtaining the liver position and the lesion position in the segmented image by using the regression convolution layer and the regression fully connected layer according to the target features.

[0085] Specifically, Figure 6 As shown, the convolution layer (Convolution) shared by the regression branch and the lesion classification branch in the instance segmentation and classification integration model can include a 7*7 convolution layer and a 1*1 convolution layer. In other words, the lesion classification unit in the classification submodule 22 can be used to classify the corresponding liver ultrasound image to be processed according to the target features using the lesion convolution layer and the lesion fully connected layer to obtain the lesion classification result; wherein, the lesion convolution layer and the regression convolution layer in the regression branch are the same convolution layer, including a 7*7 convolution layer and a 1*1 convolution layer.

[0086] It should be noted that the image processing device provided in this embodiment may also include: a model training module, which is used to input the liver training image and the training clinical information corresponding to each liver training image into the initial model corresponding to the instance segmentation and classification integrated model for iterative training to obtain a trained instance segmentation and classification integrated model; wherein the liver training image includes annotated target classification results and liver contour segmentation lines, and when there is a lesion in the liver training image (that is, the target classification result is the disease classification result of the lesion), the liver training image also includes a labeled lesion contour segmentation line. For example, designers or users can use annotation software to outline the contours of the liver and lesions on the collected ultrasound images (if the image only contains the liver, only the liver contour is outlined), fit the contour edge as accurately as possible, and mark the types of lesions, including cystic echinococcosis (CL, CE1, CE2, etc., a total of 6 categories), alveolar echinococcosis (AE1, AE2, AE3, a total of 3 categories), and other liver space-occupying diseases (liver cysts, hepatocellular carcinoma, etc., a total of 8 categories), a total of 17 categories, to obtain liver training images; then the liver training images and corresponding clinical information (such as epidemiological history and past medical history) are input into the initial model corresponding to the instance segmentation and classification integration model for training, iterative optimization, and finally a model that meets the requirements (such as accuracy greater than a threshold) is trained, that is, the trained instance segmentation and classification integration model.

[0087] Specifically, the display module 30 in this embodiment can generate a display image based on the segmented image and the target classification result, so that the display image can display the target classification result (such as the disease classification result and the derived probability of the lesion) on the segmented image. Figure 2 As shown, when the target classification result includes the disease classification result of the lesion, the displayed image may include the disease classification result of the lesion corresponding to the lesion contour segmentation line (such as CE2) and the corresponding derivation probability (such as 0.771), and the derivation probability of the liver (liver) corresponding to the lesion contour segmentation line (such as 1.000).

[0088] In the present embodiment, the specific subtype of echinococcosis is identified by the segmentation classification module 20, thereby improving the accuracy of identification; and the display image is generated by the display module 30, so that the disease classification results and corresponding derivation probabilities of the identified lesions can be directly displayed in the image, thereby avoiding misleading medical staff and reducing the misdiagnosis rate.

[0089] Corresponding to the above device embodiment, an embodiment of the present invention further provides an ultrasound device, which may include: an image processing device as provided in the above embodiment.

[0090] Corresponding to the above device embodiment, the embodiment of the present invention further provides an image processing method. The application of the image processing method described below and the image processing device described above can be referred to in correspondence with each other.

[0091] Please refer to Figure 7 , Figure 7 A flowchart of an image processing method provided by an embodiment of the present invention. The method may include:

[0092] Step 101: Acquire an ultrasound image of the liver to be processed.

[0093] Optionally, this step may include: acquiring an original ultrasound image of the liver; and preprocessing the original ultrasound image of the liver to obtain an ultrasound image of the liver to be processed.

[0094] Optionally, this step may also include: acquiring clinical information corresponding to the liver ultrasound image to be processed.

[0095] Correspondingly, step 102 may include: classifying and segmenting the liver ultrasound image to be processed according to the clinical information, and obtaining target classification results corresponding to the segmented image and the liver ultrasound image to be processed.

[0096] Step 102: Classify and segment the liver ultrasound image to be processed, and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed; wherein the target classification result includes the disease classification result of the lesion or the result that the liver has no lesion, and the disease classification result of the lesion includes the classification result of the echinococcosis type of the liver or the classification result of the non-echinococcosis type of the liver; when the target classification result is the disease classification result of the lesion, the segmented image includes a liver contour segmentation line and a lesion contour segmentation line.

[0097] Optionally, the classification results of hepatic echinococcosis species may include any one of the 6 types of cystic hepatic echinococcosis species and the 3 types of alveolar hepatic echinococcosis species; the classification results of non-hepatic echinococcosis species may include any one of the 8 types of liver space-occupying diseases other than hepatic echinococcosis species.

[0098] Optionally, when the target classification result includes the disease type classification result of the lesion, the target classification result may also include: a derived probability corresponding to the disease type classification result of the lesion.

[0099] Optionally, this step may include: using an integrated model of instance segmentation and classification to classify and segment the liver ultrasound image to be processed according to clinical information, and obtaining a segmented image and a target classification result.

[0100] Correspondingly, the method provided in this embodiment may also include: inputting the liver training image and the training clinical information corresponding to each liver training image into the initial model corresponding to the instance segmentation and classification integrated model for iterative training to obtain a trained instance segmentation and classification integrated model; wherein the liver training image includes annotated target classification results and liver contour segmentation lines, and when there is a lesion in the liver training image, the liver training image also includes annotated lesion contour segmentation lines.

[0101] Optionally, the above-mentioned integrated model of instance segmentation and classification is used to classify and segment the liver ultrasound image to be processed according to clinical information, and obtain the segmented image and target classification results, which may include:

[0102] Extracting target features in the liver ultrasound image to be processed; wherein the target features include liver features or liver features and lesion features;

[0103] Classify the liver ultrasound images to be processed according to the target features and clinical information to obtain the target classification results;

[0104] According to the target features, the liver ultrasound image to be processed is segmented to obtain a segmented image.

[0105] Optionally, the instance segmentation and classification integrated model may include a feature extraction part, a segmentation branch, a lesion classification branch, and a disease classification branch;

[0106] Correspondingly, the above-mentioned integrated model of instance segmentation and classification is used to classify and segment the liver ultrasound image to be processed according to clinical information to obtain a segmented image and a target classification result, which may include: using a segmentation branch to segment the liver ultrasound image to be processed according to target features to obtain a segmented image;

[0107] Using the lesion classification branch, the liver ultrasound images to be processed are classified according to the target features to obtain the lesion classification results;

[0108] By using the disease classification branch, the corresponding liver ultrasound images to be processed are classified into disease types according to the target characteristics and clinical information to obtain the disease classification results.

[0109] Optionally, the instance segmentation and classification integrated model adopts a deep learning network model; wherein the lesion classification branch includes a lesion convolutional layer and a lesion fully connected layer connected in sequence, and the disease type classification branch includes a ROI pooling layer, a disease type convolutional layer and a disease type fully connected layer connected in sequence.

[0110] Optionally, the lesion convolution layer includes 1 7*7 convolution layer and 1 1*1 convolution layer; the disease type convolution layer includes 3 3*3 convolution layers and 1 1*1 convolution layer.

[0111] Optionally, the method provided in this embodiment may further include: generating a display image according to the segmented image and the target classification result.

[0112] In this embodiment, the embodiment of the present invention can identify echinococcosis and other liver space-occupying diseases by classifying and segmenting the liver ultrasound image to be processed, and segment the corresponding lesion contours in the segmented image, so that more types of lesions of liver space-occupying diseases can be identified, reducing the misdiagnosis rate and missed diagnosis rate; and the liver contour can be segmented in the segmented image, so that medical staff can more conveniently understand the relative position of the liver and the lesion, and help improve the accuracy of the lesion contour.

[0113] Corresponding to the above method embodiment, the embodiment of the present invention further provides an image processing device. The image processing device described below and the image processing method described above can refer to each other.

[0114] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. The image processing device may include:

[0115] A memory D1, for storing computer programs;

[0116] The processor D2 is used to implement the steps of the image processing method provided by the above method embodiment when executing a computer program.

[0117] Specifically, the image processing device in this embodiment can be specifically an ultrasound device in an ultrasound system; or can be specifically a host connected to the ultrasound device in the ultrasound system. Fig. 9 , is a schematic diagram of a specific structure of an image processing device provided in this embodiment. The image processing device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 322 (for example, one or more processors) and memory 332, one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the image processing device 301.

[0118] The image processing device 301 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341. For example, Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0119] The steps in the image processing method described above can be implemented by the structure of an image processing device.

[0120] Corresponding to the above method embodiment, an embodiment of the present invention further provides a readable storage medium. The readable storage medium described below and the image processing method described above can refer to each other.

[0121] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method of the above method embodiment are implemented.

[0122] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0123] In addition, an embodiment of the present invention further provides an ultrasound system, including: an image processing device as provided in the above embodiment.

[0124] Optionally, when the image processing device is specifically an ultrasound device, the ultrasound system may further include an ultrasound probe and a host (such as a PC) connected to the image processing device.

[0125] Optionally, when the image processing device is specifically a host connected to an ultrasound device, the ultrasound system may further include the ultrasound device and an ultrasound probe connected to the ultrasound device.

[0126] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. As for the methods, devices, ultrasound devices, ultrasound systems, and readable storage media disclosed in the embodiments, since they correspond to the devices disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0127] The above is a detailed introduction to an image processing device, method, equipment, ultrasound equipment, ultrasound system and readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An image processing device, characterized in that: include: An acquisition module, used for acquiring an ultrasound image of the liver to be processed; A segmentation and classification module is used to classify and segment the liver ultrasound image to be processed, and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed; wherein the target classification result includes the disease classification result of the lesion or the liver lesion-free result, and the disease classification result of the lesion includes the classification result of the echinococcosis of the liver or the classification result of the non-echinococcosis of the liver; when the target classification result is the disease classification result of the lesion, the segmented image includes a liver contour segmentation line and a lesion contour segmentation line; The segmentation and classification module is specifically used to classify and segment the liver ultrasound image to be processed by using the instance segmentation and classification integrated model; the acquisition module is specifically used to acquire the liver ultrasound image to be processed, and input the liver ultrasound image to be processed into the instance segmentation and classification integrated model; wherein the instance segmentation and classification integrated model includes a feature extraction part, a segmentation branch, a lesion classification branch and a disease classification branch; The instance segmentation and classification integrated model is used to: Utilizing the feature extraction part to extract target features in the liver ultrasound image to be processed; wherein the target features include liver features or the liver features and lesion features; Using the segmentation branch, segmenting the liver ultrasound image to be processed according to the target feature to obtain the segmented image; Using the lesion classification branch, the liver ultrasound image to be processed is classified according to the target feature to obtain a lesion classification result; wherein the lesion classification result includes whether a lesion exists in the liver or whether there is no lesion in the liver; When the lesion classification result indicates that a lesion exists in the liver, using the disease classification branch, according to the target feature, classify the corresponding liver ultrasound image to be processed into a disease, and obtain the disease classification result; The instance segmentation and classification integrated model adopts a deep learning network model; wherein the lesion classification branch includes a lesion convolution layer and a lesion fully connected layer connected in sequence, and the disease type classification branch includes a ROIpooling layer, a disease type convolution layer and a disease type fully connected layer connected in sequence; The instance segmentation and classification integrated model also includes a regression branch for locating and outputting a liver position and a lesion position; the regression branch and the lesion classification branch in the instance segmentation and classification integrated model share a convolutional layer.

2. The image processing device according to claim 1, characterized in that The classification results of the hepatic echinococcosis species include any one of the 6 types of cystic hepatic echinococcosis species and the 3 types of alveolar hepatic echinococcosis species; the classification results of the non-hepatic echinococcosis species include any one of the 8 types of liver space-occupying diseases other than the hepatic echinococcosis species.

3. The image processing device according to claim 1, characterized in that When the target classification result includes the disease type classification result of the lesion, the target classification result also includes: a derivation probability corresponding to the disease type classification result of the lesion.

4. The image processing device according to claim 1, characterized in that: Also includes: A display module is used to generate a display image according to the segmented image and the target classification result.

5. The image processing device according to claim 1, characterized in that: The acquisition module comprises: An acquisition submodule, used for acquiring an original ultrasound image of the liver; The preprocessing submodule is used to preprocess the original liver ultrasound image to obtain the liver ultrasound image to be processed.

6. The image processing device according to claim 1, characterized in that: Also includes: A model training module is used to input the liver training image and the training clinical information corresponding to each of the liver training images into the initial model corresponding to the instance segmentation and classification integrated model for iterative training, so as to obtain the trained instance segmentation and classification integrated model; wherein the liver training image includes the annotated target classification result and the liver contour segmentation line, and when the liver training image has a lesion, the liver training image also includes the annotated lesion contour segmentation line.

7. The image processing device according to claim 1, characterized in that: The feature extraction part includes: A backbone extraction unit, used to extract features of the liver ultrasound image to be processed by using the backbone ResNet and FPN to obtain a feature map; A proposed region unit, used to generate a proposed region corresponding to the feature map using an RPN network; The region fusion unit is used to map the proposed region into a feature map of a preset size using a RoI Align layer, and use the feature map as the target feature.

8. The image processing device according to claim 1, characterized in that: The lesion convolution layer includes 1 7*7 convolution layer and 1 1*1 convolution layer; the disease type convolution layer includes 3 3*3 convolution layers and 1 1*1 convolution layer.

9. The image processing device according to claim 8, characterized in that: The convolutional layers shared by the regression branch and the lesion classification branch in the instance segmentation and classification integrated model include a 7*7 convolutional layer and a 1*1 convolutional layer.

10. The image processing device according to claim 1, characterized in that: When the acquisition module is also used to acquire clinical information corresponding to the liver ultrasound image to be processed, the disease classification unit is specifically used to use the disease classification branch to classify the liver ultrasound image to be processed into a disease according to the target feature and the clinical information, and obtain the disease classification result.

11. An ultrasonic device, characterized in that: include: An image processing device as claimed in any one of claims 1 to 10.

12. An image processing method, characterized in that: include: Acquire an ultrasound image of the liver to be processed, and input the image of the liver to be processed into an integrated model of instance segmentation and classification; The instance segmentation and classification integrated model is used to classify and segment the liver ultrasound image to be processed, and obtain the segmented image and the target classification result corresponding to the liver ultrasound image to be processed; wherein the target classification result includes the disease classification result of the lesion or the liver lesion-free result, and the disease classification result of the lesion includes the classification result of the echinococcosis type of the liver or the classification result of the non-echinococcosis type of the liver; when the target classification result is the disease classification result of the lesion, the segmented image includes a liver contour segmentation line and a lesion contour segmentation line; wherein the instance segmentation and classification integrated model includes a feature extraction part, a segmentation branch, a lesion classification branch, and a disease classification branch; The instance segmentation and classification integrated model is used to: Utilizing the feature extraction part to extract target features in the liver ultrasound image to be processed; wherein the target features include liver features or the liver features and lesion features; Using the segmentation branch, segmenting the liver ultrasound image to be processed according to the target feature to obtain the segmented image; Using the lesion classification branch, the liver ultrasound image to be processed is classified according to the target feature to obtain a lesion classification result; wherein the lesion classification result includes whether a lesion exists in the liver or whether there is no lesion in the liver; When the lesion classification result indicates that a lesion exists in the liver, using the disease classification branch, according to the target feature, classify the corresponding liver ultrasound image to be processed into a disease, and obtain the disease classification result; The instance segmentation and classification integrated model adopts a deep learning network model; wherein the lesion classification branch includes a lesion convolution layer and a lesion fully connected layer connected in sequence, and the disease type classification branch includes a ROIpooling layer, a disease type convolution layer and a disease type fully connected layer connected in sequence; The instance segmentation and classification integrated model also includes a regression branch for locating and outputting a liver position and a lesion position; the regression branch and the lesion classification branch in the instance segmentation and classification integrated model share a convolutional layer.

13. An image processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image processing method as claimed in claim 12 when executing the computer program.

14. An ultrasound system, characterized in that: include: The image processing apparatus according to claim 13.

15. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to claim 12 are implemented.

Images