Identification method of tumor features in ultrasonic image based on information fusion and multi-task learning, and training method and device of identification model
Through the method of information fusion and multi-task learning, combined with ultrasound images and clinical information, the classification and identification results of tumors and tissue morphology are generated, which solves the problems of insufficient accuracy and lack of interpretability of ultrasound tumor recognition in the existing technology, and improves the Identification accuracy and expert judgment efficiency.
Patent Information
- Application Number
- CN202410943686.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-29
AI Technical Summary
The existing deep learning model relies on ultrasound images in ultrasound tumor recognition and ignores clinical information, resulting in insufficient recognition accuracy and lack of interpretability, limiting its application in clinical practice.
Using a method based on information fusion and multi-task learning, the target image features and target text features are output by obtaining initial images and clinical information related to the tumor, combining image feature extraction networks and text feature extraction networks, output target image features and target text features, and input them into multi-task classification , generates tumor classification and tissue morphology identification results.
It improves the accuracy and interpretability of the tumor recognition model, enhances the efficiency and accuracy of expert judgments, supplements image features through text features, and provides scientific explanation support.
Smart Images

Figure CN119048796A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer-aided tumor recognition, and in particular to a method for recognizing tumor features in ultrasound images based on information fusion and multi-task learning, and a method and device for training a recognition model. Background Art
[0002] In recent years, with the development of imaging technology, medical imaging technologies including magnetic resonance imaging and ultrasound have played an increasingly important role in tumor identification. Ultrasound imaging has been increasingly widely used in tumor examinations of soft tissue organs such as breast, thyroid, and liver due to its advantages such as high soft tissue contrast, fast imaging speed, no radiation, and economic convenience. However, tumor feature recognition based on ultrasound images requires high professional knowledge, and the noise and shadows in ultrasound images increase the difficulty of expert judgment, resulting in large differences in the judgment results of different experts.
[0003] With the rapid development of deep learning algorithms, computer-aided diagnosis based on deep learning has received widespread attention in ultrasound tumor diagnosis. The rich accumulation of ultrasound images can provide a lot of useful information for building a "data-driven" deep learning model, which is expected to enable the deep learning model to quickly and accurately provide doctors with an important "second opinion", thereby improving the efficiency and accuracy of doctors' diagnosis. However, the existing deep learning models only rely on ultrasound images and ignore clinical information and doctors' expertise, which to some extent limits the accuracy of the model in tumor identification. In addition, the current deep learning models lack interpretability, which also limits their application in clinical practice. Summary of the invention
[0004] In view of the above problems, the present disclosure provides a method for identifying tumor features in ultrasound images based on information fusion and multi-task learning, and a method and device for training a recognition model.
[0005] According to a first aspect of the present disclosure, a method for identifying tumor features in ultrasound images based on information fusion and multi-task learning is provided, comprising:
[0006] Acquire the initial image and clinical information related to the tumor; input the initial image into the image feature extraction network and output the target image features; determine the text information based on the tumor image information and clinical information, wherein the tumor image information is obtained based on the initial image; input the text information into the text feature extraction network and output the target text features; fuse the target image features and the target text features into a multi-task classifier and output the classification recognition results and the tissue morphology recognition results.
[0007] According to an embodiment of the present disclosure, the image feature extraction network includes a channel attention layer and a fully connected layer; wherein the initial image is input into the image feature extraction network, and the target image features are output, including:
[0008] The initial image is cropped and expanded to obtain a full-area image; pixel multiplication operations are performed on the initial image and the tumor mask image to obtain an intratumoral area image, wherein the tumor mask image is obtained based on the segmentation of the initial image; pixel multiplication operations are performed on the initial image and the edge mask image to obtain an edge area image, wherein the edge mask image is obtained based on edge detection operations and morphological operations on the tumor mask image; feature extraction is performed on the full-area image, the intratumoral area image and the edge area image respectively, and full-area image features, intratumoral area image features and edge area image features are output; the full-area image features, intratumoral area image features and edge area image features are input into the channel attention layer to output fused image features; the fused image features are input into the fully connected layer to output target image features.
[0009] According to an embodiment of the present disclosure, the channel attention layer includes an average pooling sublayer, a convolution sublayer, and an activation sublayer; wherein the full-area image features, the intratumoral area image features, and the edge area image features are input into the channel attention layer, and the fused image features are output, including:
[0010] The image features of the entire region, the image features of the intratumor region and the image features of the edge region are processed according to the average pooling sublayer to obtain image pooling features; the image pooling features are processed according to the convolution sublayer to obtain image convolution features; the image convolution features are processed according to the activation sublayer to obtain image activation features; pixel multiplication operations are performed on the image features of the entire region, the image features of the intratumor region, the image features of the edge region and the image activation features to obtain fused image features.
[0011] According to an embodiment of the present disclosure, tumor image information is obtained based on the initial image, including:
[0012] The target echo factor information of the tumor is obtained according to the grayscale value of the first orientation area of the tumor, the grayscale value of the second orientation area of the tumor and the grayscale value of the third orientation area of the tumor in the initial image; the shape information of the tumor is determined according to the tumor mask image; the full area image, the intratumor area image and the edge area image are processed respectively according to the grayscale co-occurrence matrix algorithm to obtain the grayscale co-occurrence matrices corresponding to the full area image, the intratumor area image and the edge area image respectively; the grayscale co-occurrence matrices corresponding to the full area image, the intratumor area image and the edge area image are extracted to obtain the texture information corresponding to the full area image, the intratumor area image and the edge area image respectively; the tumor imaging information is obtained according to the target echo factor information of the tumor, the shape information of the tumor and the texture information.
[0013] According to an embodiment of the present disclosure, the text feature extraction network includes a fully connected layer and an activation layer; wherein text information is input into the text feature extraction network, and target text features are output, including:
[0014] The text information is processed according to the fully connected layer to obtain the initial text features; the initial text features are processed according to the activation layer to obtain the target text features.
[0015] According to an embodiment of the present disclosure, the multi-task classifier includes a main classifier and an auxiliary classifier, wherein the main classifier is used to identify the type of the tumor, and the auxiliary classifier is used to identify the tissue morphology related to the tumor; wherein the target image features and the target text features are fused and input into the multi-task classifier, and the classification recognition results and the tissue morphology recognition results are output, including:
[0016] The target image features and the target text features are fused to obtain the target fused features; the target fused features are subjected to dimensionality reduction operations to obtain the target reduced dimensionality features; the target reduced dimensionality features are processed according to the auxiliary classifier to obtain the tissue morphology recognition results; the tissue morphology recognition results and the target reduced dimensionality features are processed according to the main classifier to obtain the classification recognition results.
[0017] According to an embodiment of the present disclosure, the auxiliary classifier includes a hidden sublayer, a fully connected sublayer and an activated sublayer; wherein, the target dimension reduction features are processed by the auxiliary classifier to obtain a tissue morphology recognition result, including:
[0018] The target dimension reduction features are processed according to the hidden sub-layer to obtain the hidden features; the hidden features are processed according to the fully connected sub-layer to obtain the tissue morphology features; the tissue morphology features are processed according to the activated sub-layer to obtain the tissue morphology recognition results.
[0019] A second aspect of the present disclosure provides a training method for a tumor feature recognition model in ultrasound images, wherein the recognition model includes an image feature extraction network, a text feature extraction network, and a multi-task classifier; the training method includes:
[0020] Acquire training samples, which include sample initial images, sample clinical information, classification labels, and tissue morphology labels related to the tumor samples, wherein the classification labels characterize the severity of the tumor samples; input the sample initial images into an image feature extraction network, and output sample target image features; determine sample text information based on the sample tumor imaging information and the sample clinical information, wherein the sample tumor imaging information is obtained based on the sample initial images; input the sample text information into a text feature extraction network, and output sample target text features; fuse the sample target image features and the sample target text features into a multi-task classifier, and output sample classification recognition results and sample tissue morphology recognition results; train a recognition model based on the sample classification recognition results, the sample tissue morphology recognition results, the classification labels, and the tissue morphology labels to obtain a trained recognition model.
[0021] According to an embodiment of the present disclosure, a recognition model is trained based on sample classification recognition results, sample tissue morphology recognition results, classification labels, and tissue morphology labels to obtain a trained recognition model, including:
[0022] The sample tissue morphology recognition results and tissue morphology labels are processed according to the first loss function to obtain a tissue morphology loss value; the sample classification recognition results and classification labels are processed according to the second loss function to obtain a classification loss value; the recognition model is trained according to the tissue morphology loss value and the classification loss value to obtain a trained recognition model.
[0023] A third aspect of the present disclosure provides a device for identifying tumor features in ultrasound images based on information fusion and multi-task learning, comprising:
[0024] A first acquisition module is used to acquire initial images and clinical information related to the tumor;
[0025] A first extraction module, used for inputting the initial image into the image feature extraction network and outputting target image features;
[0026] A first determination module is used to determine text information according to tumor image information and clinical information, wherein the tumor image information is obtained based on the initial image;
[0027] The second extraction module is used to input text information into a text feature extraction network and output target text features;
[0028] The first recognition module is used to fuse the target image features and the target text features into a multi-task classifier, and output classification recognition results and tissue morphology recognition results.
[0029] A fourth aspect of the present disclosure provides a training device for a recognition model for recognizing tumors in ultrasound images, comprising:
[0030] The second acquisition module is used to acquire training samples, where the training samples include sample initial images, sample clinical information, classification labels, and tissue morphology labels related to the tumor samples, wherein the classification labels represent the severity of the tumor samples;
[0031] The third extraction module is used to input the sample initial image into the image feature extraction network and output the target image features;
[0032] A second determination module is used to determine text information based on tumor image information and clinical information, wherein the tumor image information is obtained based on the initial image;
[0033] A fourth extraction module, used for inputting text information into a text feature extraction network and outputting target text features;
[0034] The second recognition module is used to fuse the target image features and the target text features into a multi-task classifier, and output the classification recognition results and the tissue morphology recognition results;
[0035] The training module is used to train the recognition model according to the sample classification recognition results, the sample tissue morphology recognition results, the classification labels and the tissue morphology labels to obtain the trained recognition model.
[0036] According to the method for identifying tumor features in ultrasound images based on information fusion and multi-task learning, the method for training a recognition model and the device provided by the present disclosure, tumor image information is extracted based on the initial image, and the tumor image information and clinical information are fused and input into a text feature extraction network so that the information acquired by the text feature extraction network is more comprehensive, thereby improving the recognition accuracy of the recognition model; the target text features and deep target image features are fused and input into a multi-task classifier, and the target text features are effectively used to supplement the target image features learned by the image feature extraction network, thereby further improving the recognition accuracy of the recognition model; the multi-task classifier outputs the classification recognition results and tissue morphology recognition results of the tumor, and different tissue morphology recognition results can provide a scientific explanation for the classification recognition results, thereby better assisting expert judgment and improving the efficiency and accuracy of expert diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0038] Figure 1 A flow chart of a method for identifying tumor features in ultrasound images based on information fusion and multi-task learning according to an embodiment of the present disclosure is shown;
[0039] Figure 2An example schematic diagram of generating a full-region image, an intratumor region image, and a marginal region image according to an embodiment of the present disclosure is shown;
[0040] Figure 3A An example schematic diagram of a channel attention layer according to an embodiment of the present disclosure is shown;
[0041] Figure 3B An example schematic diagram of a recognition model according to an embodiment of the present disclosure is shown;
[0042] Figure 4 A flowchart of a method for training a recognition model for identifying tumors in ultrasound images according to an embodiment of the present disclosure is shown;
[0043] Figure 5 An example schematic diagram showing the recognition classification recognition result and the tissue morphology recognition result according to an embodiment of the present disclosure;
[0044] Figure 6 A structural block diagram of a device for identifying tumors in ultrasound images based on information fusion and multi-task learning according to an embodiment of the present disclosure is shown;
[0045] Figure 7 A structural block diagram of a training device for a recognition model for identifying tumors in ultrasound images according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0046] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0047] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0048] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0049] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0050] In the process of realizing the present disclosure, it was found that in recent years, with the development of imaging technology, medical imaging technologies including magnetic resonance imaging and ultrasound have played an increasingly important role in tumor identification. Ultrasound imaging has been increasingly widely used in tumor examinations of soft tissue organs such as breast, thyroid, and liver due to its advantages such as high soft tissue contrast, fast imaging speed, no radiation, and economic convenience. However, tumor feature recognition based on ultrasound images requires high professional knowledge, and the noise and shadows in ultrasound images increase the difficulty of expert judgment, resulting in large differences in the judgment results of different experts. Existing deep learning models rely on learning ultrasound images, and therefore, to a certain extent, limit the accuracy of model recognition and reduce the efficiency and accuracy of expert judgment of tumor features.
[0051] In view of this, the embodiments of the present disclosure provide a method for identifying tumor features in ultrasound images based on information fusion and multi-task learning, a method for training a recognition model, and a device. The method includes: obtaining an initial image and clinical information related to the tumor; inputting the initial image into an image feature extraction network and outputting target image features; determining text information based on tumor image information and clinical information; inputting the text information into a text feature extraction network and outputting target text features; fusion of the target image features and the target text features into a multi-task classifier and outputting classification recognition results and tissue morphology recognition results.
[0052] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0053] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.
[0054] Figure 1 A flowchart of a method for identifying tumor features in ultrasound images based on information fusion and multi-task learning according to an embodiment of the present disclosure is shown.
[0055] like Figure 1 As shown, the method 100 includes operations S110 to S150.
[0056] In operation S110 , an initial image and clinical information related to the tumor are acquired.
[0057] In operation S120, the initial image is input into an image feature extraction network, and target image features are output.
[0058] In operation S130 , text information is determined based on the tumor image information and the clinical information.
[0059] In operation S140, text information is input into a text feature extraction network, and target text features are output.
[0060] In operation S150 , the target image features and the target text features are fused and input into a multi-task classifier, and a classification recognition result and a tissue morphology recognition result are output.
[0061] According to an embodiment of the present disclosure, the recognition model includes an image feature extraction network, a text feature extraction network and a multi-task classifier.
[0062] According to an embodiment of the present disclosure, the initial image represents the original image of the breast of a tumor patient collected using an ultrasound device. The clinical information represents the clinical manifestation information of the tumor patient, for example, the patient's age information. The test data includes 1508 initial images of benign tumors and 707 initial images of malignant tumors.
[0063] According to an embodiment of the present disclosure, the image feature extraction network can be constructed based on a convolutional neural network (Densely connected convolutional networks 121, Densenet121), and the initial image is processed according to the image feature extraction network to obtain the target image feature. The target image feature represents the deep features of the image.
[0064] According to an embodiment of the present disclosure, the tumor image information is obtained based on the initial image. For example, information is extracted from the initial image to obtain the tumor image information.
[0065] According to an embodiment of the present disclosure, the tumor image information represents the ultrasound detection information related to the tumor tissue in the initial image, for example, tumor shape information, tumor texture information, etc.
[0066] According to the embodiment of the present disclosure, clinical information is important reference information for experts to diagnose benign and malignant tumors, so the clinical information is integrated with tumor imaging information to obtain text information. The text information is standardized and stored in a table file.
[0067] According to an embodiment of the present disclosure, the text feature extraction network can be constructed based on a feedforward neural network, and the text information in the table file is subjected to feature extraction according to the text feature extraction network to obtain target text features. The target text features represent the deep features of the text related to the tumor.
[0068] According to an embodiment of the present disclosure, a multi-task classifier is used to generate a recognition result corresponding to each recognition task. For example, the recognition task may be a tissue morphology recognition task and a tumor classification task, and the tissue morphology recognition task may be divided into a tumor edge recognition task, a tumor shape recognition task, and the like.
[0069] According to an embodiment of the present disclosure, the target image features and the target text features are fused, and the fused features are input into a multi-task classifier to output a classification recognition result corresponding to the tumor classification task and a tissue morphology recognition result corresponding to the tissue morphology recognition task.
[0070] According to an embodiment of the present disclosure, the classification recognition result represents the recognition result of predicting the tumor category based on the multi-task classifier, and the tissue morphology recognition result represents the recognition result of predicting the tumor tissue morphology based on the multi-task classifier.
[0071] According to an embodiment of the present disclosure, the classification recognition result and the tissue morphology recognition result are used as intermediate result information for judging the health status, and all the above steps are implemented and executed by a computer or other device.
[0072] According to the embodiments of the present disclosure, tumor imaging information is extracted based on the initial image, and the tumor imaging information and clinical information are fused and input into the text feature extraction network to make the information obtained by the recognition model more comprehensive, thereby improving the accuracy of model recognition; the target text features and deep target image features are fused and input into the multi-task classifier, and the target text features are effectively used to supplement the target image features learned by the image feature extraction network, thereby further improving the recognition accuracy of the recognition model; the classification recognition results and tissue morphology recognition results of the tumor are predicted based on the recognition model, and different tissue morphology recognition results can provide explanations for the classification recognition results, thereby better assisting expert judgment and improving the efficiency and accuracy of expert diagnosis.
[0073] According to an embodiment of the present disclosure, an image feature extraction network includes a channel attention layer and a fully connected layer.
[0074] According to an embodiment of the present disclosure, the image feature extraction network may include a channel attention layer, a fully connected layer (FC) and three independent convolutional neural network layers (Densenet121). The channel attention layer may be constructed based on the attention mechanism (Coordinate attention, CA).
[0075] According to an embodiment of the present disclosure, an initial image is input into an image feature extraction network, and target image features are output, including:
[0076] The initial image is cropped and expanded to obtain a full-area image; pixel multiplication operations are performed on the initial image and the tumor mask image to obtain an intratumoral area image, wherein the tumor mask image is obtained based on the segmentation of the initial image; pixel multiplication operations are performed on the initial image and the edge mask image to obtain an edge area image, wherein the edge mask image is obtained based on edge detection operations and morphological operations on the tumor mask image; feature extraction is performed on the full-area image, the intratumoral area image and the edge area image respectively, and full-area image features, intratumoral area image features and edge area image features are output; the full-area image features, intratumoral area image features and edge area image features are input into the channel attention layer to output fused image features; the fused image features are input into the fully connected layer to output target image features.
[0077] According to the embodiment of the present disclosure, the cropping rule is set based on the longest diameter of the tumor in the initial image, and the initial image is cropped and expanded to obtain a full-area image. The full-area image contains the characteristic information of the tumor itself and the tumor growth environment information.
[0078] For example, cropping is performed in sequence from top to bottom and from left to right. In the cropped initial image, the upper edge of the tumor in the image is 0.4 times the length of the longest diameter of the tumor, the lower edge of the tumor in the image is 0.8 times the length of the longest diameter of the tumor, the left edge of the tumor in the image is 0.6 times the length of the longest diameter of the tumor, and the right edge of the tumor in the image is 0.6 times the length of the longest diameter of the tumor.
[0079] According to the embodiments of the present disclosure, a larger cropping area is reserved below the tumor in order to fully retain the rear echo information of the tumor, which is one of the important indicators for evaluating the benign or malignant nature of breast tumors.
[0080] According to an embodiment of the present disclosure, a larger cropped area is zero-filled and expanded in the cropped initial image to obtain a full-area image, thereby retaining the original aspect ratio of the tumor. The aspect ratio is also one of the important indicators for evaluating the benign and malignant nature of breast tumors.
[0081] According to an embodiment of the present disclosure, segmenting the initial image to obtain a tumor mask image may include: assigning a value of 1 to pixels in the tumor area of the initial image and a value of 0 to pixels in the non-tumor area, thereby obtaining a tumor mask image composed of 0 and 1 pixels.
[0082] According to the embodiment of the present disclosure, after pixel multiplication operation is performed on the initial image and the tumor mask image, the above-mentioned cropping and expansion operations are performed to obtain the intratumor region image. The intratumor region image contains the key feature information of the tumor itself.
[0083] According to an embodiment of the present disclosure, edge detection is first performed on the tumor mask image, and then morphological operation is performed to obtain the edge mask image. The edge detection operation on the tumor mask image may include: detecting the tumor edge in the tumor mask image according to an edge detection algorithm.
[0084] According to an embodiment of the present disclosure, the morphological operation may include: using dilation and erosion operations in morphology to expand and contract the edge of the tumor, assigning 1 to the pixel value between the expanded and contracted contours, and assigning zero to the remaining pixels.
[0085] According to the embodiments of the present disclosure, after pixel multiplication operation is performed on the initial image and the edge mask image, the above-mentioned cropping and expansion operations are performed to obtain the edge region image. For malignant tumors, the initial image often has inaccurate edge information due to its invasiveness, while the edge region image includes the relationship information between the tumor and the surrounding tissues and has highly accurate edge information.
[0086] Figure 2 An example schematic diagram of generating a full-region image, an intratumoral region image, and a marginal region image according to an embodiment of the present disclosure is shown.
[0087] like Figure 2 As shown, the initial image 210 is cropped and expanded to obtain a full-area image 220; the initial image 210 is segmented to obtain a tumor mask image 230, and a pixel multiplication operation is performed on the initial image 210 and the tumor mask image 230 to obtain an intratumoral area image 240; an edge detection operation and a morphological operation are performed on the tumor mask image 230 to obtain an edge mask image 250, and a pixel multiplication operation is performed on the initial image 210 and the edge mask image 250 to obtain an edge area image 260.
[0088] According to an embodiment of the present disclosure, the full-area image, the intratumoral area image and the edge area image are respectively input into three independent convolutional neural network layers for feature extraction, and the full-area image features, the intratumoral area image features and the edge area image features are output respectively.
[0089] According to the embodiments of the present disclosure, each convolutional neural network layer includes a linear sublayer, which is used to convert the number of features. For example, feature conversion is performed in each linear sublayer to obtain 1024 features, and finally 1024 full-area image features, 1024 intratumoral area image features, and 1024 edge area image features are obtained.
[0090] According to an embodiment of the present disclosure, the full-region image features, the intratumor region image features and the edge region image features are feature fused according to the channel attention layer to obtain fused image features. The fully connected layer is used for feature dimensionality reduction. For example, the fused image features are reduced in dimension according to the fully connected layer to obtain 1536 target image features.
[0091] According to the embodiments of the present disclosure, an image feature extraction network is used to extract and fuse corresponding deep learning features from the full-area image of the tumor, the intratumoral area image, and the edge area image, thereby making full use of the tumor information in different areas of the image.
[0092] According to an embodiment of the present disclosure, the channel attention layer includes an average pooling sublayer, a convolution sublayer and an activation sublayer.
[0093] According to an embodiment of the present disclosure, the convolution sublayer may be constructed based on a convolutional neural network (CNN) with a convolution kernel of 1*1. The activation sublayer may be constructed based on an activation function Sigmoid.
[0094] According to an embodiment of the present disclosure, the full-area image features, the intratumoral area image features and the edge area image features are input into the channel attention layer, and the fused image features are output, including:
[0095] The image features of the entire region, the image features of the intratumor region and the image features of the edge region are processed according to the average pooling sublayer to obtain image pooling features; the image pooling features are processed according to the convolution sublayer to obtain image convolution features; the image convolution features are processed according to the activation sublayer to obtain image activation features; pixel multiplication operations are performed on the image features of the entire region, the image features of the intratumor region, the image features of the edge region and the image activation features to obtain fused image features.
[0096] For example, taking the above-mentioned 1024 full-area image features, 1024 intratumoral area image features and 1024 edge area image features as examples, the full-area image features, the intratumoral area image features and the edge area image features are processed according to the average pooling sublayer to obtain 1024*3 image pooling features, and then the 1024*3 image pooling features are processed according to the convolution sublayer to obtain 1024*3 image convolution features, and finally the 1024*3 image convolution features are processed according to the activation sublayer to obtain 1024*3 image activation features.
[0097] For example, pixel multiplication operation is performed on 1024 full-area image features and the corresponding 1024 image activation features to obtain 1024 fused image features corresponding to the full-area image features; pixel multiplication operation is performed on 1024 intratumoral area image features and the corresponding 1024 image activation features to obtain 1024 fused image features corresponding to the intratumoral area image features; pixel multiplication operation is performed on 1024 edge area image features and the corresponding 1024 image activation features to obtain 1024 fused image features corresponding to the edge area image features.
[0098] Figure 3A An example schematic diagram of a channel attention layer according to an embodiment of the present disclosure is shown.
[0099] like Figure 3A As shown, the image features of the entire region, the image features of the intratumor region and the image features of the edge region are processed according to the average pooling sublayer to obtain image pooling features; the image pooling features are processed according to the convolution sublayer to obtain image convolution features; the image convolution features are processed according to the activation sublayer to obtain image activation features; pixel multiplication operations are performed on the image features of the entire region, the image features of the intratumor region, the image features of the edge region and the image activation features to obtain fused image features.
[0100] According to an embodiment of the present disclosure, tumor image information is obtained based on the initial image, including:
[0101] The target echo factor information of the tumor is obtained according to the grayscale value of the first orientation area of the tumor, the grayscale value of the second orientation area of the tumor and the grayscale value of the third orientation area of the tumor in the initial image; the shape information of the tumor is determined according to the tumor mask image; the full area image, the intratumor area image and the edge area image are processed respectively according to the grayscale co-occurrence matrix algorithm to obtain the grayscale co-occurrence matrices corresponding to the full area image, the intratumor area image and the edge area image respectively; the grayscale co-occurrence matrices corresponding to the full area image, the intratumor area image and the edge area image are extracted to obtain the texture information corresponding to the full area image, the intratumor area image and the edge area image respectively; the tumor imaging information is obtained according to the target echo factor information of the tumor, the shape information of the tumor and the texture information.
[0102] According to an embodiment of the present disclosure, the grayscale value of the first orientation region of the tumor represents the average signal intensity of the area below the tumor in the initial image, the grayscale value of the second orientation region of the tumor represents the average signal intensity of the area to the left of the tumor in the initial image, and the grayscale value of the third orientation region of the tumor represents the average signal intensity of the area to the right of the tumor in the initial image.
[0103] In one embodiment, the tumor target echo factor information PAF is as shown in formula (1):
[0104]
[0105] in, Characterize the gray value of the second position area of the tumor, Characterize the gray value of the third position area of the tumor, Represents the visual occlusion information of pixel x, Characterize the grayscale value of the first position area of the tumor.
[0106] According to an embodiment of the present disclosure, the tumor target echo factor information may be a tumor rear echo factor. Benign tumors are often accompanied by enhanced rear echoes, while malignant tumors are accompanied by attenuated rear echoes. In order to avoid the influence of potential edge shadows, the widths of the first orientation region, the second orientation region, and the third orientation region are all two-thirds of the tumor width.
[0107] According to an embodiment of the present disclosure, the height, width, area, perimeter and other information of the tumor in the tumor mask image are extracted to calculate the tumor shape information. The tumor shape information may include the tumor area, aspect ratio and circularity.
[0108] In one embodiment, the aspect ratio As shown in formula (2):
[0109]
[0110] in, Characterize tumor height information, Characterize tumor width information.
[0111] In one embodiment, the circularity R is as shown in formula (3):
[0112]
[0113] in, Characterize tumor circumference information, Characterize tumor area information.
[0114] According to an embodiment of the present disclosure, the full-area image is processed according to the grayscale co-occurrence matrix algorithm, the spatial distance is set to 2, the grayscale level is set to 256, and four sub-grayscale co-occurrence matrices of the full-area image are obtained in the four directions of 0°, 45°, 90°, and 135°, respectively. The four sub-grayscale co-occurrence matrices are then averaged to obtain the grayscale co-occurrence matrix of the full-area image. Texture information is extracted from the grayscale co-occurrence matrix of the full-area image to obtain texture information of the full-area image. Texture information may include energy information, contrast information, correlation information, and uniformity information. Therefore, four texture information of the full-area image are finally obtained.
[0115] According to the embodiment of the present disclosure, according to the above-mentioned method of extracting texture information of the whole region image, four texture information corresponding to the intratumor region image and the edge region image are extracted in sequence, and finally 12 texture information are obtained.
[0116] According to an embodiment of the present disclosure, the text feature extraction network includes a fully connected layer and an activation layer.
[0117] According to an embodiment of the present disclosure, the text feature extraction network may include two fully connected layers, each of which is connected to an activation layer. The feature output dimension of the first fully connected layer is 1024, and the feature output dimension of the second fully connected layer is 512.
[0118] According to an embodiment of the present disclosure, text information is input into a text feature extraction network, and target text features are output, including:
[0119] The text information is processed according to the fully connected layer to obtain the initial text features; the initial text features are processed according to the activation layer to obtain the target text features.
[0120] According to an embodiment of the present disclosure, the target text feature has a deeper level of text features than the initial text feature.
[0121] According to an embodiment of the present disclosure, text information is processed according to the first fully connected layer to obtain 1024 initial text features, and the 1024 initial text features are input into the activation layer connected to the first fully connected layer to obtain 1024 target text features.
[0122] According to an embodiment of the present disclosure, 1024 text information is processed according to the second fully connected layer to obtain 512 initial text features, and the 512 initial text features are input into the activation layer connected to the second fully connected layer to obtain 512 target text features.
[0123] According to the embodiments of the present disclosure, the text information integrates the tumor imaging information and clinical information based on the doctor's prior knowledge, and uses the text feature extraction network to extract the text information to obtain the target text features, thereby effectively using the doctor's professional knowledge to deepen the recognition model's learning ability for features; the clinical information is integrated into the text feature extraction network to make the information obtained by the recognition model more comprehensive, thereby improving the accuracy of model recognition.
[0124] According to an embodiment of the present disclosure, the multi-task classifier includes a main classifier and an auxiliary classifier, the main classifier is used to identify the type of the tumor, and the auxiliary classifier is used to identify the tissue morphology related to the tumor.
[0125] According to an embodiment of the present disclosure, the target image features and the target text features are fused and input into a multi-task classifier, and the classification recognition results and the tissue morphology recognition results are output, including:
[0126] The target image features and the target text features are fused to obtain the target fused features; the target fused features are subjected to dimensionality reduction operations to obtain the target reduced dimensionality features; the target reduced dimensionality features are processed according to the auxiliary classifier to obtain the tissue morphology recognition results; the tissue morphology recognition results and the target reduced dimensionality features are processed according to the main classifier to obtain the classification recognition results.
[0127] According to an embodiment of the present disclosure, a concatenation function is used to process target image features and target text features to obtain target fusion features. For example, a concatenation function is used to process 1536 target image features and 512 target text features to obtain 2048 target fusion features.
[0128] According to an embodiment of the present disclosure, a fully connected layer can be used to perform a dimensionality reduction operation on 2048 target fusion features to obtain 512 target dimensionality reduction features, and the 512 target dimensionality reduction features are input into each auxiliary classifier to obtain the tissue morphology recognition result output by each auxiliary classifier.
[0129] According to an embodiment of the present disclosure, the multi-task classifier may include a main classifier and eight auxiliary classifiers, and outputs 0 or 1 in different dimensions of each classifier, where output 1 indicates that it belongs to this category, and output 0 indicates that it does not belong to this category.
[0130] According to an embodiment of the present disclosure, according to the Breast Imaging-Reporting and Data System (BI-RADS) standard, auxiliary classifiers 1-4 are used to identify four tissue morphology results related to the tumor, namely, shape, echo pattern, posterior echo, and calcification, respectively, and auxiliary classifiers 5-8 are used to identify tissue morphology results related to the smoothness of the tumor edge.
[0131] According to an embodiment of the present disclosure, the output dimensions of the shape auxiliary classifier are 3, corresponding to the three categories of circle, ellipse and irregular shape. If the output result is (0, 1, 0), it represents the tissue morphology recognition result that the tumor belongs to the ellipse category.
[0132] According to the embodiments of the present disclosure, the output dimensions of the echo pattern auxiliary classifier are 6, corresponding to the 6 categories of no echo, high echo, low echo, equal echo, mixed echo, and uneven echo; the output dimensions of the rear echo auxiliary classifier are 4, corresponding to the 4 categories of no change, echo enhancement, acoustic shadow, equal echo, and mixed change; the output dimensions of the calcification auxiliary classifier are 3, corresponding to the 3 categories of no calcification, coarse calcification, and microcalcification.
[0133] According to an embodiment of the present disclosure, the tissue morphology result of the smoothness of the tumor edge is obtained according to the four output results of 5-8 auxiliary classifiers. 5-8 auxiliary classifiers correspond to the recognition of four features: blur, angle, microlobulation, and burr shape. 5 auxiliary classifiers are used to identify whether the tumor edge belongs to the blur category, 6 auxiliary classifiers are used to identify whether the tumor edge belongs to the angle category, 7 auxiliary classifiers are used to identify whether the tumor edge belongs to the microlobulation category, and 8 auxiliary classifiers are used to identify whether the tumor edge belongs to the burr category. If any of the four output results belong to the same category, the tissue morphology result is that the tumor edge is not smooth, otherwise the tissue morphology result is that the tumor edge is smooth.
[0134] For example, the output result of the 5-8 auxiliary classifiers is (0, 1, 0, 0), and the tissue morphology result is that the tumor edge is not smooth; the output result of the 5-8 auxiliary classifiers is (0, 0, 0, 0), and the tissue morphology result is that the tumor edge is smooth.
[0135] According to an embodiment of the present disclosure, the main classifier may be composed of two fully connected layers, each of which is connected to an activation layer. The output dimension of the first fully connected layer is 256, connected to the first activation layer, and the first activation layer may be constructed based on the linear rectified function (Rectified Linear Unit, ReLU); the output dimension of the second fully connected layer is 1, connected to the second activation layer, and the second activation layer may be constructed based on the activation function (Sigmoid). The tissue morphology recognition result and the target dimensionality reduction feature are input into the main classifier together to obtain the classification recognition result.
[0136] According to an embodiment of the present disclosure, the classification recognition result is 1, which indicates that the tumor is malignant, and 0, which indicates that the tumor is benign.
[0137] According to an embodiment of the present disclosure, multiple auxiliary classifiers are used to identify different tissue morphology results of tumors. In the process of the main classifier processing multiple tissue morphology results and target dimensionality reduction features, the tissue morphology results can guide the recognition model to learn the features that clinical experts are concerned about, thereby improving the model recognition accuracy; in addition, different tissue morphology results can provide explanations for the final classification and recognition results, thereby better assisting expert judgment and improving the efficiency and accuracy of expert diagnosis.
[0138] According to an embodiment of the present disclosure, the auxiliary classifier includes a hidden sublayer, a fully connected sublayer, and an activation sublayer.
[0139] According to an embodiment of the present disclosure, the target dimension reduction features are processed by the auxiliary classifier to obtain a tissue morphology recognition result, including:
[0140] The target dimension reduction features are processed according to the hidden sub-layer to obtain the hidden features; the hidden features are processed according to the fully connected sub-layer to obtain the tissue morphology features; the tissue morphology features are processed according to the activated sub-layer to obtain the tissue morphology recognition results.
[0141] According to an embodiment of the present disclosure, the hidden sublayer may be constructed based on a random dropout network, and the activation sublayer may be constructed based on a rectified linear unit (ReLU). The input dimension of the fully connected sublayer may be 512.
[0142] According to an embodiment of the present disclosure, each auxiliary classifier includes a hidden sublayer, a fully connected sublayer and an activation sublayer, and the target dimensionality reduction features are input into each auxiliary classifier to obtain tissue morphology recognition results corresponding to each of the multiple auxiliary classifiers.
[0143] Figure 3B An example schematic diagram of a recognition model according to an embodiment of the present disclosure is shown.
[0144] like Figure 3B As shown, the recognition model includes a feature extraction network and a multi-task classifier, the feature extraction network includes an image feature extraction network and a text feature extraction network, the multi-task classifier includes auxiliary classifiers 1, ..., auxiliary classifier N and a main classifier, and the image feature extraction network includes three independent convolutional neural network layers and a channel attention layer. According to the initial image, the full-region image, the intratumoral region image, the edge region image, and the tumor imaging information are obtained, and the features of the full-region image, the intratumoral region image, and the edge region image are extracted according to the three convolutional neural network layers, and the full-region image features, the intratumoral region image features, and the edge region image features are output, and the full-region image features, the intratumoral region image features, and the edge region image features are input into the channel attention layer, and the fused image features are output, and then the target image features are obtained through the fully connected layer.
[0145] According to an embodiment of the present disclosure, tumor imaging information includes tumor target echo factor information, tumor shape information and texture information. The tumor imaging information and clinical information are fused to determine text information, which is input into a text feature extraction network to output target text features.
[0146] According to the embodiments of the present disclosure, the target image features and the target text features are fused to obtain the target fused features; the target fused features are subjected to dimensionality reduction operations to obtain the target reduced dimensionality features; the target reduced dimensionality features are input into each auxiliary classifier to generate the tissue morphology recognition results corresponding to the auxiliary classifier; and the multiple tissue morphology recognition results and the target reduced dimensionality features are processed according to the main classifier to obtain the classification recognition results.
[0147] Figure 4A flowchart of a method for training a recognition model for identifying tumor features in ultrasound images according to an embodiment of the present disclosure is shown.
[0148] like Figure 4 As shown, the method 400 includes operations S410 to S460.
[0149] In operation S410 , a training sample is acquired, where the training sample includes a sample initial image, sample clinical information, a classification label, and a tissue morphology label related to the tumor sample.
[0150] In operation S420, the sample initial image is input into an image feature extraction network, and sample target image features are output.
[0151] In operation S430, sample text information is determined according to the sample tumor image information and the sample clinical information.
[0152] In operation S440, the sample text information is input into a text feature extraction network, and sample target text features are output.
[0153] In operation S450, the sample target image features and the sample target text features are fused and input into a multi-task classifier, and a sample classification recognition result and a sample tissue morphology recognition result are output.
[0154] In operation S460, a recognition model is trained according to the sample classification recognition result, the sample tissue morphology recognition result, the classification label and the tissue morphology label to obtain a trained recognition model.
[0155] According to an embodiment of the present disclosure, the recognition model includes an image feature extraction network, a text feature extraction network and a multi-task classifier.
[0156] According to an embodiment of the present disclosure, the training sample includes a sample initial image, which represents an original image of a breast of a tumor patient collected using an ultrasound device. The sample clinical information represents clinical manifestation information of the tumor patient, such as the patient's age information.
[0157] According to an embodiment of the present disclosure, the classification label represents the actual development degree of the tumor in the patient's breast, for example, benign or malignant. The tissue morphology label represents the ultrasound morphology category related to the tumor tissue, for example, tumor shape category, echo pattern category, calcification category, tumor edge category, etc.
[0158] According to an embodiment of the present disclosure, training samples are collected from the database of Hospital A, which stores patient breast data from 2019 to 2022. The training samples include sample initial images, sample clinical information, classification labels, and tissue morphology labels for each tumor patient. There are 12,039 sample initial images in total, of which 7,931 sample initial images of 2,085 patients have corresponding classification labels of benign class, and 4,108 sample initial images of 1,199 patients have corresponding classification labels of malignant class.
[0159] According to an embodiment of the present disclosure, the sample tumor image information is obtained based on the sample initial image. For example, information is extracted from the sample initial image to obtain the sample tumor image information.
[0160] According to the embodiments of the present disclosure, the sample tumor image information represents the ultrasound detection information related to the tumor tissue in the sample initial image. The sample clinical information is important reference information for experts to diagnose benign and malignant tumors.
[0161] According to an embodiment of the present disclosure, sample target image features and sample target text features are fused, and the fused features are input into a multi-task classifier to output sample classification recognition results corresponding to the tumor classification task and sample tissue morphology recognition results corresponding to the tissue morphology recognition task.
[0162] According to the embodiments of the present disclosure, the performance of the model can be evaluated based on the sample classification and recognition results, the sample tissue morphology recognition results, the classification labels and the tissue morphology labels, and then the parameters of the recognition model can be adjusted, and the recognition model can be iteratively trained. When the performance meets a preset performance threshold, the training is stopped.
[0163] According to an embodiment of the present disclosure, a recognition model is trained based on sample classification recognition results, sample tissue morphology recognition results, classification labels, and tissue morphology labels to obtain a trained recognition model, including:
[0164] The sample tissue morphology recognition results and tissue morphology labels are processed according to the first loss function to obtain a tissue morphology loss value; the sample classification recognition results and classification labels are processed according to the second loss function to obtain a classification loss value; the recognition model is trained according to the tissue morphology loss value and the classification loss value to obtain a trained recognition model.
[0165] According to an embodiment of the present disclosure, the validation sample includes 787 sample initial images, and the corresponding classification labels are benign; 427 sample initial images, and the corresponding classification labels are malignant. The recognition model is trained using the PyTorch 1.7 framework. During the training process, the adaptive moment estimation (ADAM) optimizer is used for optimization, the initial learning rate is 0.0001, and the number of iterations is set to 32. The data augmentation strategies of horizontal flipping, blurring, random contrast, and random rotation (-10°-10°) are used to avoid overfitting. If the working curve (ROC) of the validation set does not improve within 15 rounds, or reaches the limit of 50 rounds, the training is terminated, and the trained model is finally obtained.
[0166] According to an embodiment of the present disclosure, when there are multiple auxiliary classifiers, the sample tissue morphology recognition results of each auxiliary classifier will be obtained. For each auxiliary classifier, the sample tissue morphology recognition results and tissue morphology labels are processed according to the first loss function to obtain the tissue morphology loss values corresponding to each auxiliary classifier.
[0167] According to an embodiment of the present disclosure, different weights are set for multiple tissue morphology loss values and classification loss values according to the different impact values of tissue morphology characteristics on sample classification and recognition results, and a target loss value is obtained, thereby training a recognition model.
[0168] In one embodiment, the target loss value As shown in formula (4):
[0169]
[0170] in, Characterizes the tissue morphology loss value of the shape auxiliary classifier, Characterizes the tissue morphology loss value of the echo pattern auxiliary classifier, Characterizes the tissue morphology loss value of the rear echo auxiliary classifier, Characterizes the tissue morphology loss value of the calcification auxiliary classifier, Characterizes the tissue morphology loss value of the fuzzy auxiliary classifier, Characterizes the tissue morphology loss value of the angular auxiliary classifier, Characterizes the tissue morphology loss value of the micro-lobed auxiliary classifier, Characterizes the tissue morphology loss value of the burr classifier, Characterizes the classification loss value of the main and auxiliary classifiers.
[0171] According to the embodiment of the present disclosure, the test data is input into the trained recognition model to obtain the classification recognition result and the tissue morphology recognition result. The classification recognition result represents the classification result of the predicted tumor, and the tissue morphology recognition result represents the tissue morphology result of the predicted tumor in terms of ultrasound manifestation.
[0172] Table 1 schematically shows a performance comparison table of classification and recognition results obtained by the method according to an embodiment of the present disclosure and the existing method.
[0173]
[0174] According to the disclosed embodiment, the convolutional neural network model used in the existing method is obtained by training based on the initial image, and the recognition model used in the method of the present application is obtained by training based on the initial image and text information. According to Table 1, the present application has achieved the highest performance in the four evaluation indicators (accuracy, sensitivity, specificity, AUC value), where the AUC value represents the area value under the receiver operating characteristic curve (ROC).
[0175] Figure 5 An example schematic diagram of the recognition classification result and the tissue morphology recognition result according to an embodiment of the present disclosure is shown.
[0176] like Figure 5 As shown in the figure, 5a represents the initial image of a benign tumor, which is identified according to the recognition model to obtain the classification recognition results and tissue morphology recognition results. The classification recognition results include a malignant probability of 0%, and the tissue morphology recognition results include: the shape result is elliptical, the edge result is smooth, the rebound mode result is no echo, the rear echo result is enhanced, and the calcification result is no calcification; 5b represents the initial image of a malignant tumor, which is identified according to the recognition model to obtain the classification recognition results and tissue morphology recognition results. The classification recognition results include a malignant probability of 89%, and the tissue morphology recognition results include: the shape result is irregular, the edge result is blurred and angular, the rebound mode result is low echo, the rear echo result is acoustic shadow, and the calcification result is no calcification.
[0177] According to the embodiment of the present disclosure, the classification recognition result and tissue morphology recognition result of the tumor are predicted based on the recognition model. The tissue morphology recognition result can provide a good explanation for the predicted malignancy probability and can also provide a reference for experts, thereby improving the efficiency and accuracy of doctors' diagnosis.
[0178] Based on the above-mentioned method for identifying tumor features in ultrasound images based on information fusion and multi-task learning, the present disclosure also provides an identification device. Figure 6 The device is described in detail.
[0179] Figure 6A structural block diagram of a device for identifying tumor features in ultrasound images based on information fusion and multi-task learning according to an embodiment of the present disclosure is shown.
[0180] like Figure 6 As shown, the recognition device 600 of this embodiment includes a first acquisition module 610 , a first extraction module 620 , a first determination module 630 , a second extraction module 640 and a first recognition module 650 .
[0181] The first acquisition module 610 is used to acquire the initial image and clinical information related to the tumor. In one embodiment, the first acquisition module 610 can be used to perform the operation S110 described above, which will not be described in detail here.
[0182] The first extraction module 620 is used to input the initial image into the image feature extraction network and output the target image features. In one embodiment, the first extraction module 620 can be used to perform the operation S120 described above, which will not be described in detail here.
[0183] The first determination module 630 is used to determine text information according to the tumor image information and the clinical information. In one embodiment, the first determination module 630 can be used to perform the operation S130 described above, which will not be described in detail here.
[0184] The second extraction module 640 is used to input the text information into the text feature extraction network and output the target text features. In one embodiment, the second extraction module 640 can be used to perform the operation S140 described above, which will not be described in detail here.
[0185] The first recognition module 650 is used to fuse the target image features and the target text features into a multi-task classifier, and output classification recognition results and tissue morphology recognition results. In one embodiment, the first recognition module 650 can be used to perform the operation S150 described above, which will not be repeated here.
[0186] According to an embodiment of the present disclosure, the first extraction module 620 includes a first generation submodule, a second generation submodule, a third generation submodule, a first extraction submodule, a second extraction submodule, and a third extraction submodule.
[0187] The first generation submodule is used to crop and expand the initial image to obtain a full-area image.
[0188] The second generation submodule is used to perform pixel multiplication operation on the initial image and the tumor mask image to obtain an intra-tumor area image, wherein the tumor mask image is obtained based on segmenting the initial image.
[0189] The third generation submodule is used to perform pixel multiplication operation on the initial image and the edge mask image to obtain an edge area image, wherein the edge mask image is obtained based on edge detection operation and morphological operation on the tumor mask image.
[0190] The first extraction submodule is used to extract features from the full-area image, the intratumoral area image and the edge area image respectively, and output features of the full-area image, features of the intratumoral area image and features of the edge area image.
[0191] The second extraction submodule is used to input the full-area image features, the intratumoral area image features and the edge area image features into the channel attention layer, and output the fused image features.
[0192] The third extraction submodule is used to input the fused image features into the fully connected layer and output the target image features.
[0193] According to an embodiment of the present disclosure, the second extraction submodule includes a first extraction unit, a second extraction unit, a third extraction unit, and a fourth extraction unit.
[0194] The first extraction unit is used to process the full-region image features, the intratumor region image features and the edge region image features according to the average pooling sublayer to obtain image pooling features.
[0195] The second extraction unit is used to process the image pooling features according to the convolution sublayer to obtain the image convolution features.
[0196] The third extraction unit is used to process the image convolution feature according to the activation sublayer to obtain the image activation feature.
[0197] The fourth extraction unit is used to perform pixel multiplication operation on the full-area image features, the intratumoral area image features, the edge area image features and the image activation features to obtain fused image features.
[0198] According to an embodiment of the present disclosure, the determination module 630 includes a first determination submodule, a second determination submodule, a third determination submodule, a fourth determination submodule, and a fifth determination submodule.
[0199] The first determination submodule is used to obtain the tumor target echo factor information according to the grayscale value of the tumor first orientation area, the grayscale value of the tumor second orientation area and the grayscale value of the tumor third orientation area of the initial image.
[0200] The second determination submodule is used to determine tumor shape information according to the tumor mask image.
[0201] The third determination submodule is used to process the full-area image, the intratumoral area image and the edge area image respectively according to the gray level co-occurrence matrix algorithm to obtain gray level co-occurrence matrices corresponding to the full-area image, the intratumoral area image and the edge area image respectively.
[0202] The fourth determination submodule is used to extract the gray level co-occurrence matrix corresponding to the full area image, the intratumor area image and the edge area image, and obtain the texture information corresponding to the full area image, the intratumor area image and the edge area image.
[0203] The fifth determination submodule is used to obtain tumor image information according to the tumor target echo factor information, tumor shape information and texture information.
[0204] According to an embodiment of the present disclosure, the second extraction module 640 includes a fourth extraction submodule and a fifth extraction submodule.
[0205] The fourth extraction submodule is used to process the text information according to the fully connected layer to obtain initial text features.
[0206] The fifth extraction submodule is used to process the initial text features according to the activation layer to obtain the target text features.
[0207] According to an embodiment of the present disclosure, the first identification module 650 includes a first identification submodule, a second identification submodule, a third identification submodule, and a fourth identification submodule.
[0208] The first recognition submodule is used to fuse the target image features and the target text features to obtain the target fusion features.
[0209] The second recognition submodule is used to perform a dimensionality reduction operation on the target fusion feature to obtain the target dimensionality reduction feature.
[0210] The third recognition submodule is used to process the target dimension reduction features according to the auxiliary classifier to obtain the tissue morphology recognition result.
[0211] The fourth recognition submodule is used to process the tissue morphology recognition result and the target dimension reduction feature according to the main classifier to obtain a classification recognition result.
[0212] According to an embodiment of the present disclosure, the third identification submodule includes a first identification unit, a second identification unit and a third identification unit.
[0213] The first recognition unit is used to process the target dimension reduction feature according to the hidden sub-layer to obtain the hidden feature.
[0214] The second recognition unit is used to process the hidden features according to the fully connected sub-layer to obtain the tissue morphology features.
[0215] The third recognition unit is used to process the tissue morphology feature according to the activated sublayer to obtain a tissue morphology recognition result.
[0216] Based on the above-mentioned training method for a recognition model for identifying tumor features in ultrasound images, the present disclosure also provides a training device for a recognition model for identifying tumor features in ultrasound images. Figure 7 The device is described in detail.
[0217] Figure 7 A structural block diagram of a training device for a recognition model for identifying tumor features in ultrasound images according to an embodiment of the present disclosure is shown.
[0218] like Figure 7 As shown, the training device 700 of the recognition model for identifying tumor features in ultrasound images of this embodiment includes a second acquisition module 710, a third extraction module 720, a second determination module 730, a fourth extraction module 740, a second recognition module 750 and a training module 760.
[0219] The second acquisition module 710 is used to acquire training samples, which include sample initial images, sample clinical information, classification labels, and tissue morphology labels related to the tumor sample. In one embodiment, the second acquisition module 710 can be used to perform the operation S410 described above, which will not be repeated here.
[0220] The third extraction module 720 is used to input the sample initial image into the image feature extraction network and output the sample target image features. In one embodiment, the third extraction module 720 can be used to perform the operation S420 described above, which will not be described in detail here.
[0221] The second determination module 730 is used to determine the sample text information according to the sample tumor image information and the sample clinical information. In one embodiment, the second determination module 730 can be used to perform the operation S430 described above, which will not be described in detail here.
[0222] The fourth extraction module 740 is used to input the sample text information into the text feature extraction network and output the sample target text features. In one embodiment, the fourth extraction module 740 can be used to perform the operation S440 described above, which will not be described in detail here.
[0223] The second recognition module 750 is used to fuse the sample target image features and the sample target text features into a multi-task classifier, and output the sample classification recognition result and the sample tissue morphology recognition result. In one embodiment, the second recognition module 750 can be used to perform the operation S450 described above, which will not be repeated here.
[0224] The training module 760 is used to train the recognition model according to the sample classification recognition result, the sample tissue morphology recognition result, the classification label and the tissue morphology label to obtain the trained recognition model. In one embodiment, the training module 760 can be used to perform the operation S460 described above, which will not be repeated here.
[0225] According to an embodiment of the present disclosure, the training module 760 includes a first training submodule, a second training submodule and a third training submodule.
[0226] The first training submodule is used to process the sample tissue morphology recognition result and the tissue morphology label according to the first loss function to obtain the tissue morphology loss value.
[0227] The second training submodule is used to process the sample classification recognition results and classification labels according to the second loss function to obtain the classification loss value.
[0228] The third training submodule is used to train the recognition model according to the tissue morphology loss value and the classification loss value to obtain the trained recognition model.
[0229] According to an embodiment of the present disclosure, any multiple modules among the modules, submodules, units, and subunits can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.
[0230] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0231] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A method for identifying tumor features in ultrasound images based on information fusion and multi-task learning. It is characterized in that The identification method is performed by an electronic device, and the method includes: Obtain initial images and clinical information related to the tumor; Inputting the initial image into an image feature extraction network and outputting target image features; Determining text information according to tumor image information and the clinical information, wherein the tumor image information is obtained based on the initial image; Inputting the text information into a text feature extraction network and outputting target text features; The target image features and the target text features are fused and input into a multi-task classifier, and a classification recognition result and a tissue morphology recognition result are output.
2. The method according to claim 1, It is characterized in that The image feature extraction network includes a channel attention layer and a fully connected layer; The initial image is input into the image feature extraction network, and the target image features are output, including: Cropping and expanding the initial image to obtain a full-area image; Performing pixel multiplication operation on the initial image and the tumor mask image to obtain an intratumoral region image, wherein the tumor mask image is obtained based on segmenting the initial image; Performing a pixel multiplication operation on the initial image and the edge mask image to obtain an edge region image, wherein the edge mask image is obtained based on performing an edge detection operation and a morphological operation on the tumor mask image; Extracting features from the full-area image, the intratumoral area image and the edge area image respectively, and outputting full-area image features, intratumoral area image features and edge area image features; Inputting the full-area image features, the intratumoral area image features and the edge area image features into the channel attention layer, and outputting fused image features; The fused image features are input into the fully connected layer, and the target image features are output.
3. The method according to claim 2, It is characterized in that The channel attention layer includes an average pooling sublayer, a convolution sublayer and an activation sublayer; The whole-region image features, the intratumor region image features and the edge region image features are input into the channel attention layer, and the fused image features are output, including: Processing the full-region image features, the intra-tumor region image features and the edge region image features according to the average pooling sublayer to obtain image pooling features; Processing the image pooling features according to the convolution sublayer to obtain image convolution features; Processing the image convolution feature according to the activation sublayer to obtain an image activation feature; The fused image feature is obtained by performing pixel multiplication operation on the full-area image feature, the intratumoral area image feature, the edge area image feature and the image activation feature.
4. The method according to claim 2, It is characterized in that Based on the initial image, obtaining the tumor image information includes: Obtaining tumor target echo factor information according to the grayscale value of the tumor first orientation region, the grayscale value of the tumor second orientation region, and the grayscale value of the tumor third orientation region of the initial image; determining tumor shape information according to the tumor mask image; Process the full-area image, the intratumoral area image and the edge area image respectively according to a gray-level co-occurrence matrix algorithm to obtain gray-level co-occurrence matrices corresponding to the full-area image, the intratumoral area image and the edge area image respectively; Extracting gray-level co-occurrence matrices corresponding to the full-area image, the intra-tumor area image, and the edge area image, respectively, to obtain texture information corresponding to the full-area image, the intra-tumor area image, and the edge area image, respectively; The tumor image information is obtained according to the tumor target echo factor information, the tumor shape information and the texture information.
5. The method according to claim 1, It is characterized in that The text feature extraction network includes a fully connected layer and an activation layer; The text information is input into a text feature extraction network, and the target text features are output, including: Processing the text information according to the fully connected layer to obtain initial text features; The initial text features are processed according to the activation layer to obtain the target text features.
6. The method according to claim 1, It is characterized in that The multi-task classifier includes a main classifier and an auxiliary classifier, wherein the main classifier is used to identify the type of the tumor, and the auxiliary classifier is used to identify the tissue morphology related to the tumor; The target image features and the target text features are fused and input into a multi-task classifier, and the classification recognition results and the tissue morphology recognition results are output, including: Fusing the target image feature and the target text feature to obtain a target fused feature; Performing a dimensionality reduction operation on the target fusion feature to obtain a target dimensionality reduction feature; Processing the target dimension reduction feature according to the auxiliary classifier to obtain the tissue morphology recognition result; The main classifier processes the tissue morphology recognition result and the target dimension reduction feature to obtain the classification recognition result.
7. The method according to claim 6, It is characterized in that The auxiliary classifier includes a hidden sublayer, a fully connected sublayer and an activation sublayer; Wherein, the target dimension reduction feature is processed by the auxiliary classifier to obtain the tissue morphology recognition result, including: Processing the target dimension reduction feature according to the hidden sublayer to obtain a hidden feature; Processing the hidden features according to the fully connected sublayer to obtain tissue morphological features; The tissue morphology feature is processed according to the activation sublayer to obtain the tissue morphology recognition result.
8. A method for training a recognition model for identifying tumor features in ultrasound images, It is characterized in that The recognition model includes an image feature extraction network, a text feature extraction network and a multi-task classifier; the training method includes: Acquire a training sample, wherein the training sample includes an initial sample image, sample clinical information, a classification label, and a tissue morphology label related to the tumor sample, wherein the classification label represents a benign or malignant pathological result of the tumor sample; Inputting the sample initial image into the image feature extraction network, and outputting sample target image features; Determining sample text information according to sample tumor image information and the sample clinical information, wherein the sample tumor image information is obtained based on the sample initial image; Inputting the sample text information into the text feature extraction network and outputting sample target text features; The sample target image features and the sample target text features are fused and input into the multi-task classifier, and the sample classification recognition result and the sample tissue morphology recognition result are output; A recognition model is trained according to the sample classification recognition result, the sample tissue morphology recognition result, the classification label and the tissue morphology label to obtain a trained recognition model.
9. The method according to claim 8, It is characterized in that According to the sample classification recognition result, the sample tissue morphology recognition result, the classification label and the tissue morphology label, a recognition model is trained to obtain a trained recognition model, including: Processing the sample tissue morphology recognition result and the tissue morphology label according to a first loss function to obtain a tissue morphology loss value; Processing the sample classification recognition result and the classification label according to a second loss function to obtain a classification loss value; According to the tissue morphology loss value and the classification loss value, a recognition model is trained to obtain a trained recognition model.
10. A device for tumor feature recognition in ultrasound images based on information fusion and multi-task learning. It is characterized in that include: A first acquisition module is used to acquire initial images and clinical information related to the tumor; A first extraction module, used for inputting the initial image into an image feature extraction network and outputting target image features; A first determination module is used to determine text information according to tumor image information and the clinical information, wherein the tumor image information is obtained based on the initial image; A second extraction module, used for inputting the text information into a text feature extraction network and outputting target text features; The first recognition module is used to fuse the target image features and the target text features into a multi-task classifier, and output a classification recognition result and a tissue morphology recognition result.
Citation Information
Patent Citations
Breast ultrasonic tumor recognition method based on deep learning
CN110264462A
Tumor image focus area prediction analysis method and system and terminal equipment
CN112801168A
Tumor classification method and system based on multi-modal fusion
CN115019405A
Multi-modal data fusion-based rectal cancer prediction method, system and equipment
CN117422964A
Lesion detection model training method, and method for recognizing lesion in image
WO2023015743A1
