Image classification method, storage medium, and program product

Abstract

The application relates to an image classification method, a storage medium and a program product. The method comprises the following steps: according to obtained medical images of a to-be-tested part in different body positions, determining first target segmentation images and second target segmentation images corresponding to lesion regions in the medical images in different body positions; the lesion region included in the first target segmentation image is a first-shape lesion region, and the lesion region included in the second target segmentation image is a second-shape lesion region; according to the first target segmentation images and the second target segmentation images in different body positions and a preset neural network model, the class of the lesion region is identified, and a target class of the lesion region is determined. By adopting the method, manpower cost and classification time can be saved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image classification method, storage medium, and program product. Background Technology

[0002] With the increasing prevalence of breast diseases among women, contemporary women are paying significantly more attention to their breast health. Currently, many women regularly visit hospitals for breast examinations to allow for early intervention if any breast problems are detected.

[0003] In related technologies, when patients go to the hospital for breast examinations, they usually first take some breast images. Then, based on their experience, doctors delineate the lesions in the patient's breast images and repeatedly compare and classify them with the existing standard breast signs to finally determine the category of the lesions in the patient's breast images.

[0004] However, the above method of classifying lesions in breast images is time-consuming and labor-intensive. Summary of the Invention

[0005] Therefore, it is necessary to provide an image classification method, storage medium, and program product that can save labor costs and classification time in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides an image classification method, the method comprising:

[0007] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion area in the medical images in each body position are determined; the lesion area included in the first target segmentation image is a lesion area of ​​a first shape, and the lesion area included in the second target segmentation image is a lesion area of ​​a second shape.

[0008] Based on the first and second target segmentation images under each body position, as well as the preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0009] In one embodiment, the aforementioned neural network model includes a first classification network and a second classification network; the above-mentioned identification of the lesion region category based on the first target segmentation image and the second target segmentation image under each body position, and the preset neural network model, to determine the target category of the lesion region, includes:

[0010] The first and second target segmentation images under each body position are input into the first classification network for classification, and the feature map and initial category corresponding to the lesion region in each target segmentation image are determined.

[0011] Based on the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, and the second classification network, the target category of the lesion region is determined.

[0012] In one embodiment, determining the target category of the lesion region based on the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network, includes:

[0013] Based on the first and second target segmentation images in each body position, the quantitative features corresponding to the lesion regions in each target segmentation image are determined; the above quantitative features are used to characterize the distribution of the lesion regions.

[0014] Based on the quantized features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, and the second classification network, the target category of the lesion region is determined.

[0015] In one embodiment, determining the target category of the lesion region based on the quantized features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network includes:

[0016] Obtain clinical characteristic information of the subjects to be tested;

[0017] Based on clinical feature information, quantitative features corresponding to lesion regions in each target segmentation image, feature maps and initial categories corresponding to lesion regions in each target segmentation image, and the second classification network, the target category of the lesion region is determined.

[0018] In one embodiment, determining the target category of the lesion region based on clinical feature information, the quantized features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network includes:

[0019] After fusing the clinical feature information, the quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps corresponding to the lesion regions in each target segmentation image, and the initial category, the data is input into the second classification network to determine the target category of the lesion region.

[0020] The second classification network is trained based on the sample feature information set corresponding to multiple sample objects. The sample feature information of each sample object includes sample clinical feature information, sample quantitative features, sample feature map, sample initial category, and lesion region annotation category.

[0021] In one embodiment, the above-mentioned determination of the first target segmentation image and the second target segmentation image corresponding to the lesion region in the medical images of the test site under different body positions based on the acquired medical images of the test site under different body positions includes:

[0022] Based on the preset first segmentation model and second segmentation model, the lesion region in the medical images of the test site under different body positions is segmented, and the first target segmentation image and second target segmentation image corresponding to the medical images under each body position are determined.

[0023] The first segmentation model is trained based on multiple first sample medical images, each of which is labeled with a lesion region of a first shape; the second segmentation model is trained based on multiple second sample medical images, each of which is labeled with a lesion region of a second shape.

[0024] In one embodiment, the first classification network is a classification network employing an attention mechanism.

[0025] In one embodiment, the site to be tested is the breast, and the different body positions include the CC axial view and the MLO internal oblique view.

[0026] Secondly, this application also provides an image classification apparatus, which includes:

[0027] The determination module is used to determine, based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion region in the medical images in each body position; the lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape.

[0028] The classification module is used to identify the category of the lesion region based on the first and second target segmentation images under each body position and the preset neural network model, and to determine the target category of the lesion region.

[0029] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0030] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion area in the medical images in each body position are determined; the lesion area included in the first target segmentation image is a lesion area of ​​a first shape, and the lesion area included in the second target segmentation image is a lesion area of ​​a second shape.

[0031] Based on the first and second target segmentation images under each body position, as well as the preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0033] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion area in the medical images in each body position are determined; the lesion area included in the first target segmentation image is a lesion area of ​​a first shape, and the lesion area included in the second target segmentation image is a lesion area of ​​a second shape.

[0034] Based on the first and second target segmentation images under each body position, as well as the preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0035] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0036] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion area in the medical images in each body position are determined; the lesion area included in the first target segmentation image is a lesion area of ​​a first shape, and the lesion area included in the second target segmentation image is a lesion area of ​​a second shape.

[0037] Based on the first and second target segmentation images under each body position, as well as the preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0038] The aforementioned image classification method, storage medium, and program product determine the first and second target segmentation images corresponding to the lesion regions in medical images of the test site under different body positions. Based on the first and second target segmentation images under each body position and a pre-set neural network model, the lesion region category is identified to determine the target category of the lesion region. Specifically, the lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape. Here, because the lesion region category can be identified through a pre-set neural network without requiring manual classification based on experience, labor costs and classification time can be saved. Simultaneously, the high error rate caused by manual classification can be avoided, improving the accuracy of the classification results. Furthermore, since this method combines information on two different lesion types—second lesions and first-shape lesions—both of which are present in images from multiple body positions, combining images from different body positions and lesions with different annotation types to classify the lesion region provides more and richer information, resulting in more accurate classification results. Attached Figure Description

[0039] Figure 1 This is a diagram illustrating the application environment of an image classification method in one embodiment;

[0040] Figure 2 This is a flowchart illustrating an image classification method in one embodiment;

[0041] Figure 3 This is a flowchart illustrating an image classification method in another embodiment;

[0042] Figure 4 This is an example diagram illustrating classification using a first classification network in another embodiment;

[0043] Figure 5 This is a flowchart illustrating an image classification method in another embodiment;

[0044] Figure 6 This is an example image showing the quantitative characteristics of the lesion region obtained in another embodiment;

[0045] Figure 7 This is a flowchart illustrating an image classification method in another embodiment;

[0046] Figure 8 This is a detailed structural example of classifying lesion areas in another embodiment;

[0047] Figure 9 This is a structural example diagram illustrating the segmentation of the lesion region in another embodiment;

[0048] Figure 10 This is a structural block diagram of an image classification device in one embodiment;

[0049] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] Currently, the assessment of breast images is primarily done manually. Doctors typically refer to existing breast imaging reporting and data systems (BI-RADS reporting system) to generate reports. This system involves the judgment and classification of numerous imaging features, resulting in a massive workload and potential for diagnostic discrepancies among doctors. Furthermore, when BI-RADS levels reach 4A, 4B, 4C, or 5, a pathological examination of the breast may be necessary to determine if a malignant lesion is present. Therefore, it is evident that existing methods for classifying lesions in breast images are time-consuming and labor-intensive. Consequently, this application provides an image classification method, storage medium, and program product to address the aforementioned technical problems.

[0052] The image classification method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, scanning device 102 is connected to and communicates with computer device 104. Scanning device 102 can transmit the data obtained after scanning the object to be tested to computer device 104 for processing. A data storage system can store the data that computer device 104 needs to process. The data storage system can be integrated into computer device 104 or placed in the cloud or on other network servers. Scanning device 102 can be used when the object to be tested is standing, lying down, or any other type of scanning device. Computer device 104 can be a terminal or a server; if it is a terminal, it can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.; if it is a server, it can be a standalone server or a server cluster composed of multiple servers. Furthermore, scanning device 102 and computer device 104 can be integrated into one device or they can be two separate devices.

[0053] In one embodiment, such as Figure 2 As shown, an image classification method is provided, which can be applied to... Figure 1 Taking a computer device as an example, the method may include the following steps:

[0054] S202, based on the acquired medical images of the test site in different body positions, determine the first target segmentation image and the second target segmentation image corresponding to the lesion area in the medical images in each body position.

[0055] Optionally, the area to be tested is the breast region. This breast region can be any one breast or both breasts of the subject. The lesion area can be a tumor, calcification, or mass in the breast region. The different body positions mentioned above include the CC axial view and the MLO endoscopic view, and of course, other positions can also be included. The medical images acquired in each position can be two-dimensional or three-dimensional images, etc. Obtaining medical images in different positions means obtaining medical images from different views.

[0056] Specifically, the part of the subject to be tested can be placed in the scanning device according to a certain body position requirement, and then the scanning device can be used to scan the part of the subject to be tested to obtain a medical image under that body position. Then the part of the subject to be tested can be placed in the scanning device according to other body position requirements and scanned to obtain a medical image under that body position. By performing this process, medical images of the part of the subject to be tested under various body positions can be obtained.

[0057] Subsequently, segmentation models, algorithms, or manual segmentation can be used to segment the medical images in each body position, obtaining two target segmented images corresponding to each position. These two target segmented images are denoted as the first target segmented image and the second target segmented image, respectively. The lesion regions included in the first target segmented image are of a first shape, meaning the lesion regions in the segmented image are cloud-like. The lesion regions included in the second target segmented image are of a second shape, meaning the lesion regions in the segmented image are star-shaped, i.e., each lesion region is a point-like, individually distributed lesion, not connected into a patch.

[0058] Typically, the first-shaped and second-shaped lesion regions can be distinguished using a threshold, such as area or volume. For example, lesion regions with an area greater than the threshold are classified as first-shaped lesion regions, while those with an area less than or equal to the threshold are classified as second-shaped lesion regions. Generally, first-shaped lesion regions are larger than second-shaped lesion regions. The size relationship of the lesion regions can be measured by area or volume; for instance, the area of ​​a first-shaped lesion region is greater than the area of ​​a second-shaped lesion region. Furthermore, in some cases, a first-shaped lesion region can also be composed of multiple second-shaped lesion regions connected together.

[0059] It should be noted that the lesion regions targeted in the first and second target segmentation images are the same lesions in medical images, but they exist in different forms in each target segmentation image.

[0060] S204, based on the first target segmentation image and the second target segmentation image under each body position, as well as the preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0061] In this step, the neural network model can be a model composed of classification networks, which may include one classification network, multiple classification networks, or other networks, such as segmentation networks, feature extraction networks, etc.

[0062] After obtaining the first and second target segmentation images under each body position, the two target segmentation images under each body position can be combined and input into a neural network model to classify the lesion region and obtain the target category of the lesion region. Alternatively, relevant feature information of the lesion region can be obtained from the two target segmentation images under each body position, and the obtained feature information can be fused and input into a neural network model to classify the lesion region and obtain the target category of the lesion region. Of course, it is also possible to combine other data and input them into a neural network model to classify the lesion region and obtain the target category of the lesion region. There are no specific limitations here, as long as the target category of the lesion region can be obtained.

[0063] In addition, when determining the target category of the lesion area, one possible implementation method is to output the probability of the lesion area belonging to each category through a neural network model, and select the category with the highest probability from the probabilities of each category as the target category of the lesion area.

[0064] As described above, the shape and other features of the lesion area in medical images of the test site under different body positions will be significantly different, and the characteristics of the lesion can be represented by different shapes of lesions. Therefore, different body positions and different shapes of lesions are used to identify the category of lesions. This combination of feature information is richer, and there are more relevant information to refer to during classification, so the classification results are more accurate.

[0065] In the aforementioned image classification method, medical images of the test site in different body positions are used to determine the first and second target segmentation images corresponding to the lesion regions in each body position. Based on the first and second target segmentation images in each body position, and a pre-set neural network model, the category of the lesion region is identified to determine its target category. Specifically, the lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape. Here, because the lesion region category can be identified through a pre-set neural network without requiring manual classification based on experience, labor costs and classification time can be saved. Simultaneously, the high error rate caused by manual classification can be avoided, improving the accuracy of the classification results. Furthermore, since this method combines information from two different lesion types—the second lesion and the first-shape lesion—and both are lesions from images in multiple body positions, combining images from different body positions and lesions with different annotation types to classify the lesion region provides more and richer information, resulting in more accurate classification results.

[0066] The above embodiments mentioned that the lesion region category is identified by two target segmentation images under each body position using a neural network model. The following embodiments will explain how the lesion category is identified when the neural network model includes two classification networks, a first classification network and a second classification network.

[0067] In another embodiment, such as Figure 3 As shown, another image classification method is provided. Based on the above embodiments, S204 may include the following steps:

[0068] S302, input the first target segmentation image and the second target segmentation image under each body position into the first classification network for classification, and determine the feature map and initial category corresponding to the lesion region in each target segmentation image.

[0069] For details, see Figure 4 As shown, after obtaining two target segmentation images for each body position, each target segmentation image for each body position can be sequentially input into the first classification network for processing such as convolution, pooling, dense convolutional block processing, and fully connected layers to obtain the feature map corresponding to each target segmentation image for each body position and the category of the lesion region. Since the category of the lesion region obtained here is not the final category, it is denoted as the initial category. This initial category can also be determined by selecting the category with the highest probability from multiple obtained classification probabilities.

[0070] Furthermore, the aforementioned first classification network is also a neural network model. Optionally, the first classification network can be a classification network employing an attention mechanism. Generally, for the test site being the breast, there are many lesion areas distributed on the breast. Therefore, this method uses target segmentation images under various body positions and various different lesion annotations (for example, lesions of the first shape in the first target segmentation image, i.e., the cloud-shaped annotation view in the figure, and lesions of the second shape in the second target segmentation image, i.e., the star-shaped annotation view in the figure). The attention mechanism classification network is used to learn the lesion feature information under various body positions and lesion annotation conditions to improve the stability and accuracy of lesion area classification.

[0071] Furthermore, the aforementioned first classification network can be a network trained using mean squared error, which can also improve the accuracy of the trained classification network and further improve the accuracy of classifying lesion areas.

[0072] S304. Based on the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network, determine the target category of the lesion region.

[0073] In this step, after obtaining the feature maps corresponding to the lesion regions in each target segmentation image under each body position and the initial category of the lesion regions, the feature maps of the lesion regions can be input into the second classification network for classification, and the target category of the lesion regions can be determined by combining the initial categories of the lesion regions; alternatively, the target segmentation images under each body position, along with the feature maps and initial categories, can be input into the second classification network for classification to obtain the target category of the lesion regions; other cases are also possible, which are not specifically limited here.

[0074] In this embodiment, the segmented images of each target in each body position are input into a first classification network for classification to obtain feature maps and initial categories of the lesion regions. These are then combined with a second classification network to determine the target category of the lesion regions. Here, two cascaded classification networks are used to determine the target category of the lesion regions, and the accuracy of the determined lesion region category can be improved through a layer-by-layer progressive classification approach. Furthermore, the first classification network is a classification network employing an attention mechanism, which can further improve the stability and accuracy of classifying lesion regions.

[0075] The above embodiments mention that the first classification network and the second classification network can be combined to identify the category of the lesion area. The specific identification process of the first classification network is described. The following embodiments will explain in detail how the second classification network specifically identifies the lesion area.

[0076] In another embodiment, such as Figure 5As shown, another image classification method is provided. Based on the above embodiments, S304 may include the following steps:

[0077] S402, based on the first target segmentation image and the second target segmentation image under each body position, determine the quantitative features corresponding to the lesion region in each target segmentation image; the above quantitative features are used to characterize the distribution of the lesion region.

[0078] In this step, the first target segmentation image is an image of a lesion region including a first shape, and the second target segmentation image is an image of a lesion region including a second shape.

[0079] Here, after obtaining the segmented images of each target in each body position, see... Figure 6 As shown, omics features can be extracted from the lesion regions in the first target segmentation images under various body positions using feature extraction models or manual extraction methods. This yields the omics features corresponding to the lesion regions in each first target segmentation image, which can also be denoted as the quantified features corresponding to the lesion regions. Here, the quantified features obtained for each lesion region in each first target segmentation image are generally multiple quantified features, with each quantified feature serving as a dimension. Therefore, each first target segmentation image can obtain N-dimensional features, where N is greater than or equal to 1.

[0080] For example, the omics features of each first target segmented image can be more than 100, such as shape features (e.g., flatness, elongation, etc.), pixel-level statistical features (e.g., entropy, 10% gray value, etc.), texture features (e.g., gray-level co-occurrence matrix, etc.).

[0081] Similarly, feature extraction models or manual extraction methods can be used to extract statistical features from the lesion regions in the second target segmentation images under various body positions, obtaining the statistical features corresponding to the lesion regions in each second target segmentation image, which can also be denoted as the quantitative features corresponding to the lesion regions. Here, the quantitative features obtained for the lesion regions in each second target segmentation image are generally multiple quantitative features, with each quantitative feature as a dimension. Thus, each second target segmentation image can obtain M-dimensional features, where M is greater than or equal to 1, and the sizes of M and N can be equal or unequal.

[0082] For example, the omics features of each second target segmentation image can be more than 50, such as distribution features (e.g., minimum distance value, number of outliers, etc.), pixel-level statistical features (e.g., entropy, 10% gray value, etc.).

[0083] S404. Based on the quantization features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, and the second classification network, determine the target category of the lesion region.

[0084] In this step, after obtaining the multi-dimensional quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps of each target segmentation image and the initial category of the lesion region can be combined, and other information can be added together and input into the second classification network for classification to obtain the target category of the lesion region.

[0085] In this embodiment, quantitative features corresponding to lesion regions in each target segmentation image are obtained through target segmentation images under each body position. Combined with feature maps of each target segmentation image, the initial category of the lesion region, and the second classification network, the target category of the lesion region is obtained. The quantitative features here can characterize the distribution of the lesion region, thus comprehensively characterizing the features of the lesion and conducting a comprehensive analysis of the lesion, thereby making the obtained lesion category more accurate.

[0086] When classifying the lesion areas of the test subjects in practice, in order to take into account the actual perception of the test subjects and further improve the accuracy, relevant clinical information can also be combined for analysis. The following examples will explain in detail how to combine clinical information to determine the category of lesion areas.

[0087] In another embodiment, such as Figure 7 As shown, another image classification method is provided. Based on the above embodiments, S404 may include the following steps:

[0088] S502, Obtain clinical characteristic information of the subject to be tested.

[0089] The aforementioned clinical characteristics information includes at least the subject's sensory information about the test site and / or the subject's medical history. Sensory information may include the subject's pain perception at the test site. Of course, the clinical characteristics information may also include other information, such as the subject's age, gender, occupation, and purpose of the imaging (e.g., physical examination, medical visit, etc.).

[0090] Specifically, before examining the subject, clinical characteristic information of the subject can be obtained through interaction with the subject, and this clinical characteristic information can be input into a computer device for storage.

[0091] S504. Based on clinical feature information, quantitative features corresponding to lesion regions in each target segmentation image, feature maps and initial categories corresponding to lesion regions in each target segmentation image, and the second classification network, determine the target category of the lesion region.

[0092] In this step, after obtaining the clinical feature information of the subject, the quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, optionally, the clinical feature information, the quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image can be fused and then input into the second classification network to determine the target category of the lesion region.

[0093] See Figure 8 The overall structure example diagram shows that after obtaining the above-mentioned information, such as two target segmentation images under two body positions, resulting in a total of four target segmentation images, four feature maps (i.e., Featuremap 1 / 2 / 3 / 4 in the diagram) and four initial categories (i.e., classification probabilities 1 / 2 / 3 / 4 in the diagram) can be obtained. The two first target segmentation images can yield two N-dimensional features, and the two second target segmentation images can yield two M-dimensional features, that is, M*2-dimensional features and N*2-dimensional features are obtained.

[0094] Next, the feature information, including the four feature maps, four initial categories, M*2 dimensional features, N*2 dimensional features, and clinical features of the subject, can be fused together and input into the second classification network to identify the lesion category and obtain the target category of the lesion region. For example, in binary classification, the second classification network can output the probabilities of belonging to two categories, such as Probability1 and Probability0, and select the category with the highest probability as the target category.

[0095] Additionally, the second classification network here can be a DenseNet network, which is typically trained before use. The second classification network is trained based on a set of sample feature information corresponding to multiple sample objects. The sample feature information for each sample object includes clinical features, quantitative features, a feature map, an initial category, and the labeled category of the lesion region.

[0096] In other words, during training, target segmentation images of multiple sample objects in multiple positions can be acquired first, and corresponding sample quantization features, sample feature maps, and initial sample categories can be obtained through the target segmentation images. At the same time, a label category can be set for each lesion region, and sample clinical feature information of each sample object can also be obtained. Then, the sample clinical feature information, sample quantization features, sample feature maps, initial sample categories, and label categories of lesion regions of each sample object can be used as a sample feature information set to train the second classification network, thus obtaining the trained second classification network.

[0097] In this embodiment, the target category of the lesion region is obtained by combining the clinical feature information of the subject under test with the quantitative features corresponding to the lesion region in each target segmentation image, the feature map of each target segmentation image, the initial category of the lesion region, and the second classification network. Here, the clinical feature information of the subject under test is combined, so that the category of the lesion region determined in the end is directly related to the individual and is more in line with the actual situation of the individual, that is, more accurate.

[0098] The following examples provide a detailed explanation of how to obtain two corresponding target segmentation images from medical images in various body positions.

[0099] In another embodiment, another image classification method is provided. Based on the above embodiments, S202 may include the following steps:

[0100] Based on the preset first segmentation model and second segmentation model, the lesion region in the medical images of the test site under different body positions is segmented, and the first target segmentation image and the second target segmentation image corresponding to the medical images under each body position are determined.

[0101] For details, see Figure 9 As shown, a detection model can be first used to detect lesion regions in medical images under various body positions, obtaining the detection results of lesion regions, that is, to first locate the lesion regions in medical images under various body positions. Then, a first segmentation model can be used to further segment the lesion regions in medical images under various body positions, obtaining a first target segmentation image under various body positions, in which the lesion regions included in each first target segmentation image are lesion regions of a first shape; at the same time, a second segmentation model can also be used to segment the lesion regions in medical images under various body positions, obtaining a second target segmentation image under various body positions, in which the lesion regions included in each second target segmentation image are lesion regions of a second shape.

[0102] In other words, the detection model and the first segmentation model can be cascaded. The detection model first locates the lesion region in the medical image (coarse segmentation), and then the detection result is input into the first segmentation model for fine segmentation, refining the edges of the lesion region to obtain the first target segmented image. Similarly, the detection model and the second segmentation model can also be cascaded. The detection model first locates the lesion region in the medical image (coarse segmentation), and then the detection result is input into the second segmentation model for fine segmentation, refining the edges of the lesion region to obtain the second target segmented image.

[0103] Here, the lesion area is segmented by a cascaded detection network and segmentation model. Since the lesion area can be quickly located by the detection model, the accuracy and efficiency of subsequent segmentation of the lesion area by the segmentation model can be improved.

[0104] Alternatively, these models can be trained before using the detection model, the first segmentation model, and the second segmentation model to segment the lesion region. The detection model can be trained using pre-collected sample images and their labeled data, which includes the bounding box information for the lesion region. The first segmentation model can be trained based on multiple first sample medical images, each labeled with a lesion region of a first shape. The second segmentation model can be trained based on multiple second sample medical images, each labeled with a lesion region of a second shape.

[0105] In this embodiment, a first segmentation model and a second segmentation model, pre-trained, are used to segment medical images in various body positions to obtain a first target segmented image and a second target segmented image in each body position. The first segmentation model is trained based on sample images of lesion regions labeled with a first shape, and the second segmentation model is trained based on sample images of lesion regions labeled with a second shape. The segmentation models trained in this way are more accurate, thereby improving the accuracy of medical image segmentation. In addition, using segmentation models to segment medical images can also improve the efficiency of segmentation when there are many medical images.

[0106] It should be noted that, Figure 4 , 6 The lines in 8, 9, etc. do not affect the substantive content of the embodiments of this application.

[0107] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0108] Based on the same inventive concept, this application also provides an image classification apparatus for implementing the image classification method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image classification apparatus embodiments provided below can be found in the limitations of the image classification method described above, and will not be repeated here.

[0109] In one embodiment, such as Figure 10 As shown, an image classification device is provided, including a determining module 11 and a classification module 12, wherein:

[0110] The determining module 11 is used to determine, based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion area in the medical images in each body position; the lesion area included in the first target segmentation image is a lesion area of ​​a first shape, and the lesion area included in the second target segmentation image is a lesion area of ​​a second shape.

[0111] The classification module 12 is used to identify the category of the lesion region based on the first target segmentation image and the second target segmentation image under each body position, as well as the preset neural network model, and to determine the target category of the lesion region.

[0112] Optionally, the site to be tested is the breast, and the different body positions include the CC axial view and the MLO internal oblique view.

[0113] In another embodiment, another image classification device is provided. Based on the above embodiments, the neural network model includes a first classification network and a second classification network; the classification module 12 may include:

[0114] The first classification unit is used to input the first target segmentation image and the second target segmentation image under each body position into the first classification network for classification, and to determine the feature map and initial category corresponding to the lesion region in each target segmentation image;

[0115] The second classification unit is used to determine the target category of the lesion region based on the feature map and initial category corresponding to the lesion region in each target segmentation image, as well as the second classification network.

[0116] Optionally, the first classification network mentioned above is a classification network that employs an attention mechanism.

[0117] In another embodiment, another image classification device is provided, in which the second classification unit may include, based on the above embodiments:

[0118] The quantization feature determination subunit is used to determine the quantization features corresponding to the lesion regions in each target segmentation image based on the first target segmentation image and the second target segmentation image under each body position; the above quantization features are used to characterize the distribution of the lesion regions;

[0119] The classification subunit is used to determine the target category of the lesion region based on the quantized features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network.

[0120] In another embodiment, another image classification device is provided. Based on the above embodiments, the classification subunit is specifically used to acquire the clinical feature information of the subject to be tested; and to determine the target category of the lesion region based on the clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network.

[0121] In another embodiment, another image classification device is provided. Based on the above embodiments, the classification subunit is specifically used to fuse clinical feature information, quantitative features corresponding to lesion regions in each target segmented image, and feature maps and initial categories corresponding to lesion regions in each target segmented image, and then input them into a second classification network to determine the target category of the lesion region. The second classification network is trained based on a set of sample feature information corresponding to multiple sample objects. The sample feature information of each sample object includes sample clinical feature information, sample quantitative features, sample feature maps, sample initial categories, and the labeled category of the lesion region.

[0122] In another embodiment, another image classification device is provided. Based on the above embodiments, the determining module 11 may include:

[0123] The segmentation unit is used to segment the lesion region in medical images of the test site under different body positions according to a preset first segmentation model and a second segmentation model, and to determine the first target segmentation image and the second target segmentation image corresponding to each body position. The first segmentation model is trained based on multiple first sample medical images, and each first sample medical image is labeled with a lesion region of a first shape. The second segmentation model is trained based on multiple second sample medical images, and each second sample medical image is labeled with a lesion region of a second shape.

[0124] Each module in the aforementioned image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0125] In one embodiment, a computer device is provided, taking a terminal as an example, whose internal structure diagram can be as follows: Figure 11 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image classification method. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0126] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0128] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion region in the medical images in each body position are determined; the lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape; based on the first target segmentation image and the second target segmentation image in each body position, and a preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0129] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0130] The first and second target segmented images under each body position are input into the first classification network for classification to determine the feature map and initial category corresponding to the lesion region in each target segmented image; based on the feature map and initial category corresponding to the lesion region in each target segmented image, and the second classification network, the target category of the lesion region is determined.

[0131] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0132] Based on the first and second target segmentation images under each body position, the quantitative features corresponding to the lesion regions in each target segmentation image are determined; the aforementioned quantitative features are used to characterize the distribution of the lesion regions; based on the quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, and the second classification network, the target category of the lesion regions is determined.

[0133] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0134] Acquire the clinical feature information of the subject; based on the clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network, determine the target category of the lesion region.

[0135] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0136] After fusing the clinical feature information, the quantitative features corresponding to the lesion regions in each target segmentation image, and the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, the data is input into the second classification network to determine the target category of the lesion region. The second classification network is trained based on the sample feature information set corresponding to multiple sample objects. The sample feature information of each sample object includes sample clinical feature information, sample quantitative features, sample feature map, sample initial category, and the labeled category of the lesion region.

[0137] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0138] Based on the preset first segmentation model and second segmentation model, the lesion region in the medical images of the test site under different body positions is segmented to determine the first target segmentation image and the second target segmentation image corresponding to each body position. The first segmentation model is trained based on multiple first sample medical images, and each first sample medical image is labeled with a lesion region of a first shape. The second segmentation model is trained based on multiple second sample medical images, and each second sample medical image is labeled with a lesion region of a second shape.

[0139] In one embodiment, the first classification network described above is a classification network employing an attention mechanism.

[0140] In one embodiment, the site to be tested is the breast, and the different body positions include the CC axial view and the MLO internal oblique view.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0142] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion region in the medical images in each body position are determined; the lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape; based on the first target segmentation image and the second target segmentation image in each body position, and a preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0143] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0144] The first and second target segmented images under each body position are input into the first classification network for classification to determine the feature map and initial category corresponding to the lesion region in each target segmented image; based on the feature map and initial category corresponding to the lesion region in each target segmented image, and the second classification network, the target category of the lesion region is determined.

[0145] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0146] Based on the first and second target segmentation images under each body position, the quantitative features corresponding to the lesion regions in each target segmentation image are determined; the aforementioned quantitative features are used to characterize the distribution of the lesion regions; based on the quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, and the second classification network, the target category of the lesion regions is determined.

[0147] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0148] Acquire the clinical feature information of the subject; based on the clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network, determine the target category of the lesion region.

[0149] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0150] After fusing the clinical feature information, the quantitative features corresponding to the lesion regions in each target segmentation image, and the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, the data is input into the second classification network to determine the target category of the lesion region. The second classification network is trained based on the sample feature information set corresponding to multiple sample objects. The sample feature information of each sample object includes sample clinical feature information, sample quantitative features, sample feature map, sample initial category, and the labeled category of the lesion region.

[0151] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0152] Based on the preset first segmentation model and second segmentation model, the lesion region in the medical images of the test site under different body positions is segmented to determine the first target segmentation image and the second target segmentation image corresponding to each body position. The first segmentation model is trained based on multiple first sample medical images, and each first sample medical image is labeled with a lesion region of a first shape. The second segmentation model is trained based on multiple second sample medical images, and each second sample medical image is labeled with a lesion region of a second shape.

[0153] In one embodiment, the first classification network described above is a classification network employing an attention mechanism.

[0154] In one embodiment, the site to be tested is the breast, and the different body positions include the CC axial view and the MLO internal oblique view.

[0155] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0156] Based on the acquired medical images of the test site in different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion region in the medical images in each body position are determined; the lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape; based on the first target segmentation image and the second target segmentation image in each body position, and a preset neural network model, the category of the lesion region is identified, and the target category of the lesion region is determined.

[0157] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0158] The first and second target segmented images under each body position are input into the first classification network for classification to determine the feature map and initial category corresponding to the lesion region in each target segmented image; based on the feature map and initial category corresponding to the lesion region in each target segmented image, and the second classification network, the target category of the lesion region is determined.

[0159] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0160] Based on the first and second target segmentation images under each body position, the quantitative features corresponding to the lesion regions in each target segmentation image are determined; the aforementioned quantitative features are used to characterize the distribution of the lesion regions; based on the quantitative features corresponding to the lesion regions in each target segmentation image, the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, and the second classification network, the target category of the lesion regions is determined.

[0161] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0162] Acquire the clinical feature information of the subject; based on the clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network, determine the target category of the lesion region.

[0163] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0164] After fusing the clinical feature information, the quantitative features corresponding to the lesion regions in each target segmentation image, and the feature maps and initial categories corresponding to the lesion regions in each target segmentation image, the data is input into the second classification network to determine the target category of the lesion region. The second classification network is trained based on the sample feature information set corresponding to multiple sample objects. The sample feature information of each sample object includes sample clinical feature information, sample quantitative features, sample feature map, sample initial category, and the labeled category of the lesion region.

[0165] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0166] Based on the preset first segmentation model and second segmentation model, the lesion region in the medical images of the test site under different body positions is segmented to determine the first target segmentation image and the second target segmentation image corresponding to each body position. The first segmentation model is trained based on multiple first sample medical images, and each first sample medical image is labeled with a lesion region of a first shape. The second segmentation model is trained based on multiple second sample medical images, and each second sample medical image is labeled with a lesion region of a second shape.

[0167] In one embodiment, the first classification network described above is a classification network employing an attention mechanism.

[0168] In one embodiment, the site to be tested is the breast, and the different body positions include the CC axial view and the MLO internal oblique view.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for classifying breast images, characterized in that, The method includes: Based on the acquired medical images of the breast tissue under different body positions, a first target segmentation image and a second target segmentation image corresponding to the lesion region in the medical images under the CC axial view and the MLO endoscopic view are determined. The lesion region included in the first target segmentation image is a lesion region of a first shape, and the lesion region included in the second target segmentation image is a lesion region of a second shape. The first shape of the lesion region refers to the lesion region in the segmentation image being a cloud-like lesion, and the second shape of the lesion region refers to the lesion region in the segmentation image being a star-shaped lesion. The first target segmentation image is obtained according to a preset first segmentation model, and the second target segmentation image is obtained according to a preset second segmentation model. The first target segmentation image and the second target segmentation image under the CC axis and the MLO oblique position are input into the first classification network for classification, and the feature map and initial category corresponding to the lesion region in each target segmentation image are determined. Based on the first target segmentation image and the second target segmentation image under the CC-axis body position, the quantitative features corresponding to the lesion regions in each target segmentation image are determined; the quantitative features are used to characterize the distribution of the lesion regions; the quantitative features corresponding to the lesion regions in the first target segmentation image under the CC-axis body position include shape features, pixel-level statistical features, and texture features; the quantitative features corresponding to the lesion regions in the second target segmentation image under the CC-axis body position include distribution features and pixel-level statistical features. Based on the first target segmentation image and the second target segmentation image under the MLO oblique position, the quantitative features corresponding to the lesion region in each target segmentation image are determined; the quantitative features are used to characterize the distribution of the lesion region; the quantitative features corresponding to the lesion region in the first target segmentation image under the MLO oblique position include shape features, pixel-level statistical features, and texture features; the quantitative features corresponding to the lesion region in the second target segmentation image under the MLO oblique position include distribution features and pixel-level statistical features. The target category of the lesion region is determined based on the quantized features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network.

2. The method according to claim 1, characterized in that, The step of determining the target category of the lesion region based on the quantized features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network includes: Obtain clinical characteristic information of the subjects to be tested; The category of the lesion region is determined based on the clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network.

3. The method according to claim 2, characterized in that, The step of determining the category of the lesion region based on the clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map and initial category corresponding to the lesion region in each target segmentation image, and the second classification network includes: The clinical feature information, the quantitative features corresponding to the lesion region in each target segmentation image, the feature map corresponding to the lesion region in each target segmentation image, and the initial category are fused together and then input into the second classification network to determine the target category of the lesion region. The second classification network is trained based on a set of sample feature information corresponding to multiple sample objects. The sample feature information of each sample object includes sample clinical feature information, sample quantitative features, sample feature map, sample initial category, and labeling category of lesion area.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the first target segmentation image and the second target segmentation image corresponding to the lesion region in the medical images of the breast tissue to be tested under different body positions based on the acquired medical images of the breast tissue to be tested includes: Based on the preset first segmentation model and second segmentation model, the lesion area in the medical images of the breast under different body positions is segmented to determine the first target segmentation image and the second target segmentation image corresponding to the medical images under the CC axis and MLO oblique body positions. The first segmentation model is trained based on multiple first sample medical images, each of which is labeled with a lesion region of a first shape; the second segmentation model is trained based on multiple second sample medical images, each of which is labeled with a lesion region of a second shape.

5. The method according to any one of claims 1-3, characterized in that, The first classification network is a classification network that uses an attention mechanism.

6. The method according to any one of claims 1-3, characterized in that, The first classification network is trained using mean squared error.

7. The method according to any one of claims 1-3, characterized in that, The number of quantization features is multiple.

8. The method according to claim 3, characterized in that, The clinical feature information includes the subject's sensory information of the test site and / or the subject's medical history.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images