Prediction method, device and equipment for neoadjuvant chemotherapy effect of breast cancer and storage medium

Abstract

The invention discloses a breast cancer neoadjuvant chemotherapy effect prediction method, device and equipment and a storage medium, and is applied to the technical field of computer vision, and the method comprises the steps: obtaining a mammary gland pathological section image of a detection object, and extracting the pathological features of the mammary gland pathological section image; acquiring a mammary gland nuclear magnetic resonance image of the detected object, and extracting image features of the mammary gland nuclear magnetic resonance image; and predicting the therapeutic effect of the breast cancer neoadjuvant chemotherapy according to the pathological features and the image features. Through the above technical means, the pathological features of the breast pathological section image and the image features of the breast nuclear magnetic resonance image are fused to comprehensively analyze the breast tumor and the microenvironment features of the breast tumor from a plurality of dimensions such as microcosmic, macroscopic, form, function and the like, so that the neoadjuvant chemotherapy effect can be accurately predicted in the early stage of neoadjuvant chemotherapy of breast cancer; the problems of inaccurate curative effect evaluation and hysteresis of curative effect evaluation in the prior art are solved, and the accuracy, reliability and timeliness of curative effect evaluation are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for predicting the effect of neoadjuvant chemotherapy for breast cancer. Background Art

[0002] Neoadjuvant chemotherapy (NAC) is an important treatment for patients with locally advanced or inoperable breast cancer. Patients with pathological complete response (pCR) confirmed by pathological examination of surgical specimens after NAC can choose breast-conserving surgery with less trauma and better postoperative aesthetics by reducing the tumor burden of the primary breast cancer lesion and axillary lymph node metastasis, and can also avoid the more invasive axillary lymph node dissection, which greatly reduces the psychological burden of patients and surgical complications. In addition, NAC can be used to obtain relevant information such as whether breast tumor tissue is sensitive to neoadjuvant chemotherapy drugs. For breast cancer patients with poor NAC treatment effect, it is necessary to change the NAC regimen in time or directly undergo surgical treatment. More importantly, whether NAC can achieve pCR is significantly correlated with the prognosis of breast cancer patients. The disease-free survival (DFS) and overall survival (OS) of patients who achieve pCR after NAC are significantly higher than those with residual tumors. However, clinically, there are great individual differences in the therapeutic responsiveness of breast cancer patients to NAC, and only about 20% of patients with locally advanced breast cancer can achieve pCR through NAC.

[0003] At present, the evaluation of breast cancer pCR mainly relies on the traditional pathological comparative analysis of pre-treatment puncture and post-treatment resection, which has the disadvantage of delayed evaluation. In addition, due to the large temporal and spatial heterogeneity of tumors, pre-treatment puncture biopsy has the problem of incomplete sampling, which cannot fully reflect the pathological state of the tumor at the baseline level of treatment and predict the response of the tumor after NAC. Therefore, in clinical work, the overall response rate of breast cancer NAC is low and the individual differences in efficacy are large. If the tumor tissue's treatment response to NAC can be accurately predicted before NAC or in the early stage of treatment and whether the tumor can obtain pCR can be determined, it will provide a more comprehensive and accurate basis for the patient's choice of surgical method, the determination of other adjuvant treatment options, and the prediction of disease prognosis.

[0004] At present, the evaluation of the efficacy of NAC for breast cancer is mainly carried out through three methods: clinical physical examination, imaging detection and pathological examination. The main basis for clinical evaluation is clinical palpation and imaging measurement. Clinical palpation judges the efficacy by comparing the changes in tumor size before and after NAC. It is greatly affected by subjective factors such as the clinical experience of the examiner, and the efficacy evaluation is prone to deviation. In addition, it is usually difficult to evaluate lesions with large breast volume or deep location. In addition, after NAC, tumor cells degenerate and necrotize, and fibrous tissue hyperplasia and granuloma formation may occur at the lesion site. In this case, the mass can still be palpated during physical examination, but pathological examination may not show tumor lesions or the tumor lesions are small, which can easily lead to underestimation of the efficacy of NAC.

[0005] The Response Evaluation Criteria in Solid Tumors (RECIST) is a currently recognized imaging evaluation standard in clinical practice. It divides the treatment effect of breast cancer into four categories, namely complete response (CR), partial response (PR), stable disease (SD), and progressive disease (PD), based on the changes in the number and volume of tumors in imaging at several time points before, during, and after NAC treatment. However, some studies have found that the response of breast cancer to NAC does not necessarily manifest as a centripetal shrinkage of the lesion, and the reduction of microscopic tumor cells after NAC may not manifest as a reduction in macroscopic tumor volume. This suggests that our NAC imaging evaluation criteria based on gross morphological changes in lesions cannot fully and truly reflect the characteristics of microscopic pathological changes in some cases.

[0006] The Miller-Payne histological grading system has been widely used in the pathological evaluation of the effect of NAC treatment for breast cancer. It is also an independent factor affecting the patient's prognosis. It is significantly correlated with the patient's disease-free survival (DFS). For each grade reduction in the grading system, the risk of death will increase by 6.7% to 14.8%. The Miller-Payne grading system is mainly based on the apoptosis of tumor cells in the pathological sections during the treatment process, that is, the degree of change in the number of tumor cells. Compared with clinical and imaging evaluation, the pathological grading system is more important for guiding treatment and judging prognosis. However, the pathological grading system requires a puncture biopsy of the primary lesion of breast cancer before NAC treatment, and a pathological examination after the end of NAC treatment (usually 6 to 8 cycles, about 4.5 to 6 months) and surgical resection of the primary lesion. The effect of NAC treatment is evaluated by pathological changes before and after treatment. However, repeated invasive puncture biopsies of breast cancer have the risk of causing local bleeding and infection. Although the final surgical pathological biopsy can accurately determine whether the tumor is residual after NAC of breast cancer, there is a significant hysteresis in the postoperative pathological evaluation, and it is impossible to timely understand the sensitivity of neoadjuvant chemotherapy drugs in the early stage of treatment. Especially for patients with poor response to NAC treatment, it is impossible to replace the neoadjuvant chemotherapy regimen in time, and it is easy to miss the best time to adjust the treatment plan. This limits its application in clinical practice to some extent.

[0007] In pathology, during or after neoadjuvant chemotherapy for breast cancer, the residual tumor tissue not only shows a decrease in the number of tumor cells, but also changes in tumor cell morphology (including reduction in tumor cell volume, shrinkage of cell membrane, enlargement of cell nucleus, and increase in necrotic / degenerative bizarre nuclei), tumor stroma components (including fibrosis or myxoid changes, inflammatory cell infiltration, hemosiderin deposition and cholesterol crystal deposition, etc.) and tumor blood vessels (reduction in tumor blood vessels). Therefore, it is urgent to develop a method that can be used to dynamically monitor tumor cells, tumor stroma and tumor blood vessels during NAC to reflect the microscopic changes of tumor tissue during NAC, thereby providing a new means for early evaluation of the efficacy of NAC for breast cancer. Summary of the invention

[0008] The present application provides a method, device, equipment and storage medium for predicting the effect of neoadjuvant chemotherapy for breast cancer. By fusing the pathological characteristics of breast pathological slice images and the image characteristics of breast magnetic resonance imaging, the breast tumor and its microenvironment characteristics are comprehensively analyzed from multiple dimensions such as micro and macro, morphology and function, so that the effect of neoadjuvant chemotherapy for breast cancer can be accurately predicted in the early stage, which solves the problems of inaccurate efficacy evaluation and hysteresis in efficacy evaluation in the prior art, and improves the accuracy, reliability and timeliness of efficacy evaluation.

[0009] In a first aspect, the present application provides a method for predicting the effect of neoadjuvant chemotherapy for breast cancer, comprising: Acquire a breast pathology slice image of a detection object, and extract pathological features of the breast pathology slice image; Acquire a breast MRI image of the test subject, and extract image features of the breast MRI image; The therapeutic effect of neoadjuvant chemotherapy for breast cancer is predicted according to the pathological characteristics and the imaging characteristics.

[0010] In a second aspect, the present application provides a device for predicting the effect of neoadjuvant chemotherapy for breast cancer, comprising: A pathological feature acquisition module is configured to acquire a breast pathological section image of a detection object and extract pathological features of the breast pathological section image; An image feature acquisition module is configured to acquire a breast MRI image of the detection object and extract image features of the breast MRI image; The treatment effect prediction module is configured to predict the treatment effect of neoadjuvant chemotherapy for breast cancer based on the pathological characteristics and the imaging characteristics.

[0011] In a third aspect, the present application provides a device for predicting the effect of neoadjuvant chemotherapy for breast cancer, comprising: One or more processors; a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for predicting the effect of neoadjuvant chemotherapy for breast cancer as described in the first aspect.

[0012] In a fourth aspect, the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the method for predicting the effect of neoadjuvant chemotherapy for breast cancer as described in the first aspect.

[0013] In the present application, the pathological characteristics of the breast pathological slice image are extracted by acquiring the breast pathological slice image of the detection object; the breast nuclear magnetic resonance image of the detection object is acquired, and the image characteristics of the breast nuclear magnetic resonance image are extracted; according to the pathological characteristics and image characteristics, the therapeutic effect of neoadjuvant chemotherapy for breast cancer is predicted. Through the above-mentioned technical means, the pathological characteristics of the breast pathological slice image can intuitively reflect the size, density and tumor interstitial components, content and distribution characteristics of tumor cells from the histological level, and the therapeutic response characteristics of neoadjuvant chemotherapy for breast cancer can be evaluated through the microscopic dimension of pathological characteristics. The imaging characteristics can comprehensively characterize the microenvironment characteristics of the tumor from multiple aspects such as the gross morphological anatomy of the tumor, the molecular function of the tumor, and the blood perfusion of the tumor. Therefore, the pathological characteristics and image characteristics can comprehensively reflect the characteristics of breast tumors and their microenvironment from multiple dimensions such as macroscopic and microscopic, morphological and functional, and provide comprehensive tumor characteristics, so that the effect of neoadjuvant chemotherapy can be accurately predicted in the early stage of neoadjuvant chemotherapy for breast cancer, solving the problems of inaccurate efficacy evaluation and hysteresis in efficacy evaluation in the prior art, and improving the accuracy, reliability and timeliness of efficacy evaluation. Moreover, pathological characteristics and imaging characteristics are multimodal characteristics. When using pathological characteristics and imaging characteristics to predict the therapeutic effect, the correlation between pathological genomics and imaging genomics is taken into account, which reduces the error caused by single modality feature evaluation and further improves the accuracy and reliability of therapeutic effect evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of a method for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in an embodiment of the present application; Figure 2 is a flow chart of extracting pathological features of breast pathological slice images provided by an embodiment of the present application; Figure 3 is a flow chart of extracting image features of breast magnetic resonance imaging provided by an embodiment of the present application; Figure 4 This is a flow chart for predicting the therapeutic effect of neoadjuvant chemotherapy provided in an embodiment of the present application; Figure 5 is a schematic diagram of the overall evaluation process of the treatment effect provided in the embodiment of the present application; Figure 6 It is a schematic diagram of the structure of a device for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in an embodiment of the present application; Figure 7 It is a schematic diagram of the structure of a device for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only part of the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0016] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0017] In related implementation methods, the efficacy of neoadjuvant chemotherapy for breast cancer can be evaluated through three methods: clinical examination, imaging detection and pathological examination.

[0018] Clinical physical examination is mainly based on clinical palpation and imaging measurements. Clinical palpation is used to judge the efficacy of neoadjuvant chemotherapy by comparing the changes in tumor size before and after neoadjuvant chemotherapy. It is greatly influenced by subjective factors such as the clinical experience of the examiner, and the efficacy evaluation is prone to deviation. In addition, it is usually difficult to evaluate lesions with large breast volume or deep location. In addition, after neoadjuvant chemotherapy, tumor cells undergo degeneration and necrosis, and fibrous tissue hyperplasia and granuloma formation may occur at the lesion site. In this case, the mass can still be palpated during physical examination, but pathological examination may not show tumor lesions or the tumor lesions are small, which can easily lead to underestimation of the efficacy of neoadjuvant chemotherapy.

[0019] Imaging tests evaluate the efficacy of neoadjuvant chemotherapy based on gross morphological changes in lesions, and mainly use solid tumor efficacy evaluation criteria. The solid tumor efficacy evaluation criteria divide the treatment effect of breast cancer into four categories, namely complete remission, partial remission, disease stability, and disease progression, based on changes in the number and volume of tumors in imaging at several time points before, during, and after NAC treatment. However, studies have found that the response of breast cancer to neoadjuvant chemotherapy does not necessarily manifest as a centripetal contraction of lesions, and the reduction of microscopic tumor cells after neoadjuvant chemotherapy may not manifest as a reduction in macroscopic tumor volume. Therefore, imaging tests cannot fully and truly reflect the characteristics of microscopic pathological changes in some cases.

[0020] Pathological examination mainly uses the pathological grading system to evaluate the efficacy of neoadjuvant chemotherapy for breast cancer. The pathological grading system requires a puncture biopsy of the primary breast cancer lesion before neoadjuvant chemotherapy, and a pathological examination after the end of neoadjuvant chemotherapy and surgical resection of the primary lesion. The efficacy of neoadjuvant chemotherapy is evaluated by pathological changes before and after treatment. However, repeated invasive puncture biopsies of breast cancer have the risk of causing local bleeding and infection. Although the final surgical pathological biopsy can accurately determine whether the tumor remains after neoadjuvant chemotherapy for breast cancer, the postoperative pathological evaluation has significant hysteresis and cannot timely understand the sensitivity of neoadjuvant chemotherapy drugs in the early stage of treatment. Especially for patients with poor response to neoadjuvant chemotherapy, it is impossible to replace the better neoadjuvant chemotherapy regimen in time, and it is easy to miss the best time to adjust the treatment regimen, which to some extent limits its application in clinical practice.

[0021] Therefore, the above-mentioned method for evaluating the efficacy of neoadjuvant chemotherapy for breast cancer has the problems of inaccurate evaluation and / or delayed evaluation, and cannot accurately predict the effect of neoadjuvant chemotherapy for breast cancer in the early stage of neoadjuvant chemotherapy.

[0022] To solve the above problems, the present embodiment provides a method for predicting the effect of neoadjuvant chemotherapy for breast cancer, which integrates the pathological characteristics of breast pathological slice images and the image characteristics of breast magnetic resonance images to comprehensively analyze breast tumors and their microenvironment characteristics from multiple dimensions such as micro and macro, morphology and function, so as to accurately predict the effect of neoadjuvant chemotherapy for breast cancer in the early stage, thereby improving the accuracy, reliability and timeliness of efficacy evaluation.

[0023] The method for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in this embodiment can be performed by a device for predicting the effect of neoadjuvant chemotherapy for breast cancer, which can be implemented by software and / or hardware, and can be composed of two or more physical entities, or one physical entity. For example, the device for predicting the effect of neoadjuvant chemotherapy for breast cancer can be a computer device with strong processing power such as a computer and a server. Among them, the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0024] The device for predicting the effect of neoadjuvant chemotherapy for breast cancer is installed with at least one type of operating system. The device for predicting the effect of neoadjuvant chemotherapy for breast cancer can install at least one application based on the operating system. The application can be an application that comes with the operating system or an application downloaded from a third-party device or server. In this embodiment, the device for predicting the effect of neoadjuvant chemotherapy for breast cancer is installed with at least an application that can execute the method for predicting the effect of neoadjuvant chemotherapy for breast cancer.

[0025] For ease of understanding, this embodiment is described by taking a computer device as an example of a subject that executes the method for predicting the effect of neoadjuvant chemotherapy for breast cancer.

[0026] Figure 1 A flowchart of a method for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in an embodiment of the present application is given. Figure 1 The method for predicting the effect of neoadjuvant chemotherapy for breast cancer specifically includes: S110, obtaining a breast pathology slice image of the detection object, and extracting pathological features of the breast pathology slice image.

[0027] Among them, the detection objects are breast cancer patients treated with neoadjuvant chemotherapy. Breast pathology slice images are digital images of breast pathology tissue slices, which can be obtained by scanning breast pathology tissue slices with a digital scanner. Pathology tissue slices generally need to be stained to make the structure inside the cells more clearly visible. Hematoxylin-eosin (HE) staining can be used to stain breast pathology tissue slices, and then a digital scanner is used to collect high-resolution breast pathology slice images.

[0028] It should be noted that breast pathology slice images can intuitively reflect the size, density, and tumor stroma components, content, and distribution characteristics of tumor cells at the histological level. Therefore, the treatment response characteristics of breast cancer patients after neoadjuvant chemotherapy can be evaluated through the microscopic dimension of tissue pathology. In this regard, this embodiment aims to extract pathological features that can cover the size, density, and tumor stroma components, content, and distribution characteristics of tumor cells in breast pathology slice images, so as to combine pathological features and imaging features in the subsequent comprehensive analysis of breast tumors and their microenvironments from multiple dimensions such as micro and macro, morphology, and function, so as to accurately evaluate the efficacy of neoadjuvant chemotherapy.

[0029] For example, when extracting pathological features of breast pathological slice images, grayscale features, texture features, shape features, and geometric features of breast pathological slice images can be extracted as pathological features by traditional feature extraction methods. Pathological features of breast pathological slice images can also be extracted by deep learning models, such as automatically learning to extract features in breast pathological slice images by convolutional neural networks, generative adversarial networks, or autoencoders, and the features extracted by the deep learning model are used as pathological features.

[0030] Optionally, semantic segmentation can be performed on the breast pathology slice image to segment the breast pathology slice image into different types of regions or cells, thereby extracting morphological features and geometric features of each type of region or cell as pathological features. Figure 2 FIG. 1 is a flow chart of extracting pathological features of breast pathological slice images provided by an embodiment of the present application. Figure 2 The step of extracting the pathological features of the breast pathological slice image specifically includes S1101-S1103: S1101, dividing a breast pathology slice image into a plurality of sub-images.

[0031] For example, the size of the breast pathology slice image is large, and it is difficult for the segmentation model to directly perform semantic segmentation on the breast pathology slice image. In this regard, the breast pathology slice image can be divided into multiple sub-images of uniform size so that the subsequent segmentation model can perform semantic segmentation on the sub-images. The size of the sub-image can be set according to actual needs.

[0032] Optionally, the breast pathology slice image can be slid and cropped by a sliding window of a preset size, and the image obtained by each sliding window cropping is used as a sub-image. When the sliding window traverses the entire breast pathology slice image, a plurality of sub-images corresponding to the breast pathology slice image can be obtained. Among them, the size and step size of the sliding window can be set according to actual conditions. In this embodiment, the breast pathology slice image can be quickly divided into a plurality of sub-images of equal size in the form of a sliding window, which effectively improves the efficiency of obtaining sub-images.

[0033] S1102 , performing semantic segmentation on each sub-image using a preset segmentation model to obtain a mask image output by the segmentation model.

[0034] The segmentation model is obtained by pre-training breast pathology slice sample images annotated with semantic information. The semantic information of the breast pathology slice sample images may include tumor cells and non-tumor cells, and may also include tumor cells, epithelial cells, vascular cells and lymphocytes, etc., and may also include tumor areas and non-tumor areas. Correspondingly, when the breast pathology slice sample images used to train the segmentation model contain a certain type of semantic information, the trained segmentation model can segment the sub-image into the type defined by the corresponding semantic information. For example, when the semantic information contained in the breast pathology slice sample images is tumor cells and non-tumor cells, the trained segmentation model can perform semantic segmentation of tumor cells and non-tumor cells on the sub-image, thereby outputting mask images of tumor cells and mask images of non-tumor cells.

[0035] Optionally, in order to enrich the feature content of the acquired pathological features, the sub-image may be segmented into regions and cells, thereby extracting the shape features and geometric features of the regions and cells from the segmented region mask images and cell mask images. Specifically, the steps of semantically segmenting the sub-image include S11021-S11022: S11021. Perform region segmentation on each sub-image using a preset region segmentation model to obtain region mask images of various region types output by the region segmentation model, where the region types include tumor regions and non-tumor regions.

[0036] The region segmentation model is pre-trained using breast pathology slice sample images annotated with semantic information of tumor regions and non-tumor regions.

[0037] For example, the trained region segmentation model can segment the sub-image into a tumor region and a non-tumor region, and then output a region mask image containing the tumor region in the sub-image and a region mask image containing the non-tumor region in the sub-image. The tumor region can be understood as a region containing tumor cells in the sub-image, and the non-tumor region is a region not containing tumor cells in the sub-image.

[0038] S11022. Perform cell segmentation on each sub-image using a preset cell segmentation model to obtain a cell mask image of the cell type output by the cell segmentation model, where the cell type includes tumor cells.

[0039] Among them, the cell segmentation model is pre-trained by breast pathology section sample images annotated with semantic information of tumor cells and non-tumor cells (or more specific cell types such as lymphocytes, epithelial cells and vascular cells).

[0040] Exemplarily, the trained cell segmentation model can segment the sub-image into tumor cells and other cells, and then output a cell mask image containing tumor cells in the sub-image and a cell mask image containing other cells in the sub-image.

[0041] This embodiment performs semantic segmentation on the sub-image respectively through the regional segmentation model and the cell segmentation model, thereby obtaining the regional mask image and the cell mask image in the sub-image, so that the cell features that can intuitively reflect the size, morphology, density and other information of the tumor cells can be extracted through the cell mask image, and the regional features that can intuitively reflect the tumor interstitial composition, content and distribution can be extracted through the regional mask image, thereby improving the comprehensiveness of the pathological features and thus improving the accuracy of the efficacy prediction.

[0042] S1103. Acquire pathological features of the breast pathological slice image based on the mask image.

[0043] Exemplarily, shape features and geometric features of the corresponding region or cell are extracted from the mask image, and the extracted shape features and geometric features are merged into pathological features of the breast pathology slice image. This embodiment performs semantic segmentation on multiple sub-images of the breast pathology slice image through a segmentation model to obtain a mask image of the region or cell in the sub-image, and then extracts the shape features and geometric features of the region or cell through the mask image, thereby obtaining pathological features reflecting information of tumor cells or tumor interstitial components, so as to subsequently analyze microscopic changes in tumor tissue through pathological features and accurately evaluate the efficacy of neoadjuvant chemotherapy.

[0044] Optionally, when the regional mask image and the cell mask image are obtained through the above steps S11021-S11022, the pathological features can be obtained through the regional mask image and the cell mask image. Specifically, the steps of obtaining the pathological features include S11031-S11033: S11031. Extract regional features of the regional mask image.

[0045] Exemplarily, the shape features and geometric features of the tumor region or non-tumor region corresponding to each regional mask image are extracted, and the shape features and geometric features of the tumor region and the shape features and geometric features of the non-tumor region are spliced ​​as the regional features of the regional mask image. Among them, the shape features of the tumor region or the non-tumor region include the edge morphology of the region, the surface protrusion or depression morphology, etc., and the geometric features of the tumor region or the non-tumor region include the area, perimeter, diameter, eccentricity, roundness and elongation of the region. It can be understood that since the tumor region is a region containing multiple tumor cells, the combination of the shape features and geometric features of the non-tumor region and the tumor region can intuitively reflect the information such as the cost, content and distribution of the tumor stroma between tumor cells.

[0046] S11032. Extract cell features from the cell mask image.

[0047] Exemplarily, the shape features and geometric features of tumor cells or other cells corresponding to each cell mask image are extracted, and the shape features and geometric features of tumor cells and other cells are spliced ​​as cell features of the cell mask image. Similarly, the shape features of tumor cells or other cells include the edge morphology, surface protrusions or depressions of cells, etc., and the geometric features of tumor cells or other cells include the area, perimeter, diameter, eccentricity, roundness and elongation of cells, etc. It can be understood that the combination of shape features and geometric features of tumor cells and other cells can intuitively reflect information such as the size, density and morphology of tumor cells.

[0048] S11033. Combine the regional features and the cell features to obtain the pathological features of the breast pathological slice image.

[0049] Exemplarily, the regional features and cell features are spliced ​​together as the pathological features of the breast pathological slice image. Since the regional features can intuitively reflect the cost, content, and distribution of the tumor stroma, and the cell features can intuitively reflect the size, density, and morphology of the tumor cells, the pathological features spliced ​​together by the regional features and the cell features can simultaneously reflect the cost, content, and distribution of the tumor stroma and the size, density, and morphology of the tumor cells. Therefore, when the pathological features are subsequently used to predict the effect of neoadjuvant chemotherapy, the microscopic changes of the tumor tissue during the neoadjuvant chemotherapy process can be analyzed from the two aspects of tumor cell morphology and tumor stroma composition, which is conducive to improving the accuracy of the efficacy evaluation.

[0050] This embodiment combines the shape features and geometric features of the tumor area and non-tumor area as well as the shape features and geometric features of tumor cells and other cells as pathological features, so that the pathological features can simultaneously reflect information such as the cost, content and distribution of the tumor stroma and information such as the size, density and morphology of tumor cells, thereby improving the comprehensiveness of the pathological features and facilitating improving the accuracy of the efficacy evaluation.

[0051] S120, obtaining a breast MRI image of the test object, and extracting image features of the breast MRI image.

[0052] Among them, breast MRI is a medical image generated by breast detection using Magnetic Resonance Imaging (MRI). For example, a strong magnetic field and radio waves can be applied to the patient's breast and axillary areas to generate breast MRI images. Breast MRI can simultaneously perform multi-parameter and multi-directional imaging of the primary breast cancer lesion and the axilla to obtain various imaging sequences. For example, a T1-weighted imaging (T1-weighted imaging, T1WI) sequence and a T2-weighted imaging (T2-weighted imaging, T2WI) sequence can be formed by T1-weighted imaging or T2-weighted imaging, a diffusion MRI (dMRI) sequence can be formed by diffusion MRI, and a dynamic contrast enhancement (Dynamic Contrast Enhanced, DCE) sequence can be formed by dynamic contrast enhancement.

[0053] Optionally, one of the T1WI sequence, T2WI sequence, dMRI sequence or DCE sequence can be obtained as a breast MRI image, the primary lesion area in the breast MRI image can be identified, and the first-order statistical features and texture features of the primary lesion area can be extracted and spliced ​​as image features. However, a single imaging sequence is not sufficient to fully characterize the microenvironmental characteristics of the tumor, and when breast cancer spreads, multiple lesions or multi-center lesions may occur, which may include lymph nodes in the axillary area. Therefore, the therapeutic effect of neoadjuvant chemotherapy cannot be accurately analyzed only by the characteristics of the primary lesion area.

[0054] To this end, multiple imaging sequences can be obtained as breast MRI images, the primary lesion area and the axillary area in the multiple imaging sequences can be identified, and the first-order statistical features and texture features of the primary lesion area and the axillary area can be extracted and spliced ​​as image features. For example, Figure 3 1 is a flow chart of extracting image features of breast MRI images provided by the embodiment of the present application. Figure 3 As shown, the step of extracting the image features of the breast MRI image specifically includes S1201-S1203: S1201. Acquire multiple breast magnetic resonance images of the subject.

[0055] For example, the T2WI sequence can provide information such as changes in tumor size at a macroscopic level, and the dMRI sequence and DCE sequence can reflect the physiological metabolism of tumor tissue and histological changes such as microvascular perfusion. In particular, dMRI is currently the only non-invasive technology that can detect the micron-level tissue structure of the human body in real time. By combining the T2WI sequence, dMRI sequence and DCE sequence, it can comprehensively characterize the microenvironment characteristics of the tumor from multiple aspects such as the gross morphology and anatomy of the tumor, the molecular function of the tumor, and the blood perfusion of the tumor. Therefore, T2WI sequences, dMRI sequences and DCE sequences can be obtained as multiple breast magnetic resonance images.

[0056] S1202: Detect the primary lesion region and the axillary region in each breast MRI image, and determine the primary lesion image and the axillary image in multiple breast MRI images according to the primary lesion region and the axillary region.

[0057] Exemplarily, the T2WI sequence, dMRI sequence and DCE sequence can be sequentially input into a pre-trained region of interest detection model, and the primary lesion region and / or axillary region in the T2WI sequence, dMRI sequence and DCE sequence can be detected by the region of interest detection model. Among them, the primary lesion region and axillary region in an imaging sequence can be detected simultaneously by a region of interest detection model, and the region of interest detection model can be trained by sample images annotated with the primary lesion region and the axillary region. Alternatively, the primary lesion region in an imaging sequence is detected by one region of interest detection model and the axillary region in an imaging sequence is detected by another region of interest. The region of interest detection model for detecting the primary lesion region can be obtained by training with sample images annotated with the primary lesion region, and the region of interest detection model for detecting the axillary region can be obtained by training with sample images annotated with the axillary region.

[0058] After the primary lesion region and axillary region in the T2WI sequence, dMRI sequence, and DCE sequence are acquired, the primary lesion region and axillary region can be cut out in the T2WI sequence, dMRI sequence, and DCE sequence as the primary lesion image and axillary image, respectively.

[0059] Optionally, pixels around the primary lesion area and the axillary area may be filled into the primary lesion image and the axillary image to obtain more detailed information around the primary lesion and the axillary area. Specifically, the steps of determining the primary lesion image and the axillary image include S12021-S12022: S12021. Determine the primary lesion region and the corresponding adjacent region in the breast magnetic resonance imaging as the primary lesion image.

[0060] Exemplarily, the edge pixel position of the primary lesion region in the corresponding imaging sequence is obtained, and the pixel points around the primary lesion region whose interval with the edge pixel position is less than the preset distance are taken as the pixel points in the adjacent region corresponding to the primary lesion region, and a connected domain is formed according to the pixel points belonging to the adjacent region as the adjacent region corresponding to the primary lesion region, and the primary lesion region and the corresponding adjacent region are intercepted in the imaging sequence as the primary lesion image. The preset distance can be set based on actual needs.

[0061] S12022. Determine the axillary region and the corresponding adjacent region in the breast MRI image as an axillary image.

[0062] Exemplarily, the edge pixel position of the axillary area in the corresponding imaging sequence is obtained, and the pixel points around the axillary area whose intervals from the edge pixel position are less than a preset distance are used as the pixel points in the neighboring area corresponding to the axillary area. A connected domain is formed according to the pixel points belonging to the neighboring area as the neighboring area corresponding to the axillary area, and the axillary area and the corresponding neighboring area are intercepted as the axillary image in the imaging sequence.

[0063] It should be noted that the axillary region includes a left axillary region and a right axillary region. Accordingly, the left axillary region and the corresponding adjacent region in the imaging sequence can be determined as a left axillary image, and the right axillary region and the corresponding adjacent region can be determined as a right axillary image.

[0064] In this embodiment, the axillary region and the corresponding adjacent region in the imaging sequence are used as the axillary image, and the primary lesion region and the corresponding adjacent region in the imaging sequence are used as the primary lesion image, so that the intratumoral features and peritumoral features of the primary lesion and the axilla can be extracted in the primary lesion image and the axillary image later, and the detailed information of the primary lesion features and the axillary features can be enriched, so that the image features generated based on the primary lesion features and the axillary features can more intuitively and comprehensively reflect the microenvironmental features of the tumor, thereby improving the accuracy and reliability of the efficacy prediction.

[0065] S1203. Based on the primary lesion image and the axillary image, image features of multiple breast magnetic resonance images are obtained.

[0066] Exemplarily, the corresponding primary lesion images and axillary images are acquired in the T2WI sequence, the dMRI sequence, and the DCE sequence, that is, three primary lesion images, three left axillary images, and three right axillary images are acquired. The first-order statistical features and texture features of the three primary lesion images are extracted, the first-order statistical features and texture features of the three left axillary images are extracted, and the first-order statistical features and texture features of the three right axillary images are extracted. Afterwards, the extracted multiple first-order statistical features and multiple texture features are spliced ​​as image features.

[0067] It should be noted that in multiple imaging sequences, the same spatial point is not at the same pixel position, and the intensity range is not uniform in multiple imaging sequences. This results in the spatial points and intensities of the primary lesion images and axillary images obtained from multiple imaging sequences not being aligned and unified. As a result, the image features stitched together based on the features of the primary lesion images and axillary images cannot accurately characterize the microenvironment of the tumor, affecting the accuracy and reliability of subsequent efficacy predictions. To this end, before obtaining the image features, each axillary image and each primary lesion image may be preprocessed for spatial and intensity normalization, and image features may be generated based on the features of the preprocessed axillary images and primary lesion images. Exemplarily, the steps of obtaining image features include S12031-S12033: S12031. Perform normalization preprocessing on multiple axillary images, and extract axillary features of the preprocessed axillary images.

[0068] Exemplarily, the left axillary images corresponding to the T2WI sequence, the dMRI sequence, and the DCE sequence are spatially normalized and aligned to obtain a plurality of left axillary images with spatial point alignment. The plurality of left axillary images with spatial point alignment are subjected to intensity normalization processing to obtain a plurality of left axillary images with intensities in the same range. The normalization preprocessing process of the right axillary image is similar.

[0069] Optionally, when multiple left axillary images are spatially normalized and aligned, feature space points in the multiple left axillary images are extracted, the feature space points in the multiple left axillary images are matched to determine matching feature space points, relative transformation parameters of the multiple left axillary images are determined based on multiple groups of matching feature space points, and the multiple left axillary images are transformed by the relative transformation parameters to transform the same feature space point in the multiple left axillary images to the same pixel point, thereby achieving spatial point alignment of the multiple left axillary images. In the process of left axillary image conversion, the pixel values ​​at non-integer pixel points can be supplemented by an interpolation algorithm. Afterwards, according to the pixel value range of the multiple left axillary images after spatial point alignment, the pixel values ​​of each pixel point of the multiple left axillary images are adjusted to the same pixel value range, thereby achieving a unified intensity range of the multiple left axillary images.

[0070] After normalization preprocessing, the first-order statistical features and texture features are extracted from the axillary region of the axillary image of the T2WI sequence to obtain the intra-tumor features of the axillary in the T2WI sequence. The first-order statistical features and texture features are extracted from the adjacent regions corresponding to the axillary region of the preprocessed axillary image to obtain the peritumoral features of the axillary in the T2WI sequence. The intra-tumoral features and peritumoral features of the axillary in the T2WI sequence are spliced ​​to obtain the axillary features of the T2WI sequence. The acquisition process of the axillary features of the dMRI sequence and the DCE sequence is similar. Among them, the first-order statistical features include the average brightness, grayscale value, maximum pixel value, minimum pixel value, etc. of the corresponding image, which reflect the brightness distribution and variation range of the axillary region and the adjacent region.

[0071] S12032. Perform normalization preprocessing on multiple primary lesion images, and extract primary lesion features of the preprocessed primary lesion images.

[0072] Exemplarily, the primary lesion images corresponding to the T2WI sequence, the dMRI sequence, and the DCE sequence are spatially normalized and aligned to obtain multiple primary lesion images with spatial point alignment. The multiple primary lesion images with spatial point alignment are intensity normalized to obtain multiple primary lesion images with intensities in the same range.

[0073] Optionally, when multiple primary lesion images are aligned by spatial normalization, feature space points in multiple primary lesion images are extracted, feature space points in multiple primary lesion images are matched to determine matching feature space points, relative transformation parameters of multiple primary lesion images are determined according to multiple groups of matching feature space points, and multiple primary lesion images are transformed by relative transformation parameters to transform the same feature space point in multiple primary lesion images to the same pixel point, thereby achieving spatial point alignment of multiple primary lesion images. In the process of primary lesion image conversion, pixel values ​​at non-integer pixel points can be supplemented by an interpolation algorithm. Afterwards, according to the pixel value range of multiple primary lesion images after spatial point alignment, the pixel values ​​of each pixel point of multiple primary lesion images are adjusted to the same pixel value range, thereby achieving a unified intensity range of multiple primary lesion images.

[0074] After normalization preprocessing, the first-order statistical features and texture features are extracted from the primary lesion area of ​​the primary lesion image of the T2WI sequence to obtain the primary lesion intratumor features of the T2WI sequence, and the first-order statistical features and texture features are extracted from the corresponding adjacent areas of the primary lesion area of ​​the preprocessed primary lesion image to obtain the primary lesion peritumoral features of the T2WI sequence. The primary lesion intratumoral features and primary lesion peritumoral features of the T2WI sequence are spliced ​​to obtain the primary lesion features of the T2WI sequence. The acquisition process of the primary lesion features of the dMRI sequence and the DCE sequence is similar.

[0075] S12033. Combine the axillary features and the primary lesion features to obtain image features of multiple breast magnetic resonance images.

[0076] Exemplarily, the primary lesion characteristics and axillary characteristics of the T2WI sequence, dMRI sequence, and DCE sequence are spliced ​​as image features. Among them, the primary lesion characteristics and axillary characteristics of the T2WI sequence can reflect information such as changes in tumor size at a macroscopic level, and the primary lesion characteristics and axillary characteristics of the dMRI sequence and DCE sequence can reflect histological changes such as physiological metabolism and microvascular perfusion of tumor tissue. Accordingly, the image features obtained by combining the primary lesion characteristics and axillary characteristics of the T2WI sequence, dMRI sequence, and DCE sequence can comprehensively reflect the microenvironment information of the tumor from multiple aspects such as gross morphological anatomy of the tumor, molecular function of the tumor, and blood perfusion of the tumor.

[0077] This embodiment performs spatial normalization alignment and intensity normalization processing on multiple primary lesion images to ensure that the spatial points and intensities of the multiple primary lesion images are aligned and unified, and performs spatial normalization alignment and intensity normalization processing on multiple axillary images to ensure that the spatial points and intensities of the multiple axillary images are aligned and unified, thereby ensuring that the image features spliced ​​based on the features of the primary lesion images and the axillary images can accurately characterize the microenvironment of the tumor and improve the accuracy of subsequent efficacy prediction.

[0078] S130. Predict the therapeutic effect of neoadjuvant chemotherapy for breast cancer based on pathological and imaging features.

[0079] During or after neoadjuvant chemotherapy for breast cancer, the residual tumor tissue not only shows a decrease in the number of tumor cells, but also changes in tumor cell morphology, tumor stroma components, and tumor blood vessels. Pathological features can reflect information about tumor cell morphology and tumor stroma components, and imaging features can reflect information about tumor blood vessels. Therefore, combining pathological features and imaging features can monitor information about tumor cells, tumor stroma, and tumor blood vessels during neoadjuvant chemotherapy, thereby reflecting the microscopic changes in tumor tissue during neoadjuvant chemotherapy and predicting the therapeutic effect of neoadjuvant chemotherapy based on the changes.

[0080] For example, the pathological features and imaging features can be spliced ​​and input into a pre-trained efficacy prediction model, which can comprehensively analyze breast tumors and their microenvironmental characteristics from multiple dimensions such as macroscopic and microscopic, morphological and functional aspects based on the pathological features and imaging features, and output the therapeutic effect of neoadjuvant chemotherapy. The efficacy prediction model is a neural network model, such as a convolutional neural network or a recursive neural network.

[0081] In general, the training of neural network models requires a large amount of sample data to ensure good prediction results, and the sample data collection process and model training process are relatively complicated. However, the decision tree model can show good prediction results in a small number of samples and can effectively process high-order feature data. The model is easier to explain, so the decision tree model can be used to predict the treatment effect of neoadjuvant chemotherapy for breast cancer.

[0082] Optional, Figure 4 : is a flow chart for predicting the therapeutic effect of neoadjuvant chemotherapy provided in the embodiment of the present application. Figure 4 As shown, the step of predicting the therapeutic effect of neoadjuvant chemotherapy specifically includes S1301-S1302: S1301. Concatenate pathological features and imaging features to obtain multimodal features.

[0083] S1302. Input the multimodal features into a preset decision tree model to obtain the therapeutic effect of neoadjuvant chemotherapy for breast cancer predicted by the decision tree model.

[0084] For example, the pathological features and imaging features are spliced ​​to form multimodal features and then input into a pre-trained decision tree model. The decision tree model comprehensively analyzes the breast tumor and its microenvironment characteristics from multiple dimensions such as macroscopic and microscopic, morphological and functional based on the pathological features and imaging features, and outputs the therapeutic effect of neoadjuvant chemotherapy. Among them, the decision tree model can be trained in advance by integrating the features of the sample pathological features and the sample imaging features and the corresponding annotated therapeutic effects.

[0085] This embodiment can provide more comprehensive, detailed and accurate information support for efficacy evaluation by fusing pathological features and imaging features to obtain multimodal features, thereby reducing the error caused by single modality features. The multimodal features are analyzed through a decision tree model to predict the treatment effect, while ensuring the accuracy of the prediction, without the need for complex training and sample collection processes.

[0086] In order to facilitate a more intuitive understanding of the process of evaluating the therapeutic effect of neoadjuvant chemotherapy based on breast pathological slice images and breast magnetic resonance imaging in this application, Figure 5 The schematic diagram of the overall evaluation process of the treatment effect is described as an example. Figure 5As shown, the breast pathology slice image is divided into multiple sub-images, and the multiple sub-images are segmented by model A to obtain a regional mask image, and the multiple sub-images are segmented by model B to obtain a cell mask image. The shape features and geometric features of the region in the regional mask image are extracted, and the shape features and geometric features of the cells in the cell mask image are extracted, and the shape features and geometric features of the region and the shape features and geometric features of the cells are spliced ​​to form a pathological feature X. Breast magnetic resonance imaging includes T2WI sequence, dMRI sequence and DCE sequence. The primary lesion area and axillary area are identified in the T2WI sequence, the primary lesion area and axillary area are identified in the dMRI sequence, and the primary lesion area and axillary area are identified in the DCE sequence. The primary lesion area and the area around the axillary area are filled to obtain the primary lesion image and the axillary image, and the axillary image is normalized and preprocessed. Then, the first-order statistical features and texture features of the primary lesion image and the axillary image of the T2WI sequence, dMRI sequence, and DCE sequence are extracted, and the axillary intratumor features and axillary peritumoral features of the T2WI sequence, the primary lesion intratumor features and axillary peritumoral features of the T2WI sequence, the axillary intratumor features and axillary peritumoral features of the dMRI sequence, the primary lesion intratumor features and axillary peritumoral features of the dMRI sequence, the axillary intratumor features and axillary peritumoral features of the DCE sequence, and the primary lesion intratumor features and primary lesion peritumoral features of the DCE sequence are obtained, and then these features are spliced ​​to form the image feature Y. After that, the image feature Y and the pathological feature X are spliced ​​to obtain the multimodal feature, and the multimodal feature is input into the model C to obtain the therapeutic effect of neoadjuvant chemotherapy for breast cancer output by the model C. Among them, model A and model B are region segmentation model and cell segmentation model respectively, and model C is a decision tree model.

[0087] In summary, the method for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in the embodiment of the present application, by obtaining the breast pathological section image of the detection object, extracting the pathological features of the breast pathological section image; obtaining the breast nuclear magnetic resonance image of the detection object, extracting the image features of the breast nuclear magnetic resonance image; predicting the therapeutic effect of neoadjuvant chemotherapy for breast cancer according to the pathological features and image features. Through the above-mentioned technical means, the pathological features of the breast pathological section image can intuitively reflect the size, density and tumor interstitial components, content and distribution characteristics of tumor cells from the histological level, and the therapeutic response characteristics of neoadjuvant chemotherapy for breast cancer can be evaluated through the microscopic dimension of pathological features. Image features can comprehensively characterize the microenvironmental characteristics of tumors from multiple aspects such as gross morphological anatomy of tumors, molecular functions of tumors, and blood perfusion of tumors. Therefore, pathological features and image features can comprehensively reflect the characteristics of breast tumors and their microenvironment from multiple dimensions such as macroscopic and microscopic, morphological and functional, and provide comprehensive tumor features, so that the effect of neoadjuvant chemotherapy can be accurately predicted in the early stage of neoadjuvant chemotherapy for breast cancer, solving the problems of inaccurate efficacy evaluation and hysteresis in efficacy evaluation in the prior art, and improving the accuracy, reliability and timeliness of efficacy evaluation. Moreover, pathological characteristics and imaging characteristics are multimodal characteristics. When using pathological characteristics and imaging characteristics to predict the therapeutic effect, the correlation between pathological genomics and imaging genomics is taken into account, which reduces the error caused by single modality feature evaluation and further improves the accuracy and reliability of therapeutic effect evaluation.

[0088] Based on the above embodiments, Figure 6 This is a schematic diagram of the structure of a device for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in an embodiment of the present application. Figure 6 The breast cancer neoadjuvant chemotherapy effect prediction device provided in this embodiment specifically includes: a pathological feature acquisition module 21, an image feature acquisition module 22 and a treatment effect prediction module 23.

[0089] The pathological feature acquisition module 21 is configured to acquire a breast pathological slice image of the detection object and extract the pathological features of the breast pathological slice image; The image feature acquisition module 22 is configured to acquire a breast MRI image of the detection object and extract image features of the breast MRI image; The treatment effect prediction module 23 is configured to predict the treatment effect of neoadjuvant chemotherapy for breast cancer based on pathological characteristics and imaging characteristics.

[0090] On the basis of the above embodiment, the pathological feature acquisition module 21 includes: an image division unit, configured to divide the breast pathological slice image into multiple sub-images; a semantic segmentation unit, configured to perform semantic segmentation on each sub-image through a preset segmentation model to obtain a mask image output by the segmentation model; a pathological feature acquisition unit, configured to acquire the pathological features of the breast pathological slice image based on the mask image.

[0091] On the basis of the above embodiment, the semantic segmentation unit includes: a region segmentation subunit, configured to perform region segmentation on each sub-image through a preset region segmentation model, and obtain region mask images of various region types output by the region segmentation model, and the region types include tumor regions and non-tumor regions; a cell segmentation subunit, configured to perform cell segmentation on each sub-image through a preset cell segmentation model, and obtain cell mask images of cell types output by the cell segmentation model, and the cell types include tumor cells.

[0092] Based on the above embodiment, the pathological feature acquisition unit includes: a regional feature extraction subunit, configured to extract regional features of a regional mask image; a cell feature extraction subunit, configured to extract cell features of a cell mask image; and a pathological feature acquisition subunit, configured to splice regional features and cell features to obtain pathological features of a breast pathological slice image.

[0093] On the basis of the above-mentioned embodiment, the image feature acquisition module 22 includes: an image acquisition unit, configured to acquire multiple breast MRI images of the detection object; an image determination unit, configured to detect the primary lesion area and the axillary area in each breast MRI image, and determine the primary lesion image and the axillary image in the multiple breast MRI images according to the primary lesion area and the axillary area; the image feature acquisition unit, configured to acquire the image features of the multiple breast MRI images based on the primary lesion image and the axillary image.

[0094] Based on the above embodiment, the image determination unit includes: a primary lesion image determination subunit, configured to determine the primary lesion area and the corresponding adjacent area in the breast magnetic resonance image as the primary lesion image; an axillary image determination subunit, configured to determine the axillary area and the corresponding adjacent area in the breast magnetic resonance image as the axillary image.

[0095] On the basis of the above embodiment, the image feature acquisition unit includes: a first preprocessing subunit, configured to perform normalization preprocessing on multiple axillary images and extract axillary features of the preprocessed axillary images; a second preprocessing subunit, configured to perform normalization preprocessing on multiple primary lesion images and extract primary lesion features of the preprocessed primary lesion images; an image feature acquisition subunit, configured to splice the axillary features and the primary lesion features to obtain image features of multiple breast magnetic resonance images.

[0096] Based on the above embodiment, the treatment effect prediction module 23 includes: a multimodal feature generation unit, configured to splice pathological features and imaging features to obtain multimodal features; a treatment effect prediction unit, configured to input the multimodal features into a preset decision tree model to obtain the treatment effect of neoadjuvant chemotherapy for breast cancer predicted by the decision tree model.

[0097] As mentioned above, the breast cancer neoadjuvant chemotherapy effect prediction device provided by the embodiment of the present application obtains the breast pathological slice image of the detection object, extracts the pathological features of the breast pathological slice image; obtains the breast nuclear magnetic resonance image of the detection object, extracts the image features of the breast nuclear magnetic resonance image; predicts the therapeutic effect of breast cancer neoadjuvant chemotherapy according to the pathological features and image features. Through the above technical means, the pathological features of the breast pathological slice image can intuitively reflect the size, density and tumor interstitial components, content and distribution characteristics of tumor cells from the histological level, and the therapeutic response characteristics of breast cancer neoadjuvant chemotherapy can be evaluated through the microscopic dimension of pathological features. Image features can comprehensively characterize the microenvironmental characteristics of the tumor from multiple aspects such as the gross morphological anatomy of the tumor, the molecular function of the tumor, and the blood perfusion of the tumor. Therefore, the pathological features and image features can comprehensively reflect the characteristics of breast tumors and their microenvironment from multiple dimensions such as macroscopic and microscopic, morphological and functional, and provide comprehensive tumor features, so that the effect of neoadjuvant chemotherapy can be accurately predicted in the early stage of neoadjuvant chemotherapy for breast cancer, solving the problems of inaccurate efficacy evaluation and hysteresis in efficacy evaluation in the prior art, and improving the accuracy, reliability and timeliness of efficacy evaluation. Moreover, pathological characteristics and imaging characteristics are multimodal characteristics. When using pathological characteristics and imaging characteristics to predict the therapeutic effect, the correlation between pathological genomics and imaging genomics is taken into account, which reduces the error caused by single modality feature evaluation and further improves the accuracy and reliability of therapeutic effect evaluation.

[0098] The breast cancer neoadjuvant chemotherapy effect prediction device provided in the embodiment of the present application can be used to execute the breast cancer neoadjuvant chemotherapy effect prediction method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0099] Figure 7 This is a schematic diagram of a device for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in an embodiment of the present application. Figure 7 The device for predicting the effect of neoadjuvant chemotherapy for breast cancer includes: a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 in the device for predicting the effect of neoadjuvant chemotherapy for breast cancer can be one or more, and the number of memories 32 in the device for predicting the effect of neoadjuvant chemotherapy for breast cancer can be one or more. The processor 31, the memory 32, the communication device 33, the input device 34, and the output device 35 of the device for predicting the effect of neoadjuvant chemotherapy for breast cancer can be connected via a bus or other means.

[0100] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for predicting the effect of neoadjuvant chemotherapy for breast cancer in any embodiment of the present application (for example, the pathological feature acquisition module 21, the image feature acquisition module 22 and the treatment effect prediction module 23 in the device for predicting the effect of neoadjuvant chemotherapy for breast cancer). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0101] The communication device 33 is used for data transmission.

[0102] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, realizes the above-mentioned method for predicting the effect of neoadjuvant chemotherapy for breast cancer.

[0103] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0104] The above-mentioned breast cancer neoadjuvant chemotherapy effect prediction device can be used to execute the breast cancer neoadjuvant chemotherapy effect prediction method provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0105] The embodiment of the present application also provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute a method for predicting the effect of neoadjuvant chemotherapy for breast cancer. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer includes: obtaining a breast pathological section image of the detection object, and extracting pathological features of the breast pathological section image; obtaining a breast magnetic resonance imaging of the detection object, and extracting image features of the breast magnetic resonance imaging; and predicting the treatment effect of neoadjuvant chemotherapy for breast cancer based on the pathological features and the image features.

[0106] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0107] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the above-mentioned method for predicting the effect of neoadjuvant chemotherapy for breast cancer, and the computer executable instructions can also execute the relevant operations in the method for predicting the effect of neoadjuvant chemotherapy for breast cancer provided in any embodiment of the present application.

[0108] The breast cancer neoadjuvant chemotherapy effect prediction device, breast cancer neoadjuvant chemotherapy effect prediction system, storage medium and breast cancer neoadjuvant chemotherapy effect prediction equipment provided in the above embodiments can execute the breast cancer neoadjuvant chemotherapy effect prediction method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the breast cancer neoadjuvant chemotherapy effect prediction method provided in any embodiment of the present application.

[0109] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the protection scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for predicting the effect of neoadjuvant chemotherapy for breast cancer, characterized in that: include: Acquire a breast pathology slice image of a detection object, and extract pathological features of the breast pathology slice image; Acquire a breast MRI image of the test subject, and extract image features of the breast MRI image; The therapeutic effect of neoadjuvant chemotherapy for breast cancer is predicted according to the pathological characteristics and the imaging characteristics.

2. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 1, characterized in that: The step of extracting the pathological features of the breast pathological slice image comprises: Dividing the breast pathology slice image into a plurality of sub-images; Performing semantic segmentation on each of the sub-images using a preset segmentation model to obtain a mask image output by the segmentation model; The pathological features of the breast pathological slice image are acquired based on the mask image.

3. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 2, characterized in that: The semantic segmentation of each sub-image is performed by using a preset segmentation model to obtain a mask image output by the segmentation model, including: Performing region segmentation on each of the sub-images using a preset region segmentation model to obtain region mask images of various region types output by the region segmentation model, wherein the region types include tumor regions and non-tumor regions; Cell segmentation is performed on each of the sub-images using a preset cell segmentation model to obtain a cell mask image of a cell type output by the cell segmentation model, wherein the cell type includes tumor cells.

4. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 3, characterized in that: The step of acquiring the pathological features of the breast pathological slice image based on the mask image includes: Extracting regional features of the regional mask image; Extracting cell features of the cell mask image; The regional features and the cell features are spliced ​​to obtain the pathological features of the breast pathological section image.

5. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 1, characterized in that: The step of acquiring a breast MRI image of the detection object and extracting image features of the breast MRI image comprises: Acquire multiple breast magnetic resonance images of the subject; Detecting the primary lesion region and the axillary region in each of the breast MRI images, and determining the primary lesion image and the axillary image respectively in the multiple breast MRI images according to the primary lesion region and the axillary region; Based on the primary lesion image and the axillary image, image features of the multiple breast magnetic resonance images are acquired.

6. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 5, characterized in that: The method of respectively determining a primary lesion image and an axillary image from the plurality of breast MRI images according to the primary lesion area and the axillary area comprises: Determining the primary lesion region and the corresponding adjacent region in the breast magnetic resonance imaging as the primary lesion image; The axillary region and the corresponding adjacent region in the breast magnetic resonance image are determined as an axillary image.

7. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 5, characterized in that: The step of acquiring the image features of the plurality of breast magnetic resonance images based on the primary lesion image and the axillary image includes: performing normalization preprocessing on the plurality of axillary images, and extracting axillary features of the preprocessed axillary images; performing normalization preprocessing on the plurality of primary lesion images, and extracting primary lesion features of the preprocessed primary lesion images; The axillary features and the primary lesion features are spliced ​​to obtain the image features of the multiple breast magnetic resonance images.

8. The method for predicting the effect of neoadjuvant chemotherapy for breast cancer according to claim 1, characterized in that: The predicting of the therapeutic effect of neoadjuvant chemotherapy for breast cancer according to the pathological features and the imaging features includes: splicing the pathological features and the imaging features to obtain multimodal features; The multimodal features are input into a preset decision tree model to obtain the therapeutic effect of neoadjuvant chemotherapy for breast cancer predicted by the decision tree model.

9. A device for predicting the effect of neoadjuvant chemotherapy for breast cancer, characterized in that: include: A pathological feature acquisition module is configured to acquire a breast pathological section image of a detection object and extract pathological features of the breast pathological section image; An image feature acquisition module is configured to acquire a breast MRI image of the detection object and extract image features of the breast MRI image; The treatment effect prediction module is configured to predict the treatment effect of neoadjuvant chemotherapy for breast cancer based on the pathological characteristics and the imaging characteristics.

10. A device for predicting the effect of neoadjuvant chemotherapy for breast cancer, characterized in that: include: one or more processors; A storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for predicting the effect of neoadjuvant chemotherapy for breast cancer as described in any one of claims 1-8.

11. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the method for predicting the effect of neoadjuvant chemotherapy for breast cancer as described in any one of claims 1-8.

Images