Auxiliary system for breast cancer radiotherapy based on machine vision

Abstract

The invention relates to an auxiliary system for breast cancer radiotherapy based on machine vision, and belongs to the technical field of breast cancer radiotherapy. The system comprises a historical case model training module, a patient case model adjusting module, an optimization feedback learning module, a patient medical image processing module, a physical form change feature extraction module, a deformation quantity time difference prediction module and a three-dimensional structure reconstruction module. The system can quickly predict the deformation quantity of each dimension of a target organ of a breast cancer patient at a certain time point in the future during the edema period after a breast protection operation, and has relatively good accuracy and stability, so that an auxiliary radiotherapy treatment scheme for the breast cancer target organ of the patient can be guided; the system has an important clinical application scene, the treatment efficiency of breast cancer radiotherapy can be greatly improved, the problems of concentrated computing power and long consumed time in the prior art are solved, and a radiotherapy execution scheme can be formed according to a deformation quantity prediction result before radiotherapy operation.

Description

Technical Field

[0001] The present application relates to an auxiliary system for breast cancer radiotherapy based on machine vision, belonging to the technical field of breast cancer radiotherapy. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women, which poses a serious threat to women's health. Surgery is an important treatment for breast cancer. Breast cancer surgery is roughly divided into five stages: original local excision, radical mastectomy, extended radical mastectomy, modified radical mastectomy and breast-conserving surgery. Among them, modified radical mastectomy and breast-conserving surgery are currently the commonly used surgical procedures for the treatment of breast cancer. In particular, breast-conserving surgery takes into account both the treatment effect and the aesthetics, and is the best surgical method for early clinical stages I and II breast cancer. After breast-conserving surgery removes the primary malignant tumor, in order to prevent tumor recurrence, radiotherapy is often used as an adjuvant treatment. Radiotherapy is a way to use radiation to treat diseases. The principle is that the energy generated by a large amount of radiation can destroy the chromosomes of cells, stop the cells from growing, and thus eliminate cancer cells that can divide and grow rapidly.

[0003] However, after breast-conserving surgery, patients are very likely to develop edema at the surgical site. Depending on the patient's condition, the edema may last for several weeks or even months. However, edema is an undesirable condition during radiotherapy, because the key to radiotherapy is the accurate three-dimensional structure of the target organ. Only by calculating the appropriate dose of radiation based on the three-dimensional structure of the target organ can the target organ be accurately treated without affecting the surrounding normal tissues or organs.

[0004] The occurrence of edema makes the previous three-dimensional structure outline results of the target organ unusable, so it is necessary to re-take medical images of the target organ and outline it. In addition, it is necessary to re-integrate the treatment plan based on the outline results and perform a large number of complex calculations and verifications in order to formulate a radiotherapy radiation dose distribution treatment plan. Formulating a radiotherapy radiation dose distribution treatment plan is a process with huge computational complexity, many influencing factors and the need for complex verification. Therefore, it is time-consuming, especially when the radiotherapy task is heavy. It is very difficult to provide a radiotherapy radiation dose distribution treatment plan in a timely manner according to the needs of the patient.

[0005] In view of the above-mentioned problems existing in the prior art, and taking into account the characteristics of breast cancer compared with other cancers in the radiotherapy process, that is, the preset accuracy of the target organ delineation of breast cancer during radiotherapy planning can be relatively low, a deep learning network can be used for auxiliary predictive analysis, and predictive analysis can be performed on the edema of breast cancer patients who have undergone breast-conserving surgery during the adjuvant radiotherapy process. The delineation results of future target organs can be predicted based on historical medical images. As long as the predicted delineation results are not much different from the actual delineation results, the radiotherapy radiation dose distribution treatment plan can be formulated in advance while ensuring the preset delineation accuracy requirements. This can not only meet the needs of breast cancer radiotherapy, but also provide available solutions in a timely manner, and can solve the problems of delay in the current existing technology and the congestion caused by concentrated computing power requirements when there are many radiotherapy cases. Summary of the invention

[0006] In order to solve the above problems, a machine vision-based auxiliary system for breast cancer radiotherapy is provided. The system can guide the radiotherapy auxiliary treatment process for target organs of breast cancer patients who develop edema after breast-conserving surgery. It can greatly improve the treatment efficiency of breast cancer radiotherapy while also having good accuracy and stability, and has important clinical application scenarios.

[0007] The present application provides an auxiliary system for breast cancer radiotherapy based on machine vision, the system comprising a historical case model training module, a patient case model adjustment module and an optimization feedback learning module, as well as a patient medical image processing module, a physical morphology change feature extraction module, a deformation time difference prediction module and a three-dimensional structure reconstruction module; The historical case model training module calls the patient medical image processing module, the physical morphology change feature extraction module and the deformation time difference prediction module, and processes and learns the medical image data in the historical cases based on the case data of breast cancer patients who developed edema after breast-conserving surgery in the past in the database, and trains the model to obtain a basic training model, which can predict the three-dimensional structure of the target organ of the patient at a certain time point in the future; The patient case model adjustment module is adjusted based on the basic training model, adopts a transfer learning algorithm, adjusts the model network parameters to match the characteristics of various breast cancer patient types at different breast cancer stages, and iteratively trains and optimizes the model, thereby obtaining the matching of the deformation prediction effect of the target organ of breast cancer patients at different stages, and obtaining an adjusted prediction model for predicting the three-dimensional structure of the target organ at a certain time point in the future of the patient; The optimization feedback learning module calls the patient medical image processing module and the physical morphology change feature extraction module to process the medical image of a specific patient target organ to be predicted, and uses the back propagation algorithm to continuously optimize the adjusted prediction model, and adjusts the model parameters and model structure according to the feedback data to improve the accuracy and stability of the model, and obtains the final prediction model for predicting the three-dimensional structure of the target organ of the patient at a certain time point in the future; The patient medical image processing module analyzes and processes multi-angle medical images at the same time point, and can obtain the three-dimensional structure of the target organ at that time point; The physical morphology change feature extraction module uses image segmentation and geometric modeling methods to extract the volume and shape parameters of the target organ of breast cancer patients, and calculates the deformation amount of the target organ in different dimensions. The deformation amount of the target organ in different dimensions can comprehensively characterize the changes in the three-dimensional shape of the target organ of breast cancer patients at different time intervals. The deformation time difference prediction module adopts a long short-term memory network to analyze the sequence data of the deformation of the target organ over time, and trains to obtain a prediction model that can predict the deformation of the patient's target organ in different dimensions at a certain time point in the future; The three-dimensional structure reconstruction module uses a stereoscopic vision matching algorithm to obtain the three-dimensional structure prediction result of the patient's target organ at a certain future time point based on the obtained deformation prediction result of the target organ in different dimensions at the future time point and the historical medical imaging data of the target organ.

[0008] Optionally, the patient case model adjustment module includes a parameter adjustment submodule, a case breast cancer stage adaptive adjustment submodule and a patient model verification submodule; The parameter adjustment submodule adopts a transfer learning algorithm based on the basic training model to analyze the network parameters of the model, and adjusts the network layer weights and activation functions according to the breast cancer stage information of the patient in the case to obtain a parameter adjustment model; The case breast cancer stage adaptive adjustment submodule is based on a parameter adjustment model, adopts a gradient boosting machine, adjusts model parameters in combination with different breast cancer stages of patients, verifies the feasibility of the model under different breast cancer stages, and obtains an adaptive adjustment model for different breast cancer stages; The patient model verification submodule is based on an adaptive adjustment model for different breast cancer stages, uses cross-validation to analyze the performance of the model on the data, adjusts and optimizes the layer structure and parameters of the model according to the test results, and obtains an adjusted prediction model.

[0009] Optionally, in the patient model verification submodule, the adaptive adjustment model for different breast cancer stages is evaluated and optimized by a cross-validation method, by training and testing the model multiple times on different data subsets, and recording the accuracy and recall of the model in each iteration, and then adjusting and optimizing the layer structure and parameters of the model based on the test results.

[0010] Optionally, adjustments and optimizations to the model's layer structure and parameters include increasing or decreasing the number of network layers, adjusting the filter size of the convolutional layer, and changing the number of neurons in the fully connected layer.

[0011] Optionally, the deformation variable time difference prediction module includes a residual network feature learning submodule, a time series analysis submodule, an LSTM modeling submodule and a change prediction submodule; The residual network feature learning submodule extracts deep features through multiple convolutional layers and activation layers, then converts the deep features into shape variables of different dimensions of the target organ through a fully connected layer, adjusts the network weights using a back-propagation algorithm, and then optimizes the model through multiple iterative training; The time series analysis submodule uses time series analysis technology to process the shape variables of different dimensions of the target organ to generate time series data; The LSTM modeling submodule uses a long short-term memory network based on the generated time series data to learn the time series data, capture the changing trend of the shape variable of the target organ over time, and obtain the LSTM prediction model; The change prediction submodule is based on the LSTM prediction model and uses a trend analysis algorithm to predict and analyze the change in the shape of the target organ at a certain time point in the future.

[0012] Optionally, the patient medical image processing module includes a multi-view analysis submodule, an image fusion submodule and a three-dimensional modeling submodule; The multi-view analysis submodule analyzes multi-view medical images, acquires image data from multiple angles and differentiated focal lengths, pre-processes the images and generates a multi-view image set; The image fusion submodule is based on a multi-view image set, uses digital image processing technology, and adopts scale-invariant feature transformation to fuse images of different viewpoints to generate fused image data; The three-dimensional modeling submodule obtains the three-dimensional structure of the patient's target organ based on the fused image data and applying a stereo vision matching algorithm.

[0013] Optionally, the three-dimensional modeling submodule applies a stereo vision matching algorithm to analyze the similarities and differences between images at different angles, calculates the precise positions of different parts of the target organ in three-dimensional space, and obtains the three-dimensional structure of the patient's target organ through point cloud generation, mesh construction and surface reconstruction processes.

[0014] Optionally, the optimization feedback learning module includes a data feedback submodule, a model adjustment submodule and a performance evaluation submodule; The data feedback submodule collects the prediction results of the model at the difference data points for comparison based on the deformation characteristics of the patient's target organ in different dimensions, using mean square error and root mean square error, so as to identify the deviation and error pattern in the model prediction and generate model prediction feedback data; The model adjustment submodule uses a gradient descent algorithm based on the model prediction feedback data to adjust the network parameters of the estimation model, including modifying the network layer weights and bias parameters, optimizing the model's ability to match the data pattern, and generating adjusted model parameters; The performance evaluation submodule is based on the adjusted model parameters, applies cross-validation and indicator calculation methods, uses multiple data sets to evaluate the performance of the model, analyzes the prediction accuracy and generalization ability of the model under different conditions, and obtains the final prediction model for predicting the three-dimensional structure of the target organ at a certain time point in the future.

[0015] The beneficial effects of this application are: The machine vision-based auxiliary system for breast cancer radiotherapy provided in the present application includes a historical case model training module, a patient case model adjustment module and an optimization feedback learning module, as well as a patient medical image processing module, a physical morphology change feature extraction module, a deformation variable time difference prediction module and a three-dimensional structure reconstruction module. The historical case model training module and the patient case model adjustment module can be updated and iterated as the case data increases, and have continuous learning capabilities. In addition, for a specific patient, the historical medical image data of the specific patient during the period of edema can be fully utilized through the optimization feedback learning module, and the latest medical data for the specific patient can be continuously increased. Imaging data can be used to continuously modify and optimize the model. The patient case model adjustment module can not only improve the reuse of data and models, effectively reduce the training time and cost of the model, and improve the training efficiency of the model, but also benefit from the transfer learning algorithm. It can also train and optimize personalized models for patients at different stages of breast cancer, so that the generated prediction model of the deformation variables of each dimension of the target organ at a certain time point in the future has good accuracy and stability, thereby guiding the process of radiotherapy adjuvant treatment for breast cancer patients who develop edema after breast-conserving surgery. It has important clinical application scenarios and can greatly improve the treatment efficiency of breast cancer radiotherapy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 This is a schematic diagram of the structure of a machine vision-based auxiliary system for breast cancer radiotherapy according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application is described in detail below with reference to embodiments, but the present application is not limited to these embodiments.

[0018] like Figure 1 As shown, the present application provides an auxiliary system for breast cancer radiotherapy based on machine vision, including a historical case model training module, a patient case model adjustment module and an optimization feedback learning module, as well as four tool modules: a patient medical image processing module, a physical morphology change feature extraction module, a deformation time difference prediction module and a three-dimensional structure reconstruction module; The historical case model training module is based on the case data of breast cancer patients in the database who also developed edema during radiotherapy after breast-conserving surgery. It processes and learns the medical imaging data during the period of edema in the historical cases, and trains the model to obtain a basic training model. The basic training model can be used to output the three-dimensional structure of the target organ changes caused by edema at a certain point in the future.

[0019] Specifically, the historical case model training module analyzes and processes multiple medical images at the same time point during the edema period by calling the patient medical image processing module, and can obtain the three-dimensional structure of the target organ at that time point, and analyze and compare the three-dimensional structures of the target organs at different time points, and can obtain the three-dimensional structure of the target organs of the patient at different time points. Specifically, by calling the physical feature morphological change feature extraction module, the image segmentation and geometric modeling methods can be used to extract the current volume and shape parameters of the target organs of breast cancer patients, and then calculate the deformation variables of the target organs of breast cancer patients in different dimensions. The deformation variables in different dimensions can reflect the change state of the three-dimensional shape of the target organs of breast cancer patients at different time intervals, so as to measure the impact of edema on the target organs, and by analyzing the three-dimensional structure of the target organs at different time points during the edema period, the data information of the deformation variables of the target organs in different dimensions can be added to the patient's data, so that the data information can be used to predict the change direction of the target organs caused by edema at a certain time point in the future.

[0020] Specifically, by calling the deformation time difference prediction module, the deformation time difference prediction module can use the long short-term memory network to analyze the sequence data of the deformation of the target organ changing with time during edema based on the data information of the deformation of the target organ in different dimensions, and combine the patient case information to predict the deformation of each dimension of the target organ at a certain time point in the future, and generate the predicted deformation results of each dimension of the target organ, that is, obtain the three-dimensional structure of the target organ at a certain time point in the future.

[0021] The patient case model adjustment module is adjusted based on the basic training model obtained by the time difference prediction module in the historical case model training module. Specifically, the transfer learning algorithm is used to adjust the network parameters to match the characteristics of various types of breast cancer patients at different stages of breast cancer, and the model is iteratively trained to optimize the model to obtain the matching of the deformation prediction effect of the target organ of breast cancer patients at different stages, and finally obtain the adjusted prediction model of the deformation of each dimension of the target organ at a certain time point in the future. Here, according to the visualization needs, the three-dimensional structure reconstruction module can be selectively called to generate the three-dimensional structure of the target organ at a certain time point in the future. The reason why the patient case model adjustment module is needed here to distinguish breast cancer patients at different stages of breast cancer is that when edema occurs in breast cancer patients at different stages, the state and change of edema are often quite different. If the transfer learning algorithm is not used for distinction, the accuracy of the model is poor. By using the transfer learning algorithm to distinguish patients at different stages of breast cancer, it can not only make full use of historical case results, reduce training time and cost, but also match the different deformation caused by edema for different types of patients according to different stages of breast cancer.

[0022] It should be noted that in the present application, the three-dimensional structures of the target organs obtained above are all intermediate state results in the optimization process, not the final prediction results, and the three-dimensional structure reconstruction module is called only for visualizing the intermediate state. In the present application, the three-dimensional structure prediction results of the target organs output by the optimization feedback learning module are the final output results, and based on the three-dimensional structure prediction results of the target organs of the patients during the edema period outputted by the final output, the radiation dose distribution algorithm suitable for breast cancer is used to give the distribution results of the radiation dose of the radiation therapy in the three-dimensional space of the target organ, which requires a lot of computing power and time, and requires complex verification. The present application can predict the three-dimensional structure of the target organ at a certain time point in the future based on historical medical images, so that there is sufficient time to calculate and verify the radiation dose distribution scheme based on the three-dimensional structure, which can solve the problems existing in the current prior art. The radiation dose distribution algorithm suitable for breast cancer can adopt the commonly used methods in the field, and the relevant information and processing methods are well known to those skilled in the art, and the related products and applications are very mature, so they are not elaborated here.

[0023] The three-dimensional structure reconstruction module is based on the predicted results of the deformation of the target organ at a future time point, and integrates the deformation prediction data of each dimension to obtain the three-dimensional structure of the target organ at a certain time point. It is basically the inverse process of the physical morphological change feature extraction module. The three-dimensional structure reconstruction module can obtain the predicted deformation results of the target organ in each dimension at a future time point, use the stereo vision matching algorithm, and combine the historical medical impact of the case to establish the three-dimensional structure of the target organ at that time point in the future, that is, to outline the three-dimensional structure of the target organ based on the medical image; in addition, since it is necessary to predict the three-dimensional structure of the target organ at a future time point, it is also possible to obtain the predicted three-dimensional structure of the target organ based on the deformation result data of each dimension of the target organ and the medical images of the historical target organ.

[0024] Generally, radiotherapy adjuvant treatment for breast cancer usually lasts for one or two months. Therefore, during the radiotherapy treatment of the target organ with edema, the patient often takes and retains historical medical images. In addition, new medical images of the target organ are taken. However, after the shooting, the medical images are processed, and a specific plan for the distribution of radiotherapy radiation dose is given after the processing, and the radiotherapy plan is verified. Due to factors such as the complexity of the process, the large amount of calculation, and the complexity of the verification process, it is often too late to design a plan for the radiotherapy radiation dose after taking medical images before the radiotherapy radiation, and it causes problems such as insufficient computing power and untimely radiotherapy. However, the latest medical images of the patient's target organ, together with the historical medical images taken during the edema period, can be used to fine-tune the model previously obtained based on the time difference prediction model, which can improve the prediction accuracy of the model. The specific processing method in this application is described as follows.

[0025] The three-dimensional structure result of the target organ finally predicted by the present application scheme is finally given by the optimization feedback learning module. In the optimization feedback learning module, the patient medical image processing module is first used to perform three-dimensional modeling processing on the medical images of the target organ that can be obtained during the edema period of the specific patient, so as to obtain the historical data of the three-dimensional structure of the patient's target organ in the edema state; then the physical morphology change feature extraction module is called to extract and calculate the deformation amount of the target organ of the specific breast cancer patient in different dimensions, so as to obtain the data that can be obtained based on the specific patient and can be applied to analysis and prediction in time, and the back propagation algorithm is applied to continuously optimize the prediction model generated by the patient case model adjustment module to predict the three-dimensional structure of the target organ at a certain time point in the future, and adjust the model parameters and model structure according to the feedback data to improve the accuracy and stability of the model, and output the final optimized prediction model for predicting the three-dimensional structure of the target organ at a certain time point in the future, which can reflect the prediction result of the deformation of the target organ caused by edema of the patient during the edema period.

[0026] The historical case model training module uses the medical imaging image data set of historical breast cancer patients in the database, the basic case information of the patients, etc. for processing, and uses the time difference prediction model to establish the time series information of the deformation variable of the target organ. However, the model trained by the historical case model training module has a certain universality, especially for patients at different stages of breast cancer, the deformation variable of the target organ when edema occurs is different over time, so it is necessary to adjust the basic training model. The main purpose of the adjustment is to enable the adjusted prediction model to match the deformation variable prediction results of the target organ of patients at different stages of breast cancer. It should be noted that in the present application scheme, only breast cancer at different stages is distinguished and matched, and the skills and technicians can also selectively match other information of the patient, for example but not limited to BMI, age, weight, height, past medical history, number of radiotherapy, etc.

[0027] The physical morphology change feature extraction module mainly extracts the deformation of the target organs of breast cancer patients in different dimensions. The specific number and direction of different dimensions can be selected according to the situation. The more dimensions, the more accurate the representation of the three-dimensional structure of the target organ, but it will also increase the calculation amount of three-dimensional structure modeling. For example, the simplest three dimensions are the x-axis, y-axis and z-axis in space. Those skilled in the art know that only the above three dimensions cannot accurately represent the three-dimensional structure of the target organ. Therefore, technical personnel in this field need to select the required vector dimensions according to needs for the representation of the three-dimensional structure of the patient's target organ.

[0028] The reason why the physical morphological change feature extraction module is needed is because it is necessary to quantify the three-dimensional structure of the patient's target organ. For example, in the present application, the three-dimensional structure of the target organ is characterized by the deformation variables in different dimensions, and the quantified three-dimensional structure of the target organ can be represented as time series data of the deformation variable, so that it can be applied to the time difference prediction module for model training and optimization.

[0029] The deformation time difference prediction module mainly uses the deformation data of the target organ in different dimensions. In addition, the data set may also include but is not limited to the following data information: patient case information such as BMI, age, weight, height, medical history, historical radiotherapy times and historical radiotherapy radiation dose, etc., the deformation data of the target organ in different dimensions extracted by the physical morphological change feature extraction module, and then the deformation time difference prediction module is used to predict the deformation results of the target organ in each dimension corresponding to a certain time point of future radiotherapy treatment based on the time series data, so that the three-dimensional structure reconstruction module can be called to obtain the three-dimensional structure of the patient's target organ at that time point in the future.

[0030] In the patient medical image processing module, the residual network is first used to learn the features of the medical image dataset of the case. Before feature learning, the target organ medical images in the image dataset need to be preprocessed, including size standardization and normalization to adapt to the network input. Specifically, in the shape variable time difference prediction module, the residual network is first used for preliminary feature learning. The core of the residual network lies in its residual block. The gradient vanishing problem in deep network training is solved by introducing jump connections. During the training process, the residual network extracts deep features in the image through multiple convolutional layers and activation layers, and then converts these features into shape variables of different dimensions of the target organ through the fully connected layer, and uses the back propagation algorithm to adjust the network weights. Through multiple iterative training, the model achieves a high accuracy in estimating the shape variables of different dimensions of the target organ. Then, the long short-term memory network is used to analyze the sequence data of the shape variables of the target organ over time, and the shape variables of each dimension of the target organ at a certain time point in the future are predicted in combination with the patient case information. After the long short-term memory network training is completed, the model obtained can accurately estimate the shape variables of different dimensions of the target organ at a certain time point in the future based on the patient's case information.

[0031] If necessary, the three-dimensional structure of the target organ can be constructed based on the prediction results. However, it should be pointed out that the main purpose of the historical case model training module is to obtain a prediction model that can predict the deformation variables of different dimensions of the target organ at future time points through historical case data training. The construction of the three-dimensional structure is only to monitor the optimization effect of the prediction model on the prediction of the three-dimensional structure, which is mainly for improving the degree of visualization and is not a necessary module.

[0032] In the patient case model adjustment module, based on the already trained prediction model for predicting the deformation variables of different dimensions of the target organ at future time points, transfer learning is used to adjust it to adapt to the prediction of changes in the target organs of patients at different stages of breast cancer. Since the deformation variable change trends of the target organs of patients at different stages of breast cancer due to edema are quite different, it is necessary to distinguish the prediction model according to the different stages of breast cancer that the patients are in. The specific stage division can be determined based on the medical record information or the condition analysis combined with the doctor's judgment. The divided stages can also be artificially and accurately divided according to the commonly used division methods in this field or according to the requirements of the degree of precision, such as dividing the intervals according to the growth rate of the tumor. The specific setting can be based on the skills and choices of the technicians.

[0033] Since it takes a lot of time to train multiple prediction models that match patients at different stages of breast cancer, the patient case model adjustment module is used here in conjunction with the historical case model training module to improve the model training efficiency, and transfer learning is used to fine-tune the prediction model based on the trained prediction model of different dimensional deformation variables of the target organ at future time points. This is because transfer learning allows the model to use the knowledge gained on one task and apply it to another related task. Specifically, the model is transferred from the deformation variable prediction model of the target organ of patients at one stage of breast cancer to the deformation variable prediction model of the target organ of patients at different stages of breast cancer, so as to quickly obtain patients who are adapted to different stages of breast cancer development and the prediction model has good fitness.

[0034] During the transfer learning training process, the other layers of the basic training model are first fixed, and only the top layer is adjusted so that the model can recognize specific features in different stages of breast cancer. Then, the last few layers in the network are fine-tuned, such as the last layer, two layers, or three layers. The specific adjustments are made according to the technical staff, so that the adjusted model can better adapt to the new data set characteristics. During the fine-tuning process, continuous iterative training and parameter adjustment are used to ensure the generalization ability and accuracy of the model in predicting the shape variables of the target organs of patients at different stages of breast cancer. The fine-tuned model finally obtained has high accuracy, so that it can be applied to the target organs of patients at different stages of breast cancer, and finally can be used to predict the shape variables of the target organs of patients in different dimensions at a certain point in the future, so as to obtain the three-dimensional structure of the target organs of patients at that specific point in time.

[0035] The physical morphological change feature extraction module is based on the structure of the target organ obtained by the patient medical image processing module, and then uses image segmentation technology to calculate the deformation of the target organ in different dimensions at different time intervals. The choice of dimensions can be selected according to the needs of technical personnel in this field. For example, different vectors can be uniformly selected in three-dimensional space and the corresponding deformation can be calculated. It can also be selected non-uniformly according to the sensitivity of the change of the deformation. For example, for areas where the deformation changes little or areas that are morphologically flat, the number of vector dimensions can be reduced, and for areas where the surface state of the target organ is complex and changes frequently, the selection of vector dimensions can be appropriately increased. For example, for patients who are most prone to edema and have a greater impact on edema during breast-conserving surgery for breast cancer, relatively dense latitudes are selected for characterization and depiction. The primary purpose is to characterize the three-dimensional structure of the patient's target organ as accurately as possible.

[0036] Specifically, the image of the target organ can be separated from the complex medical image using image segmentation technology. This can be done through deep learning algorithms such as convolutional neural networks. The convolutional neural network can separate the target organ object through edge recognition and key feature distinction to ensure the accurate extraction of the target organ object. After that, the target organ object is combined with the patient's historical medical images and its shape variable is calculated according to the contour and size of the target organ using geometric modeling technology. This module can accurately calculate the shape variable of the target organ in different dimensions. This method can improve the feature extraction accuracy of the shape variable by combining image processing and geometric calculation. The obtained shape variable data can provide specific data for the subsequent time difference prediction analysis module.

[0037] In the shape variable time difference prediction module, the residual network is first used for preliminary feature learning. The core of the residual network lies in its residual block. The gradient vanishing problem in deep network training is solved by introducing jump connections. During the training process, the residual network extracts deep features in the image through multiple convolutional layers and activation layers, and then converts these features into shape variables of different dimensions of the target organ through the fully connected layer, and uses the back propagation algorithm to adjust the network weights. Through multiple iterative training, the model achieves a higher accuracy in estimating the shape variables of different dimensions of the target organ. Then, the long short-term memory network LSTM is used to analyze and predict the time series data of shape variables of different dimensions.

[0038] Because of its internal forget gate, input gate, and output gate structures, LSTM can maintain and adjust the memory of past information, making the LSTM network very suitable for processing time-dependent data, and can learn and remember long-term dependencies in sequences. By inputting the historical data of the shape variables of the target organ in different dimensions, the LSTM network can identify and learn the change pattern of the shape variables of the target organ in different dimensions, so that it can predict the shape variables of the target organ in different dimensions at a certain time point in the future based on historical data, that is, obtain the three-dimensional structure of the patient's target organ at that time point in the future. It should be noted that the patient's historical radiotherapy radiation dose and the patient's basic case information such as age, height, weight, BMI, medical history, historical radiotherapy times, etc. can also be input as needed for learning, and the accuracy of the prediction can be improved by adding more variables. Skilled and technical personnel can set it according to needs.

[0039] In the three-dimensional structure reconstruction module, the three-dimensional structure of the target organ at a certain time point is constructed by a stereo vision matching algorithm. This module can collect historical medical images of the patient's target organ at different angles and at different times, and then use a stereo matching algorithm to analyze the parallax between these images for the medical images at each time point, thereby reconstructing the three-dimensional structure of the target organ of the patient at that time point in the future. Alternatively, it can also be based on the prediction results of the time difference of the deformation variables in different dimensions at a certain time point in the future, combined with the three-dimensional structure of the target organ of the historical patient, to reconstruct the three-dimensional structure of the target organ of the patient at that time point in the future. In short, the three-dimensional structure reconstruction module can construct a three-dimensional structure based on the patient's medical images, and predict the construction of a three-dimensional structure based on the deformation variable results in different dimensions. Among them, the construction of the three-dimensional structure based on the medical image can be implemented separately or the function of the patient's medical image processing module can be directly called to implement part of the function. For this, the skills and technicians can choose according to their needs.

[0040] In the patient medical image processing module, the patient's medical images are processed through a stereo vision matching algorithm. The key technology of this module is the three-dimensional reconstruction algorithm. By performing geometric analysis on multiple two-dimensional images, the position of each pixel in three-dimensional space is calculated, thereby generating the three-dimensional structure of the target organ. The three-dimensional structure is the most intuitive view of the three-dimensional state of the target organ at a specific point in time.

[0041] In the optimization feedback learning module, by calling the patient medical image processing module, the three-dimensional structural historical data of the target organ at the time point of the medical image acquisition that can be obtained during the edema of the specific patient can be obtained, and the previously trained model can also be used to predict the deformation of the patient's target organ at a specific time point. Therefore, by using the optimization feedback learning module, the prediction time point in the model can be set to be the same as the actual time point of the medical image acquisition that can be obtained for the patient, so that the prediction model based on the deformation of different dimensions can obtain the three-dimensional structural prediction result of the target organ that has passed and has an accurate medical image at the past time point, and the actual three-dimensional structural result of the target organ obtained according to the medical image at the time point is combined for comparison and learning. Not only can the preliminary prediction effect of the prediction model be visually evaluated, but also the physical morphology change feature extraction module can be used to extract the actual deformation of each dimension of the target organ from the three-dimensional structure of the target organ at the time point of the medical image acquisition that can be obtained from the specific patient, and then the actual data is used to fine-tune the previous prediction model, so that the prediction accuracy of the prediction model is higher.

[0042] Specifically, the back propagation algorithm is used to continuously optimize and adjust the previously trained model. Back propagation is a common algorithm in deep learning. By calculating the prediction error and feeding the error back to the network, the weight and bias parameters in the network can be adjusted to optimize the parameters of the model. In the optimization process of the back propagation algorithm, the prediction direction of the model can be adjusted through fine adjustments, so that the fine-tuned prediction model can more accurately predict the shape of the target organ at a future time point, thereby reflecting the accurate three-dimensional structure of the target organ at a certain time point in the future. The existence of the optimization feedback learning module enables the model to have continuous learning capabilities. During the radiotherapy process, the latest available medical images can be continuously input and the optimization feedback learning module can be called, so that the model can be continuously optimized.

[0043] In addition, the medical imaging image data input for each case can even be used as data for the historical case model training module, thereby continuously optimizing the training model effect obtained by the historical case model training module. When performing model prediction for a specific patient, the patient case model adjustment module can be directly called based on the basic training model to obtain a personalized prediction model for the specific case. The existence of the patient case model adjustment module can ensure better prediction results, while the use of the historical case model training module can also improve the reuse of the prediction model. In addition, by optimizing the feedback learning module, the system can also have the ability to continuously learn and optimize.

[0044] As a specific implementation method, the patient case model adjustment module includes a parameter adjustment submodule, a case breast cancer stage adaptive adjustment submodule and a patient model verification submodule. The parameter adjustment submodule is based on the basic estimation model and adopts a transfer learning algorithm to analyze the network parameters of the model, and adjusts the network layer weights and activation functions according to the breast cancer stage information of the patient in the case to generate a parameter adjustment model; the case breast cancer stage adaptive adjustment submodule is based on the parameter adjustment model and adopts a gradient boosting machine to adjust the model parameters in combination with the different breast cancer stages of the patient, and verifies the feasibility of the model under different breast cancer stages to generate adaptive adjustment models for different breast cancer stages; the patient model verification submodule is based on the adaptive adjustment model for different breast cancer stages, and adopts cross-validation to analyze the performance of the model on the data, and adjusts and optimizes the layer structure and parameters of the model according to the test results to generate a fine-tuned model.

[0045] In the patient model validation submodule, the cross-validation method is used to evaluate and optimize the adaptive adjustment model for different breast cancer stages. Specifically, the model is trained and tested multiple times on different data subsets. The performance of the model, such as accuracy and recall, is recorded in each iteration, and the layer structure and parameters of the model are adjusted and optimized based on these test results. In addition, adjustment operations such as increasing or decreasing the number of network layers, adjusting the filter size of the convolutional layer, or changing the number of neurons in the fully connected layer can also be performed, and finally a prediction model with higher prediction accuracy and stronger generalization ability is obtained.

[0046] As a specific implementation method, the shape variable time difference prediction module includes a residual network feature learning submodule, a time series analysis submodule, an LSTM modeling submodule and a change prediction submodule; the residual network feature learning submodule extracts deep-level features through multiple convolutional layers and activation layers, and then converts these features into shape variables of different dimensions of the target organ through a fully connected layer, and uses a back propagation algorithm to adjust the network weights, and optimizes the model through multiple iterative training; the time series analysis submodule uses time series analysis technology to process historical data of shape variable changes over time and generate time series data; the LSTM modeling submodule uses a long short-term memory network based on the time series data to learn the time series data, capture the change trend of the shape variable of the target organ over time, and generate an LSTM prediction model; the change prediction submodule uses a trend analysis algorithm based on the LSTM prediction model to predict and analyze the shape variable changes of the target organ at a certain point in the future.

[0047] As a specific implementation method, the patient medical image processing module includes a multi-view analysis submodule, an image fusion submodule, and a three-dimensional modeling submodule; the multi-view analysis submodule analyzes multi-view medical images, obtains image data from multiple angles and differentiated focal lengths, pre-processes the images and generates a multi-view image set; the image fusion submodule is based on the multi-view image set, uses digital image processing technology, and adopts scale-invariant feature transformation to fuse images of differentiated perspectives to generate fused image data; the three-dimensional modeling submodule is based on the fused image data and applies a stereo vision matching algorithm to reconstruct the three-dimensional structure of the patient's target organ. In the three-dimensional modeling submodule, a stereo vision matching algorithm is applied to analyze the similarities and differences between images at different angles, and the precise positions of different parts of the target organ in three-dimensional space are calculated, which involves a large number of geometric calculations, such as point cloud generation, mesh construction, and surface reconstruction.

[0048] As a specific implementation method, the optimization feedback learning module includes a data feedback submodule, a model adjustment submodule and a performance evaluation submodule; the data feedback submodule is based on the deformation characteristics of the patient's target organ in different dimensions, uses mean square error and root mean square error, collects the prediction results of the model at the difference data points for comparison, thereby identifying the deviation and error pattern in the model prediction, and generating model prediction feedback data; the model adjustment submodule is based on the model prediction feedback data, uses the gradient descent algorithm to adjust the network parameters of the estimation model, including modifying the network layer weights and bias parameters, optimizing the model's matching ability for the data pattern, and generating adjusted model parameters; the performance evaluation submodule is based on the adjusted model parameters, applies cross-validation and indicator calculation methods, evaluates the performance of the model on multiple data sets, analyzes the prediction accuracy and generalization ability of the model under different conditions, and generates an optimized prediction model for predicting the three-dimensional structure of the target organ at a certain point in the future.

[0049] It should be noted that the three-dimensional structure reconstruction module in the present application is mainly used to use the predicted deformation data combined with the patient's historical medical images to obtain the predicted result of the three-dimensional structure of the target organ caused by edema at a certain time point in the future for the specific patient. The present application also involves a patient medical image processing module, which mainly obtains the three-dimensional structure of the target organ based on medical images. The description of the two in the present application is not distinguished or even interchangeable. This is because in actual implementation, the three-dimensional structure reconstruction module can realize different functions from the patient medical image processing module, each independently. Another way is that since the three-dimensional structure reconstruction module cannot obtain the final three-dimensional structure prediction result of the target organ only by relying on the predicted deformation data, for the convenience of implementation, the patient medical image processing module can also be used as a part of the three-dimensional structure reconstruction module, so that the three-dimensional structure reconstruction module can simultaneously realize the above-mentioned two three-dimensional structure output processing methods. Through the above explanation, those skilled in the art can distinguish the functions implemented in various places in this application and the specific meanings they refer to.

[0050] The present application also provides a machine vision-based breast cancer radiotherapy auxiliary method, which is performed based on the above-mentioned machine vision-based breast cancer radiotherapy auxiliary system, and includes the following steps: S1. Based on a historical case data set of patients with edema after breast-conserving surgery for breast cancer, the data set includes medical imaging data of the target organs of the patients. The images are analyzed to establish a three-dimensional structure, and different vector dimensions are used to characterize the three-dimensional structure, and the changes in different dimensions of the target organs of the patients in different time periods are obtained. The time series is trained using a time difference prediction model to obtain a basic prediction model for the deformation of each dimension of the target organ at a certain time point in the future; S2. Based on the basic prediction model of the shape variables of each dimension of the target organ at a certain time point in the future, a transfer learning algorithm is used to adjust the structure and parameters of the network, identify and match the shape variables of the target organs of patients at different stages of breast cancer, perform iterative model training, and generate an adjusted prediction model of the shape variables of each dimension of the target organ at a certain time point in the future; S3, inputting the medical image data of the target organ of the patient that needs to be predicted, and constructing the three-dimensional structure of the target organ of the patient, for the case data of the new patient that needs to be predicted, inputting the historical edema period data of the case into the adjusted prediction model of the deformation variables of each dimension of the target organ at a certain point in the future, using the back propagation algorithm, optimizing the adjusted prediction model of the deformation variables of each dimension of the target organ at a certain point in the future, and adjusting the model parameters according to the feedback data to generate a prediction model of the deformation variables of each dimension of the target organ at a certain point in the future with higher accuracy and stability; S4, tracking the patient's treatment status, and returning to step S3 to optimize the model every time new medical imaging data of the patient's target organ is taken; S5. When there is a new patient who needs to be tracked and predicted, return to step S3 and repeat steps S3 to S4.

[0051] The output of step S3 is used to design and give subsequent radiotherapy plans, while step S4 is because radiotherapy treatment often lasts for 1 to 2 months, so radiotherapy will need to be repeated continuously. Due to edema, the target organ continues to deform, and there are new medical imaging results of the patient. Therefore, you can enter S4 to continuously track the medical imaging data of the case and return to S3 for continuous optimization.

[0052] In addition, for the training process of the basic training model in step S1 and the adjusted prediction model in S2, the basic training model in S1 and the adjusted prediction model in S2 can be retrained regularly according to the increase in cases, for example, after the number of cases increases to a fixed number. This is because the case data is constantly increasing, and the continuous learning ability of the system can be ensured by increasing the amount of historical case data.

[0053] In the present application, the historical case model training module and the patient case model adjustment module can be updated and iterated as the case data increases, and have continuous learning capabilities. In addition, for a specific patient, the optimization feedback learning module can ensure that the latest medical imaging data for the specific patient is continuously increased, thereby modifying and optimizing the model. In addition, through the patient case model adjustment module, the reuse of data and models can be improved, the training time and cost of the model can be effectively reduced, and the training efficiency of the model can be improved. Thanks to the transfer learning algorithm, personalized models can also be trained and optimized for patients at different stages of breast cancer, so that the generated prediction model of the deformation variables of each dimension of the target organ at a certain point in the future has good accuracy and stability.

[0054] The above is only the embodiment of the present application, and the protection scope of the present application is not limited by these specific embodiments, but is determined by the claims of the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the technical ideas and principles of the present application should be included in the protection scope of the present application.

Claims

1. A machine vision-based auxiliary system for breast cancer radiotherapy, characterized in that: The system includes a historical case model training module, a patient case model adjustment module and an optimization feedback learning module, as well as a patient medical image processing module, a physical morphology change feature extraction module, a shape variable time difference prediction module and a three-dimensional structure reconstruction module; The historical case model training module calls the patient medical image processing module, the physical morphology change feature extraction module and the deformation time difference prediction module, and processes and learns the medical image data in the historical cases based on the case data of breast cancer patients who developed edema after breast-conserving surgery in the past in the database, and trains the model to obtain a basic training model, which can predict the three-dimensional structure of the target organ of the patient at a certain time point in the future; The patient case model adjustment module is adjusted based on the basic training model, adopts a transfer learning algorithm, adjusts the model network parameters to match the characteristics of various breast cancer patient types at different breast cancer stages, and iteratively trains and optimizes the model, thereby obtaining the matching of the deformation prediction effect of the target organ of breast cancer patients at different stages, and obtaining an adjusted prediction model for predicting the three-dimensional structure of the target organ at a certain time point in the future of the patient; The optimization feedback learning module calls the patient medical image processing module and the physical morphology change feature extraction module to process the medical image of a specific patient target organ to be predicted, and uses the back propagation algorithm to continuously optimize the adjusted prediction model, and adjusts the model parameters and model structure according to the feedback data to improve the accuracy and stability of the model, and obtains the final prediction model for predicting the three-dimensional structure of the target organ of the patient at a certain time point in the future; The patient medical image processing module analyzes and processes multi-angle medical images at the same time point, and can obtain the three-dimensional structure of the target organ at that time point; The physical morphology change feature extraction module uses image segmentation and geometric modeling methods to extract the volume and shape parameters of the target organ of breast cancer patients, and calculates the deformation amount of the target organ in different dimensions. The deformation amount of the target organ in different dimensions can comprehensively characterize the changes in the three-dimensional shape of the target organ of breast cancer patients at different time intervals. The deformation time difference prediction module adopts a long short-term memory network to analyze the sequence data of the deformation of the target organ over time, and trains to obtain a prediction model that can predict the deformation of the patient's target organ in different dimensions at a certain time point in the future; The three-dimensional structure reconstruction module uses a stereoscopic vision matching algorithm to obtain the three-dimensional structure prediction result of the patient's target organ at a certain future time point based on the obtained deformation prediction result of the target organ in different dimensions at the future time point and the historical medical imaging data of the target organ.

2. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 1, characterized in that: The patient case model adjustment module includes a parameter adjustment submodule, a case breast cancer stage adaptive adjustment submodule and a patient model verification submodule; The parameter adjustment submodule adopts a transfer learning algorithm based on the basic training model to analyze the network parameters of the model, and adjusts the network layer weights and activation functions according to the breast cancer stage information of the patient in the case to obtain a parameter adjustment model; The case breast cancer stage adaptive adjustment submodule is based on a parameter adjustment model, adopts a gradient boosting machine, adjusts model parameters in combination with different breast cancer stages of patients, verifies the feasibility of the model under different breast cancer stages, and obtains an adaptive adjustment model for different breast cancer stages; The patient model verification submodule is based on an adaptive adjustment model for different breast cancer stages, uses cross-validation to analyze the performance of the model on the data, adjusts and optimizes the layer structure and parameters of the model according to the test results, and obtains an adjusted prediction model.

3. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 2, characterized in that: In the patient model verification submodule, the adaptive adjustment model for different breast cancer stages is evaluated and optimized by cross-validation method. The model is trained and tested on different data subsets multiple times, and the accuracy and recall of the model are recorded in each iteration. Then, the layer structure and parameters of the model are adjusted and optimized based on the test results.

4. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 3, characterized in that: Adjusting and optimizing the model’s layer structure and parameters includes increasing or decreasing the number of network layers, adjusting the filter size of the convolutional layer, and changing the number of neurons in the fully connected layer.

5. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 1, characterized in that: The deformation time difference prediction module includes a residual network feature learning submodule, a time series analysis submodule, an LSTM modeling submodule and a change prediction submodule; The residual network feature learning submodule extracts deep features through multiple convolutional layers and activation layers, then converts the deep features into shape variables of different dimensions of the target organ through a fully connected layer, adjusts the network weights using a back-propagation algorithm, and then optimizes the model through multiple iterative training; The time series analysis submodule uses time series analysis technology to process the shape variables of different dimensions of the target organ to generate time series data; The LSTM modeling submodule uses a long short-term memory network based on the generated time series data to learn the time series data, capture the changing trend of the shape variable of the target organ over time, and obtain the LSTM prediction model; The change prediction submodule is based on the LSTM prediction model and uses a trend analysis algorithm to predict and analyze the change in the shape of the target organ at a certain time point in the future.

6. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 1, characterized in that: The patient medical image processing module includes a multi-view analysis submodule, an image fusion submodule and a three-dimensional modeling submodule; The multi-view analysis submodule analyzes multi-view medical images, acquires image data from multiple angles and differentiated focal lengths, pre-processes the images and generates a multi-view image set; The image fusion submodule is based on a multi-view image set, uses digital image processing technology, and adopts scale-invariant feature transformation to fuse images of different viewpoints to generate fused image data; The three-dimensional modeling submodule obtains the three-dimensional structure of the patient's target organ based on the fused image data and applying a stereo vision matching algorithm.

7. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 6, characterized in that: The 3D modeling submodule uses a stereo vision matching algorithm to analyze the similarities and differences between images at different angles, calculates the precise positions of different parts of the target organ in the 3D space, and obtains the 3D structure of the patient's target organ through point cloud generation, mesh construction and surface reconstruction.

8. The machine vision-based breast cancer radiotherapy auxiliary system according to claim 1, characterized in that: The optimization feedback learning module includes a data feedback submodule, a model adjustment submodule and a performance evaluation submodule; The data feedback submodule collects the prediction results of the model at the difference data points for comparison based on the deformation characteristics of the patient's target organ in different dimensions, using mean square error and root mean square error, so as to identify the deviation and error pattern in the model prediction and generate model prediction feedback data; The model adjustment submodule uses a gradient descent algorithm based on the model prediction feedback data to adjust the network parameters of the estimation model, including modifying the network layer weights and bias parameters, optimizing the model's ability to match the data pattern, and generating adjusted model parameters; The performance evaluation submodule is based on the adjusted model parameters, applies cross-validation and indicator calculation methods, uses multiple data sets to evaluate the performance of the model, analyzes the prediction accuracy and generalization ability of the model under different conditions, and obtains the final prediction model for predicting the three-dimensional structure of the target organ at a certain time point in the future.

Images