LLM-based intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer

Abstract

This application discloses an intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM, relating to the field of medical technology. The method includes: an acquisition unit for acquiring a multimodal temporal feature matrix of the target object within a preset time window; a feature processing unit for performing channel-independent temporal feature extraction and sparse bottleneck aggregation on the multimodal temporal feature matrix to obtain a low-dimensional feature matrix; a text processing unit for generating statistically enhanced prompt text based on the changing trends of target indicators in the multimodal temporal feature matrix and converting it into a text embedding sequence; and a decision-making unit for adaptively fusing the low-dimensional feature matrix and the text embedding sequence, and inferring and outputting the triggering decision result and corresponding explanatory text. This application solves the problems of model parameter explosion and severe overfitting caused by extremely high feature dimensions and extremely few training samples in existing online adaptive radiotherapy triggering decision-making technologies for prostate cancer.

Description

Technical Field

[0001] This application relates to the field of medical technology, and more specifically, to an intelligent triggering decision device for online adaptive radiotherapy of prostate cancer based on LLM. Background Technology

[0002] Online adaptive radiotherapy for prostate cancer typically involves daily updates to the radiotherapy plan to correct for target displacement caused by changes in bladder fullness and rectal contents, thereby improving treatment precision. This process requires re-delineating the target area and organs at risk, and recalculating the dose, based on cone-beam CT images taken before fractionated treatment.

[0003] In related technologies, some studies have attempted to use deep learning models such as convolutional neural networks or Transformers to process radiomics, dosimetry, and geometric features to construct end-to-end triggering decision models. However, in this clinical scenario, a single patient involves dozens of fractionated treatments, resulting in a limited overall training sample size, while the extracted multimodal features have a high dimensionality, far exceeding 1000 dimensions. The number of parameters in the fully connected layers of traditional deep networks increases quadratically with the feature dimension, easily leading to an explosion of model parameters under small sample conditions, severe overfitting during training, and generalization ability that fails to meet the needs of practical clinical applications.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an LLM-based intelligent triggering decision device for online adaptive radiotherapy of prostate cancer, which at least solves the technical problems of model parameter explosion and severe overfitting caused by the extremely high feature dimension and extremely small number of training samples in the prior art of online adaptive radiotherapy triggering decision technology for prostate cancer.

[0006] According to one aspect of the embodiments of this application, an intelligent triggering decision device for online adaptive radiotherapy of prostate cancer based on LLM is provided, comprising: an acquisition unit, configured to acquire a multimodal temporal feature matrix of a target object within a preset time window, wherein the multimodal temporal feature matrix includes radiomics features, dosimetry features, geometric features, and clinical features related to prostate cancer; a feature processing unit, configured to perform channel-independent temporal feature extraction and sparse bottleneck aggregation on the multimodal temporal feature matrix to obtain a low-dimensional feature matrix, wherein the number of rows in the low-dimensional feature matrix is ​​independent of the original feature dimension of the multimodal temporal feature matrix, and sparse bottleneck aggregation is used to selectively extract a preset number of target features from the multimodal temporal feature matrix; a text processing unit, configured to generate statistically enhanced prompt text based on the changing trend of target indicators in the multimodal temporal feature matrix, and convert the statistically enhanced prompt text into a text embedding sequence; and a decision unit, configured to perform adaptive multimodal fusion of the low-dimensional feature matrix and the text embedding sequence, and perform inference through a pre-trained large language model to output the triggering decision result of online adaptive radiotherapy and the explanatory text corresponding to the triggering decision result.

[0007] Optionally, the feature processing unit includes: a convolution processing subunit, used to perform causal convolution operations on each feature channel in the multimodal temporal feature matrix to obtain a convolutional feature matrix, wherein all feature channels in the multimodal temporal feature matrix share the same convolution kernel, and in the causal convolution operation, the output of the current time step depends on the input of the current time step and the historical time steps; an aggregation processing subunit, used to perform temporal aggregation on the convolutional feature matrix along the time dimension to obtain a scalar feature vector; a mapping processing subunit, used to map the scalar feature vector into a key matrix and a value matrix through a shared projection vector, wherein each row of the key matrix and the value matrix corresponds to a mapping representation of a feature channel; a computation processing subunit, used to perform cross-attention calculation on the key matrix and the value matrix using a preset latent query vector to obtain a bottleneck feature matrix; and a linear transformation processing subunit, used to perform a linear transformation on the bottleneck feature matrix through an alignment matrix to make the vector dimension of the transformed features consistent with the word vector dimension of the large language model, thereby obtaining a low-dimensional feature matrix.

[0008] Optionally, the shared projection vector includes a key projection vector and a value projection vector, wherein the key projection vector and the value projection vector have the same dimension and are independent of the number of feature channels.

[0009] Optionally, the text processing unit includes: an extraction processing subunit for extracting a time series sequence of at least one target indicator from a multimodal time series feature matrix; a sequence processing subunit for calculating the trend slope and variance of the time series sequence of each target indicator; a semantic processing subunit for mapping the trend slope of each target indicator to a trend semantic description and mapping the variance of each target indicator to a stability semantic description; a text processing subunit for filling the trend semantic description and stability semantic description into a preset prompt word template to generate statistically enhanced prompt text; and an embedding processing subunit for inputting the statistically enhanced prompt text into the embedding layer of a large language model to obtain a text embedding sequence, wherein the text embedding sequence is a vector sequence.

[0010] Optionally, the target metrics include at least one of bladder volume, rectal volume, and target coverage.

[0011] Optionally, the semantic processing subunit includes: a first trend semantic processing module, configured to map a trend semantic description to first trend semantic information when the trend slope is greater than a first preset threshold, wherein the first trend semantic information describes that the target indicator is rising and the rising rate is greater than a preset rate; a second trend semantic processing module, configured to map a trend semantic description to second trend semantic information when the trend slope is greater than a second preset threshold and less than or equal to a first preset threshold, wherein the second trend semantic information describes that the target indicator is rising and the rising rate is less than or equal to a preset rate; and a third trend semantic processing module, configured to map a trend semantic description to second trend semantic information when the trend slope is greater than or equal to a third preset threshold and less than or equal to a second preset threshold. Under a preset threshold, the trend semantic description is mapped to third trend semantic information, which describes the target indicator as remaining within the target range. The fourth trend semantic processing module maps the trend semantic description to fourth trend semantic information when the trend slope is greater than or equal to the fourth preset threshold and less than the third preset threshold. The fourth trend semantic information describes the target indicator as decreasing, and the decreasing speed is less than or equal to a preset speed. The fifth trend semantic processing module maps the trend semantic description to fifth trend semantic information when the trend slope is less than the fourth preset threshold. The fifth trend semantic information describes the target indicator as decreasing, and the decreasing speed is greater than a preset speed.

[0012] Optionally, the semantic processing subunit further includes: a coefficient processing module, used to determine the coefficient of variation based on the variance and mean; a first stability semantic processing module, used to map the stability semantic description to first stability semantic information when the coefficient of variation is greater than a first variation threshold, the first stability semantic information being used to describe that the fluctuation amplitude of the target indicator is greater than a preset fluctuation threshold; a second stability semantic processing module, used to map the stability semantic description to second stability semantic information when the coefficient of variation is greater than a second variation threshold and less than or equal to the first variation threshold, the second stability semantic information being used to describe that the fluctuation amplitude of the target indicator is less than or equal to the preset fluctuation threshold, and the fluctuation amplitude of the target indicator is not maintained within a preset range; and a third stability semantic processing module, used to map the stability semantic description to third stability semantic information when the coefficient of variation is less than or equal to the second variation threshold, the third stability semantic information being used to describe that the fluctuation amplitude of the target indicator is maintained within a preset range.

[0013] Optionally, the decision unit includes: a weighted processing subunit, used to weight the low-dimensional feature matrix using gating parameters to obtain a weighted numerical embedding sequence, wherein the gating parameters are used to dynamically adjust the weights of the low-dimensional feature matrix in the adaptive multimodal fusion process according to the image data quality; a concatenation processing subunit, used to concatenate the weighted numerical embedding sequence and the text embedding sequence along the sequence dimension to obtain a multimodal fusion sequence; an input processing subunit, used to input the multimodal fusion sequence into a large language model, wherein a hybrid expert structure is embedded in the large language model; an activation processing subunit, used to activate at least one expert network according to the multimodal fusion sequence through a gating network of the hybrid expert structure; and an output processing subunit, used to process the multimodal fusion sequence through at least one expert network and output the trigger decision result for online adaptive radiotherapy and the explanatory text corresponding to the trigger decision result.

[0014] Optionally, the expert network includes a first expert network and a second expert network, with the first expert network used to handle scenarios related to bladder changes and the second expert network used to handle scenarios related to rectal changes.

[0015] Optionally, the output processing subunit includes: a label output module for outputting binary labels, which are used to indicate whether online adaptive radiotherapy is triggered or not; and a text output module for generating structured text based on the large language model, including decision-making basis, target indicator analysis, and clinical recommendations.

[0016] As described above, the feature processing unit performs channel-independent temporal feature extraction and sparse bottleneck aggregation on the multimodal temporal feature matrix to obtain a low-dimensional feature matrix. The number of rows in this low-dimensional feature matrix is ​​independent of the original feature dimension of the multimodal temporal feature matrix. This allows the number of rows in the low-dimensional feature matrix output by the feature processing unit to be determined by a preset number, regardless of the original feature dimension. Furthermore, the number of trainable parameters in the feature processing unit during temporal feature extraction and sparse aggregation is decoupled from the original feature dimension and does not increase with the feature dimension. This helps avoid the parameter explosion caused by the quadratic increase in the number of parameters in traditional fully connected layers. Moreover, sparse bottleneck aggregation is used to selectively extract a preset number of target features from the multimodal temporal feature matrix, compressing thousands of features into a low-dimensional core feature representation. While retaining key decision information, this significantly reduces the feature dimension, effectively mitigating the problem of overfitting in high-dimensional feature spaces under small sample conditions. Furthermore, the text processing unit generates statistically enhanced prompt text based on the changing trends of the target indicators and converts it into a text embedding sequence. This allows the large language model to leverage pre-trained medical knowledge for reasoning, rather than relying solely on a limited number of training samples to learn the mapping relationship between features and labels. The decision-making unit adaptively fuses the low-dimensional feature matrix and the text embedding sequence, and uses the pre-trained large language model for reasoning. While fusing numerical and semantic features, it learns decision-making patterns for different clinical scenarios through expert network division of labor. This improves the model's generalization ability under limited sample conditions. Thus, even with very few training samples, it can achieve accurate decision-making with a parameter count decoupled from the original feature dimension. This solves the technical problem of model parameter explosion and severe overfitting caused by extremely high feature dimensions and very few training samples in existing online adaptive radiotherapy triggering decision-making technologies for prostate cancer. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of an optional LLM-based intelligent triggering decision device for online adaptive radiotherapy of prostate cancer according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating an optional multimodal temporal feature matrix construction process according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the decision-making process of an optional LLM-based intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer, according to an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of the processing flow of an optional feature processing unit according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the processing flow of an optional text processing unit according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the processing flow of an optional decision unit according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0027] According to an embodiment of this application, an embodiment of an intelligent triggering decision device for online adaptive radiotherapy of prostate cancer based on LLM is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0029] Figure 1 This is a schematic diagram of the structure of an LLM-based intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer according to an embodiment of this application, as shown below. Figure 1 As shown, this LLM-based intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer includes:

[0030] The acquisition unit is used to acquire the multimodal temporal feature matrix of the target object within a preset time window. The multimodal temporal feature matrix includes radiomics features, dosimetry features, geometric features and clinical features related to prostate cancer.

[0031] Optionally, a Large Language Model (LLM) can be a general-purpose large language model that can be fine-tuned to adapt to medical decision-making scenarios, or a medical vertical domain model that can be pre-trained on medical corpora to have a richer reserve of clinical knowledge, or a lightweight large language model that can be fine-tuned for parameters through techniques such as low-rank adaptation, making it suitable for small-sample learning scenarios.

[0032] Alternatively, Online Adaptive Radiotherapy (OART) can refer to a radiotherapy technique that improves the precision of prostate cancer treatment by re-optimizing the radiotherapy plan before each fractionation to correct for target displacement caused by diurnal changes in bladder fullness and rectal contents.

[0033] For example, the acquisition unit can specifically be used to acquire cone-beam computed tomography (CBCT) images, dose prediction data, and electronic medical record data of prostate cancer patients (corresponding to the target subjects mentioned above) before the current fractionation of treatment and for the past five consecutive fractions (i.e., time window L=5, L can also be other values). First, the CBCT images from each fraction are automatically segmented to delineate the contours of the prostate target area, bladder, and rectum. Then, radiomics features based on shape, texture, and wavelet transform can be extracted, for example, a total of 800 dimensions. Simultaneously, the dose distribution under the anatomical structures on the day can be predicted based on the Monte Carlo dose calculation engine (one of the dosimetric features), and target area dose coverage indicators (one of the dosimetric features, such as D95%, V95%) and organ-at-risk dose constraint indicators (one of the dosimetric features, such as bladder V40, rectum V50), for example, a total of 150 dimensions. Furthermore, the geometric changes (geometric features) of daily anatomical structures relative to the planned CT scan are calculated, including bladder volume, rectal volume, three-dimensional vector of target centroid displacement, and Dice similarity coefficient, totaling 100 dimensions. Finally, clinical features such as patient age, baseline prostate-specific antigen (PSA), Gleason score, tumor lymph node metastasis stage, number of completed CT scans, and cumulative adverse reaction scores are obtained from the electronic medical record, totaling 50 dimensions. Concatenating these four types of features in chronological order yields a 5×1100 multimodal temporal feature matrix.

[0034] For example, the LLM-based online adaptive radiotherapy intelligent triggering decision device for prostate cancer can be applied to target diseases, which may be, but are not limited to, prostate cancer or lung cancer. For instance, the acquisition unit can be used to acquire four-dimensional cone-beam CT images, dosimetric data, and clinical records of lung cancer patients prior to their three most recent fractionated treatments. The tumor target area and organs at risk (such as the heart, esophagus, and lung) in each fractionated four-dimensional cone-beam CT image are automatically segmented, and texture features based on gray-level co-occurrence matrix and gray-level run-length matrix, as well as shape features based on morphology, are extracted, resulting in a total of 600 radiomics features. A fast dose prediction model using a convolutional neural network is used to calculate the dose distribution under the anatomical structures on the day of treatment, extracting dosimetric parameters such as target area D98%, homogeneity index, and cardiac V30 and esophageal V35, resulting in a total of 100 dimensions. Geometric features such as the displacement of the tumor target area's centroid, rotation angle, and the rate of change in the volume of organs at risk are calculated, resulting in a total of 80 dimensions. Clinical characteristics such as patient age, performance status score, pulmonary function indicators, history of chemotherapy, and radiation pneumonitis score were obtained from the hospital information system, totaling 40 dimensions. All features were arranged chronologically to construct a 3×820 multimodal temporal feature matrix for subsequent triggering decision analysis.

[0035] Optionally, the target object may refer to a patient requiring online adaptive radiotherapy decision-making. In at least one embodiment, it may be an individual corresponding to a specific patient identifier. The preset time window may refer to a range of historical data used to analyze the trend of changes in the patient's anatomy. For example, considering that the anatomical changes in prostate cancer patients are mainly related to the most recent fractions of treatment, the preset time window may be set to include the current fraction (t) and its past L fractions, i.e., [t-L+1, t-L+2...t].

[0036] Optionally, the multimodal temporal feature matrix can represent the matrix formed by the changes of features extracted from different modal data of the target object over time within a preset time window. The rows of this matrix correspond to time steps, and the columns correspond to features.

[0037] like Figure 2 As shown, for example, the acquisition unit can acquire the multimodal temporal feature matrix of the target object within a preset time window in the following way:

[0038] Step 1: Time Series Data Acquisition

[0039] The acquisition unit acquires the original data sequence of the target object at the current interval t and its past L intervals, forming a time window [t-L+1, t-L+2...t]. For example, the preset time window length L can be set to 5, indicating that the data of the current interval and the past 4 intervals are considered.

[0040] Step 2: Multimodal Feature Extraction

[0041] For each segment within a preset time window, the acquisition unit can extract the following four types of features using a feature extractor:

[0042] radiomics features ( This can refer to quantitative features extracted from cone-beam computed tomography images prior to fractionated treatment, used to quantify the morphology and texture of tumors and normal tissues. For example, it can include texture features, shape features, wavelet transform features, etc., extracted based on automatic delineation of the prostate, bladder, and rectum.

[0043] Dosimetric characteristics ( Dosimetric features can refer to key parameters extracted from the dose distribution predicted based on the anatomical structures of the day using a dose calculation engine (such as a Monte Carlo dosimetry engine). For example, dosimetric features can include target volume dose coverage metrics and organ at risk (OARS) dose constraint metrics. Target volume dose coverage metrics can include D95% (95% of the target volume receiving the dose), D98%, V95% (the percentage of the volume receiving 95% of the prescribed dose), etc. Organ at risk dose constraint metrics can include V40 (the percentage of the volume receiving a 40 Gy dose) for the bladder, V50 for the rectum, D2cc for the rectum, etc.

[0044] Geometric features ( This can quantify the geometric changes of anatomical structures on the day relative to a reference (such as a planned CT scan). For example, geometric features may include: changes in the volume of organs at risk (e.g., bladder volume, rectal volume and their rate of change), spatial positional shifts (e.g., target centroid displacement, organ at risk centroid displacement), and morphological similarities (e.g., Dessian similarity coefficient, Hausdorff distance, overlap rate, etc. between the target and organs at risk).

[0045] Clinical features ( It can characterize the baseline information and static or slow-changing characteristics of the treatment response of the target subject, including clinical indicators such as age, baseline prostate-specific antigen level, and Gleason score.

[0046] Step 3: Feature Matrix Construction:

[0047] The acquisition unit concatenates the four types of features—radiomics features, dosimetry features, geometric features, and clinical features—at each time step to obtain a multimodal temporal feature matrix: Where L is the preset time window length, and M is the total feature dimension. The total feature dimension M can be equal to the sum of the radiomics feature dimension, dosimetry feature dimension, geometric feature dimension, and clinical feature dimension, and M is much greater than 1000.

[0048] The acquisition unit can collect and integrate multi-source heterogeneous data on the patient's current and historical pre-fractionation treatments. This facilitates the creation of a comprehensive high-dimensional digital profile describing changes in the patient's condition, helping to capture anatomical and dose changes that may affect radiotherapy planning from multiple dimensions, rather than relying on a single indicator, thereby improving the accuracy of decision-making. Since prostate cancer radiotherapy is significantly affected by organ filling, a high-dimensional multimodal temporal feature matrix can capture subtle anatomical deformation trends and their dosimetric consequences.

[0049] The feature processing unit is used to perform channel-independent temporal feature extraction and sparse bottleneck aggregation on the multimodal temporal feature matrix to obtain a low-dimensional feature matrix. The number of rows in the low-dimensional feature matrix is ​​independent of the original feature dimension of the multimodal temporal feature matrix. Sparse bottleneck aggregation is used to selectively extract a preset number of target features from the multimodal temporal feature matrix.

[0050] For example, the feature processing unit can specifically be used to perform dimensionality reduction and feature extraction on the multimodal temporal feature matrix output by the acquisition unit. First, the convolution processing subunit can perform a causal convolution operation on each feature channel in the feature matrix, with all feature channels sharing the same micro-convolution kernel. This causal convolution operation helps ensure that the output of each time step depends only on the information of the current and previous time steps, thereby extracting the local change trend of each feature within the time window. Since the convolution kernel is shared among all feature channels, the number of trainable parameters can remain constant regardless of the original feature dimension. Next, the aggregation processing subunit performs a pooling operation on the convolved feature matrix along the time dimension, compressing the time series of each feature channel into a single representative value, obtaining a scalar feature vector. The mapping processing subunit maps each scalar feature to a low-dimensional vector space by sharing the projection vector, generating a key matrix and a value matrix. Since the projection vector is shared among all feature channels, the number of parameters in this step is also independent of the original feature dimension. Subsequently, the computational processing unit utilizes a preset number of learnable latent query vectors to selectively extract the most relevant key information for the current decision from the key and value matrices through a cross-attention mechanism, obtaining the bottleneck feature matrix. The number of rows in this bottleneck feature matrix is ​​the preset number of latent queries, which is much smaller than the original feature dimension, facilitating sparse aggregation from high-dimensional features to low-dimensional core features. Finally, the linear transformation processing unit transforms the vector dimension of the bottleneck feature matrix into the embedding dimension of the large language model through an alignment matrix, obtaining the final low-dimensional feature matrix for use by subsequent decision units.

[0051] For example, the feature processing unit can process the multimodal temporal feature matrix using different parameter configurations. First, the convolution processing subunit sets shared convolution kernels of different sizes according to the temporal sensitivity requirements of the clinical scenario, and performs causal convolution operations on all feature channels. The size of the convolution kernel can determine the length of historical information aggregated in the time dimension for each feature. For example, for rapidly fluctuating organ volume features, such as bladder volume or rectal volume, a smaller convolution kernel can be used to capture recent changes; for slowly changing clinical baseline features, such as patient age or baseline prostate-specific antigen levels, a larger convolution kernel can be used to smooth noise. After convolution, the aggregation processing subunit can select different temporal aggregation methods according to the feature type. For example, for dosimetric features, average pooling can be used to obtain the average dose level; for geometric displacement features, max pooling can be used to capture extreme offsets and obtain scalar feature vectors; for radiomics features, the aggregation processing subunit can use median pooling to exclude outlier interference. The mapping processing unit maps scalar features to a small vector space using shared low-dimensional projection vectors. The dimension of this small vector space can be flexibly configured according to computational resource requirements. The computation processing unit utilizes learnable latent query vectors and extracts key information from massive features through a cross-attention mechanism. The number of latent queries can be adjusted according to the level of granularity required for clinical decision-making. For example, a larger number of queries can be set for complex cases requiring more comprehensive evaluation, while a smaller number can be set for rapid screening scenarios. The final number of rows in the bottleneck feature matrix is ​​the set number of latent queries, completely independent of the original feature dimension. The linear transformation processing unit maps the bottleneck features to the corresponding vector space using an alignment matrix, based on the embedding dimension requirements of different large language models, resulting in a low-dimensional feature matrix adapted to the subsequent model input.

[0052] Optionally, channel-independent temporal feature extraction can refer to the operation of extracting temporal information for each feature in the feature matrix separately, without mixing information between different channels. For example, the trend of bladder volume change is processed as an independent channel, without mixing it with rectal texture features. This helps to accurately capture the evolution pattern of each feature itself, is beneficial for extracting the short-term trend of a single feature, and avoids cross-feature information mixing.

[0053] Alternatively, sparse bottleneck aggregation can refer to a mechanism that selectively and sparsely extracts the K most critical pieces of information from a large number of original features and aggregates them into a low-dimensional bottleneck representation. Here, sparsity means that not all original features contribute equally to the final decision; the model needs to learn to focus only on the most important parts, which is beneficial for improving information density. For example, from 1000 original features, only 64 key features that best represent the current anatomical and dosing status can be extracted.

[0054] Optionally, the low-dimensional feature matrix can represent the data structure output by the feature processing unit, which has a dimension much lower than the original feature matrix. The number of rows in the low-dimensional feature matrix represents the number of extracted key features and is independent of the dimension of the original features. For example, the number of rows in the low-dimensional feature matrix can be set to 64. Each row of the low-dimensional feature matrix represents an aggregated key feature vector, and each column corresponds to the value of the feature in a certain dimension of the vector space.

[0055] Optionally, the number of rows in the low-dimensional feature matrix is ​​independent of the original feature dimension of the multimodal temporal feature matrix. This indicates that the number of key features output by the feature processing method of this application is a preset hyperparameter, the size of which does not increase with the increase of the input feature dimension M. For example, regardless of whether the input M is 1000-dimensional or 2000-dimensional, the output K can be stably maintained at 64 dimensions, which can decouple the number of parameters from the feature dimension.

[0056] The feature processing unit can compress the original feature matrix with thousands of dimensions into a low-dimensional feature matrix with a very small number of parameters through independent channel processing and sparse aggregation mechanism. This helps to fundamentally avoid the explosion of model parameters, enabling effective training even with limited samples, while extracting the most critical decision information.

[0057] The text processing unit is used to generate statistically enhanced prompt text based on the changing trends of target indicators in the multimodal temporal feature matrix, and to convert the statistically enhanced prompt text into a text embedding sequence.

[0058] For example, the text processing unit can be used to transform numerical temporal trends into semantic descriptions that clinicians can understand. For instance, the text processing unit may include an extraction processing subunit, a sequence processing subunit, a semantic processing subunit, a text processing subunit, and an embedding processing subunit. First, the extraction processing subunit can select core indicators of clinical interest as target indicators from the multimodal temporal feature matrix output by the acquisition unit, such as bladder volume, rectal volume, and target dose coverage. The selection of target indicators can be configured according to clinical guidelines and treatment protocols for different cancer types. The sequence processing subunit can perform statistical analysis on the temporal sequence of each target indicator, calculating the trend and fluctuation degree of the target indicator within a time window, for example, obtaining the rate of increase or decrease of the trend through linear fitting, and calculating the stability level of the indicator through the degree of dispersion. The semantic processing subunit, according to preset semantic mapping rules, transforms the calculated trend rate and fluctuation degree into corresponding Chinese clinical descriptions, which is beneficial for simulating the professional language used by clinicians during ward rounds and case discussions. The text processing subunit fills in the transformed semantic description according to a preset prompt word template. The template can include structured paragraphs such as patient treatment trend analysis and current dose assessment, generating complete statistically enhanced prompt text. Finally, the embedding processing subunit inputs the generated prompt text into the embedding layer of the large language model. Through word segmentation and vector mapping, a text embedding sequence is obtained. This text embedding sequence can retain the semantic information of the original text in vector form, which can then be used by the subsequent decision-making unit for interactive reasoning with the large language model.

[0059] For example, the text processing unit can employ differentiated target indicator selection and semantic mapping strategies based on the needs of different clinical scenarios. For prostate cancer radiotherapy, the extraction and processing subunit can focus on indicators closely related to target localization, such as bladder fullness, rectal contents status, and prostate centroid displacement; for lung cancer radiotherapy, it focuses on tumor location changes and lung function indicators under the influence of respiratory motion. The sequence processing subunit uses different statistical analysis methods for different indicators. For indicators exhibiting periodic changes, it calculates the fluctuation characteristics within the period; for indicators exhibiting monotonic trends, it focuses on analyzing the rate of change and cumulative offset. The semantic processing subunit configures multi-level semantic mapping rules, which can map the same magnitude of change to different semantic intensities based on clinical risk levels. For example, in routine monitoring scenarios, a small change in rectal volume might be mapped as "slight fluctuation"; while in high-risk scenarios approaching dose constraints, the same magnitude of change might be mapped as "significantly abnormal, requiring vigilance for overdose risk." The text processing subunit supports dynamic selection of multiple prompt word templates. Based on the patient's current treatment stage, cumulative dose level, and past response history, it selects the most suitable template structure, making the generated prompt text more targeted and clinically relevant. For example, for patients in the early stages of treatment, the template focuses on baseline change monitoring; for patients nearing the end of treatment, the template focuses on cumulative dose assessment and adverse reaction warnings. The embedding processing subunit can adapt the generated prompt text according to the word segmentation characteristics and embedding dimensions of different large language models, ensuring that the text embedding sequence can be effectively integrated with the numerical features of subsequent modules.

[0060] Optionally, target metrics can refer to the core characteristics that clinicians are most concerned with and that have the greatest impact on decision-making. For example, these could include bladder volume, rectal volume, and target volume D95% dose coverage. The selection of these metrics can be based on clinical experience and expert knowledge.

[0061] Optionally, the statistical enhancement prompt text can refer to a semantically rich natural language description that transforms the statistical trends of numerical features into a preset template. This statistical enhancement prompt text can enhance the expressiveness of the input information, enabling the model to understand the clinical meaning behind the data. For example, "In the past 5 fractions, the bladder volume has shown a sharp downward trend with extremely unstable fluctuations, and the current volume is 120cc."

[0062] Optionally, a text embedding sequence can represent a sequence of vectors obtained after inputting natural language text into the embedding layer of a large language model. These vectors are internal representations that the model can understand and process. For example, after a sentence is processed by a word segmenter, vectors corresponding to each word can be generated, and these vectors, arranged in order, can form a sequence.

[0063] The text processing unit can transform numerical changes into semantically rich Chinese descriptions that are familiar to clinicians. This allows the subsequent large language model to draw upon its pre-trained medical knowledge for analogy and reasoning, thereby making judgments that are more consistent with clinical logic and improving the interpretability of the decision-making process.

[0064] The decision unit is used to adaptively fuse the low-dimensional feature matrix and the text embedding sequence, and to use a pre-trained large language model for reasoning to output the trigger decision result of online adaptive radiotherapy and the corresponding explanatory text.

[0065] For example, the decision-making unit may include a weighted processing subunit, a concatenation processing subunit, an input processing subunit, an activation processing subunit, and an output processing subunit. The weighted processing subunit can be used to weight the low-dimensional feature matrix using gating parameters to obtain a weighted numerical embedding sequence. The gating parameters are learnable parameters that can be used to dynamically adjust the weights of the low-dimensional feature matrix in the adaptive multimodal fusion process based on the image quality of the cone-beam CT image. For example, in cases where the image contains metal artifacts or motion blur, the gating parameters can be set to smaller values ​​to reduce the fusion weights of the numerical features. The concatenation processing subunit can be used to concatenate the weighted numerical embedding sequence with the text embedding sequence along the sequence dimension to obtain a multimodal fusion sequence. The input processing subunit can be used to input the multimodal fusion sequence into a large language model, in which a hybrid expert structure can be embedded. The activation processing subunit can be used to activate at least one expert network based on the input features of the multimodal fusion sequence through a gating network with a hybrid expert structure. For example, for cases mainly presenting as changes in bladder volume, the gating network can primarily activate the first expert network responsible for handling scenarios related to bladder changes. The output processing subunit can be used to process the multimodal fusion sequence through the activated at least one expert network, outputting the triggering decision result for online adaptive radiotherapy and the corresponding explanatory text. The output processing subunit may include a label output module and a text output module. The label output module can be used to output binary labels, which indicate whether online adaptive radiotherapy is triggered or not. The text output module can utilize the generation capabilities of a large language model to output structured text including decision-making basis, target indicator analysis, and clinical recommendations.

[0066] For example, the decision-making unit can employ different expert network configurations and fusion strategies to process the low-dimensional feature matrix and text embedding sequence. For instance, the weighted processing subunit can set multiple gating parameters, each corresponding to the fusion weight of different feature channels in the low-dimensional feature matrix. For example, for geometric feature channels significantly affected by image quality, independent gating parameters can be set to automatically reduce the weight of these features when image quality deteriorates; for relatively robust clinical feature channels, higher fusion weights can be maintained. The stitching processing subunit can employ a non-linear fusion method, transforming the weighted numerical embedding sequence and the text embedding sequence separately before stitching, ensuring that the features of the two modalities are in a more suitable representation space before fusion. The input processing subunit can input the fused multimodal sequence into a large language model that has undergone low-rank adaptation fine-tuning. The large language model can embed a hybrid expert structure in multiple Transformer layers. The hybrid expert structure can include four expert networks, respectively used to handle scenarios related to bladder changes, rectal changes, multi-organ coupling changes, and basic scenarios with no significant changes. The gating network can calculate the matching degree between the multimodal fusion sequence and each expert network, and activate the corresponding expert network based on the matching degree. For complex cases with simultaneous bladder and rectal changes, the gating network can activate multiple expert networks simultaneously and weightedly combine their outputs. The label output module in the output processing subunit can output binary labels that trigger the decision. The binary labels can be represented as numerical values ​​of "1" or "0", where "1" represents "triggering online adaptive radiotherapy" and "0" represents "not triggering online adaptive radiotherapy". The text output module can generate structured explanatory text based on the autoregressive generation capability of the large language model. The explanatory text can include changes in key indicators on which the decision is based, specific values ​​exceeding dose constraints, and suggestions for subsequent clinical treatment, for clinicians to refer to when confirming decisions.

[0067] Optionally, the pre-trained large language model can be implanted with a hybrid expert structure, or with a low-rank adaptation module. This module can achieve domain adaptation of parameters by adding trainable low-rank matrices between model layers. Alternatively, a cue fine-tuning structure can be implanted to guide the large language model to focus on decision-making patterns in specific clinical scenarios by optimizing continuous cue vectors. Adapter modules can also be implanted to insert small neural network modules between frozen pre-trained model layers, which is beneficial for flexible switching between multiple scenarios and specialized reasoning.

[0068] Optionally, adaptive multimodal fusion can refer to dynamically adjusting the weights of information from different modalities in the fusion process based on data or context. For example, when the quality of the cone-beam computed tomography (CBCT) images on the same day is poor, leading to a decrease in the reliability of the extracted radiomics features, the fusion mechanism can automatically reduce the weight of numerical features and rely more on textual features that reflect historical trends for decision-making.

[0069] Optionally, the explanatory text can characterize the natural language description generated by the model to support its decision-making conclusions. This natural language description can not only explain the conclusions but also provide a basis for the decision, such as: "It is recommended to trigger a replanning. Reason: The rectal volume has shown a sharp upward trend due to gas accumulation in the past three sessions, causing the prostate to be pushed forward. The current predicted plan shows that the rectal V50 has exceeded the safety limit by 5%."

[0070] For example, the complete decision-making process of the LLM-based intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer provided in this application can start from multimodal data acquisition, proceed through feature dimensionality reduction, semantic transformation, and multimodal fusion, and finally output an interpretable decision result. For example, the acquisition unit can be responsible for constructing a high-dimensional multimodal temporal feature matrix. This multimodal temporal feature matrix contains radiomics, dosimetry, geometric, and clinical features of the target object within a preset time window L, with a total feature dimension M exceeding 1000. The feature processing unit can compress this high-dimensional multimodal temporal feature matrix into a low-dimensional feature matrix. Where K can be 64. The feature processing unit facilitates significant dimensionality reduction from thousand-dimensional features to tens-dimensional core features, and the number of trainable parameters of the feature processing unit is independent of the original feature dimension M. Meanwhile, the text processing unit can process data in parallel. The text processing unit can process multimodal temporal feature matrices... The statistical trends of the target indicators are extracted, the numerical change patterns are transformed into Chinese medical semantic descriptions, statistically enhanced prompt text is generated, and the statistically enhanced prompt text is converted into a text embedding sequence through the embedding layer of a large language model. The decision unit is responsible for processing the low-dimensional feature matrix output by the feature processing unit. Text embedding sequences generated by the text processing unit A multimodal fusion sequence is formed by fusion through an adaptive multimodal fusion mechanism. This multimodal fusion sequence can simultaneously contain precise information from numerical features and clinical interpretability from semantic features. The fused multimodal sequence is input into a large language model embedded with a hybrid expert structure. The decision-making unit dynamically activates the corresponding expert network to perform reasoning based on the patterns of the input features. Finally, the decision-making unit outputs the trigger decision for online adaptive radiotherapy and the corresponding explanatory text. This multi-unit collaborative design helps the model achieve accurate decision-making with minimal sample size and possesses clinical interpretability.

[0071] like Figure 3 As shown in the embodiments of this application, the decision-making method for an LLM-based intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer can be summarized as six modules (or six stages), specifically including:

[0072] Module 1 (Data Input): The acquisition unit is responsible for data input, acquiring the historical L fractions of data for the target object within a preset time window, as well as the imaging and clinical data of the current fraction t. The historical L fractions of data include imaging and clinical records prior to multiple past fractions of treatment, while the current fraction t data includes cone-beam computed tomography (CBCT) images and real-time clinical information prior to this treatment.

[0073] Module 2 (High-Dimensional Multimodal Feature Construction): The acquisition unit constructs high-dimensional multimodal features by extracting four types of features: radiomics features, dosimetry features, geometric features, and clinical features. The acquisition unit concatenates these features at each time step to form a multimodal temporal feature matrix X. This matrix can have dimensions L×M, where M is the total feature dimension and is greater than or equal to 1000. This matrix is ​​then processed in parallel via a dual-track processing path.

[0074] Module 3 (Numerical Trajectory Compression): In the numerical trajectory compression path, the feature processing unit processes the multimodal temporal feature matrix. The convolution processing subunit extracts the temporal trend of each feature through channel-independent causal convolution, and the computation processing subunit performs sparse bottleneck aggregation based on the latent query vector, key matrix, and value matrix using a cross-attention mechanism, finally outputting a low-dimensional feature matrix. The matrix dimension of this low-dimensional feature matrix can be... Where K is the preset number of key features. It can be the embedding dimension of a large language model.

[0075] Module 4 (Semantic Statistical Enhancement): In the semantic enhancement path, the text processing unit processes the multimodal temporal feature matrix. The extraction processing subunit extracts the time series of the target indicators from the multimodal temporal feature matrix. The sequence processing subunit calculates the statistics of the time series of each target indicator, including trend slope, variance of fluctuation, and current state value. The semantic processing subunit transforms the numerical statistics into semantic descriptions through trend semantic mapping, for example, mapping an upward trend to a description such as "sharp rise." The text processing subunit fills the trend semantic description and stability semantic description into a preset prompt word template, generating statistically enhanced prompt text. The embedding processing subunit inputs the statistically enhanced prompt text into the embedding layer of the large language model to obtain the text embedding sequence. .

[0076] Module 5 (Fusion and Hybrid Expert Model Inference): The decision-making unit fuses and infers the results of the dual-track processing. The weighted processing subunit weights the low-dimensional feature matrix using an adaptive gating fusion parameter α. The concatenation processing subunit concatenates the weighted numerical embedding sequence and the text embedding sequence along the sequence dimension to obtain a multimodal fusion sequence. The input processing subunit inputs the multimodal fusion sequence into a large language model embedded with a hybrid expert structure. The activation processing subunit activates at least one expert network for lightweight inference based on the multimodal fusion sequence through the gating network of the hybrid expert structure.

[0077] Module Six (Decision and Interpretation): The output processing subunit outputs the final results. The label output module outputs the trigger decision result for online adaptive radiotherapy, i.e., a binary classification result indicating whether online adaptive radiotherapy should be triggered. The text output module outputs structured Chinese clinical interpretation text based on the generation capabilities of a large language model. The above results are provided for clinicians' reference to assist them in making final decisions.

[0078] In some optional embodiments, the feature processing unit includes: a convolution processing subunit, an aggregation processing subunit, a mapping processing subunit, a computation processing subunit, and a linear transformation processing subunit. The convolution processing subunit performs causal convolution operations on each feature channel in the multimodal temporal feature matrix to obtain a convolutional feature matrix. All feature channels in the multimodal temporal feature matrix share the same convolution kernel. In the causal convolution operation, the output of the current time step depends on the inputs of the current time step and previous time steps. The aggregation processing subunit performs temporal aggregation on the convolutional feature matrix along the time dimension to obtain a scalar feature vector. The mapping processing subunit maps the scalar feature vector to a key matrix and a value matrix using a shared projection vector. Each row of the key matrix and the value matrix corresponds to a mapping representation of a feature channel. The computation processing subunit performs cross-attention calculation on the key matrix and the value matrix using a preset latent query vector to obtain a bottleneck feature matrix. The linear transformation processing subunit performs a linear transformation on the bottleneck feature matrix using an alignment matrix, making the vector dimension of the transformed features consistent with the word vector dimension of the large language model, thus obtaining a low-dimensional feature matrix.

[0079] For example, for a feature channel representing bladder volume, the convolution processing unit can use its values ​​at multiple time steps within a time window as an input sequence. It then extracts the local change trend of this feature channel in the time dimension through causal convolution operations. The convolution kernel is shared across all feature channels, making the number of trainable parameters independent of the total number of feature channels. The aggregation processing unit can use average pooling to aggregate multiple values ​​of each feature channel within a time window into a representative scalar value, thereby compressing the temporal information into a static representation. The shared projection vector can include a key projection vector and a value projection vector. The key projection vector and the value projection vector have the same dimension and are independent of the number of feature channels. The mapping processing unit can multiply each scalar element in the scalar feature vector with the key projection vector element-wise to obtain the vector representation of the corresponding row in the key matrix, and then multiply each scalar element with the value projection vector element-wise to obtain the vector representation of the corresponding row in the value matrix. Each row of the key matrix and the value matrix corresponds to a mapping representation of a feature channel. The number of pre-defined potential query vectors is much smaller than the original feature dimension of the multimodal temporal feature matrix. Each potential query vector can serve as a tool for extracting key information from the high-dimensional feature space. The computational processing subunit can calculate the similarity between each potential query vector and each row in the key matrix, and then perform weighted aggregation on each row in the value matrix based on the similarity to obtain the corresponding row vector in the bottleneck feature matrix. The dimension of the alignment matrix is ​​jointly determined by the vector dimension of the bottleneck feature matrix and the word vector dimension of the large language model.

[0080] Optionally, the mapping processing unit can map the aggregated scalar feature vectors into key and value matrices using shared projection vectors. Each row of the key and value matrices corresponds to a mapping representation of a feature channel, and since the projection vectors are shared across all feature channels, the number of trainable parameters in this step remains constant. The computation processing unit can perform cross-attention computation on the key and value matrices using a preset number of potential query vectors. The number of potential query vectors can be adjusted according to the level of granularity required for clinical decision-making; more potential query vectors can be set for complex cases requiring more comprehensive evaluation, while fewer can be set for rapid screening scenarios.

[0081] like Figure 4 As shown, for example, the feature processing unit performs channel-independent temporal feature extraction and sparse bottleneck aggregation on the multimodal temporal feature matrix to obtain a low-dimensional feature matrix. The process may include numerical temporal modeling, sparse bottleneck aggregation based on cross-attention, and dimensional alignment to the large language model input space.

[0082] The purpose of the feature processing unit is to compress massive high-dimensional features into a small number of key features with extremely low parameter counts and align them to the input space of a large language model. In numerical temporal modeling, the feature processing unit can receive the multimodal temporal feature matrix output by the acquisition unit as input. The convolution processing subunit can perform one-dimensional causal convolution on each of the M feature channels in the multimodal temporal feature matrix, i.e., channel-independent scalar causal convolution, to obtain the convolutional feature matrix. The convolution kernel size can be k. The temporal trend of each feature is extracted, and all feature channels in the multimodal temporal feature matrix share the same micro-convolution kernel. Subsequently, the aggregation processing subunit performs temporal average pooling along the time dimension on the convolutional feature matrix to obtain the aggregated scalar feature vector. Causal convolution is a convolution operation that guarantees strict temporal unidirectional dependency. The core constraint of causal convolution is that the output of the current time step can only depend on the input of the current and past time steps and cannot use future information. The reason for using causal convolution in the online adaptive radiotherapy scenario for prostate cancer is that clinical decisions must be based on historical data for prediction and cannot use information from future fractions. Causal convolution can be constrained by padding the input sequence with zeros at k-1 positions on the left. This helps ensure that the convolution window at each time step only covers the historical data at that time step and before, thereby extracting trend features that conform to the temporal causal relationship.

[0083] For the i-th feature channel (i=1,2……M), its time series is: The convolution processing subunit can perform causal convolution, as shown in formula (1):

[0084] Formula (1);

[0085] in, This can refer to the original time series of the i-th feature, with dimension L, representing the value of the feature over L time steps within a preset time window. For example, if the feature is bladder volume, and L=5, then this vector contains the bladder volume measurements from the most recent 5 time steps. This can refer to a shared scalar convolutional kernel containing k learnable parameters; for example, when k=3, the convolutional kernel is... This convolutional kernel is shared across all M feature channels and does not perform specialized learning for specific features, thus ensuring that the number of parameters is independent of M. This can refer to a one-dimensional causal convolution operator. For time step t, its output value is... This is the weighted sum of the current time and the previous k-1 time steps. For example, when k=3 and t=4, Only historical information for t=2,3,4 is used. This can refer to a time series after causal convolution, still with a dimension of L. The output value at each time step incorporates local trend information from that moment and previous times. For example, if the original sequence shows a downward trend, the sequence after causal convolution will strengthen this trend feature through weighted combination.

[0086] After performing the above causal convolution operation on all M feature channels, the convolutional processing subunit obtains the complete convolutional feature matrix. .

[0087] Parameter analysis: Regardless of the feature dimension M, the number of parameters in the shared convolution kernel is always fixed at k, which is conducive to achieving complete decoupling between the number of parameters and the feature dimension and helps to avoid overfitting on small samples.

[0088] The feature processing unit can use the cross-attention mechanism to extract K key information from M high-dimensional features, where K is much smaller than M.

[0089] First, a scalar-to-vector mapping can be performed. Since the attention mechanism needs to calculate similarity in a vector space, the scalar features after causal convolution need to be mapped to a smaller vector space. The aggregation processing subunit can perform temporal aggregation on the convolutional feature matrix to obtain a scalar feature vector. The aggregation processing subunit performs temporal aggregation on the convolutional feature matrix... For time-series aggregation, such as average pooling, refer to formula (2):

[0090] Formula (2);

[0091] in, It can refer to the scalar feature vector output by the aggregation processing subunit, which contains... Each of the scalar elements corresponds to the average level of a raw feature channel over the entire time window. It can refer to the convolutional feature matrix output by the convolutional processing subunit. It can represent an average pooling operation along a specified dimension. This indicates aggregation along the time dimension (i.e., the row direction of the matrix).

[0092] In the sparse bottleneck aggregation process based on cross-attention, the mapping processing subunit can map scalar feature vectors into key and value matrices through shared projection vectors. The mapping processing subunit uses shared scalar projection weights to map each feature value to a small vector space, as shown in formulas (3) and (4):

[0093] Formula (3);

[0094] Formula (4);

[0095] in, It can refer to the feature vector after temporal aggregation, containing M scalar elements. Average pooling can be applied to the convolutional feature matrix. Aggregate along the time dimension. This can refer to a shared projection vector, with dimensions of... For example, 8-dimensional. These two vectors can be learnable parameters, used to generate the key matrix and value matrix respectively. The key feature is that all M features share the same set of projection weights, rather than each feature having an independent projection matrix, which ensures that the number of parameters is completely independent of M. This can refer to the broadcast operation symbol after element-wise multiplication. The specific calculation process is as follows: For The i-th scalar element scalar elements and projection vector Perform element-wise multiplication to obtain a dimensional vector By performing this operation on all M scalar elements, the key matrix is ​​obtained. For example, if (8-dimensional), then . This can refer to the mapped key matrix and value matrix. Each row of the key matrix corresponds to a mapping representation of a feature channel, used to calculate the similarity with the query vector; each row of the value matrix corresponds to a mapping representation of a feature channel, used for weighted aggregation to obtain the final output.

[0096] Parameter analysis: The number of parameters for the shared projection vector is This is completely independent of the feature dimension M. Regardless of whether the feature dimension M is 1000 or 10000, the number of parameters remains constant. This is beneficial for achieving dimensionality reduction of parameter features.

[0097] Then, cross-attention calculation can be performed. The computational processing subunit is used to perform cross-attention calculation on the key matrix and value matrix using a preset latent query vector to obtain the bottleneck feature matrix. The computational processing subunit can introduce learnable latent query vectors. For example, 64×8 can be used as a tool to extract key information from a high-dimensional feature space.

[0098] The computational processing subunit performs scaled dot product attention:

[0099] Formula (5);

[0100] in, It can refer to the preset potential query vector used by the computational processing subunit. It can refer to the key matrix output by the mapping processing subunit. It can refer to the value matrix output by the mapping processing subunit. The dot product of the latent query vectors and the key matrix can be used to obtain the attention score matrix, which represents the degree of attention paid by the K latent query vectors to the M features. Dimensional changes can be: Among them, dimension parameters The preset number of key features, Let be the dimension of the small vector space. This represents the original feature dimension. This can represent the scaling factor, which is the square root of the dimension of the small vector space, divided by the attention score. It helps maintain the stability of the gradient during training.

[0101] Softmax performs normalization along the M-dimensional plane, ensuring that the sum of the attention weights for each potential query vector is 1. After softmax, the dimension remains the same. The attention weights calculated using Softmax are multiplied by the value matrix to finally output the bottleneck feature matrix. These are the K aggregated bottleneck feature vectors. The dimensionality can be changed as follows: .

[0102] Parameter analysis: Potential query vector The number of parameters is It is independent of the feature dimension M.

[0103] During the process of dimensional alignment to the input space of a large language model, the linear transformation processing subunit can perform a linear transformation on the bottleneck feature matrix through the alignment matrix, so that the vector dimension of the transformed features is consistent with the word vector dimension of the large language model, thus obtaining a low-dimensional feature matrix.

[0104] The linear transformation processing subunit maps the bottleneck feature matrix to the input dimension of the large language model:

[0105] Formula (6);

[0106] in: It can refer to the alignment matrix used by the linear transformation processing subunit, such as . It can represent the embedding dimension of a large language model, that is, the word vector dimension of the large language model, or the dimension of the output vector of the embedding layer of the large language model, such as 768 or 1024. It can refer to the low-dimensional feature matrix of the final output of the linear transformation processing subunit.

[0107] Total parameter count: Shared convolution kernel: k parameters, such as 3.

[0108] Shared projection vector: There are 16 parameters, for example.

[0109] Potential query vector: There are 512 parameters.

[0110] Alignment matrix: There are 6144 parameters.

[0111] There are approximately 6700 parameters in total, which are completely independent of the feature dimension M.

[0112] Traditional fully connected layers require 128,000 parameters to map 1000-dimensional features to 128-dimensional features. With 100 patients (approximately 2500 training samples), the parameter-to-sample ratio is as high as 51:1, leading to severe overfitting. This application's embodiment combines channel-independent convolution (3 parameters), shared projection (16 parameters), latent query (512 parameters), and alignment matrix (6144 parameters), totaling approximately 6700 parameters, reducing the parameter-to-sample ratio to 2.7:1. Furthermore, this number of parameters is independent of the feature dimension M; even if the feature dimension increases to 2000 or 3000 dimensions, the number of parameters remains unchanged.

[0113] In this embodiment, the feature processing unit can achieve parameter compression from high-dimensional temporal features to low-dimensional bottleneck representation. The input multimodal temporal feature matrix... First, the temporal trends of each feature are extracted using channel-independent causal convolution, resulting in the convolutional feature matrix. At this point, the temporal information is encoded, but the feature dimension remains unchanged. Subsequently, temporal aggregation compresses the time dimension into a single representative value, yielding a scalar feature vector. It contains M scalar features. These M scalar features are mapped to a small vector space through a shared projection vector, forming a key matrix and a value matrix. Then, K learnable latent query vectors selectively extract key information from the M features through a cross-attention mechanism, outputting a bottleneck feature matrix. This facilitates a significant dimensionality reduction from M dimensions to K dimensions. Finally, the dimension of the bottleneck feature matrix is ​​expanded to the input space of the large language model using an alignment matrix, resulting in a low-dimensional feature matrix. It can transform raw high-dimensional temporal features into low-dimensional dense representations compatible with large language models. Throughout the process, the number of parameters remains small and is completely decoupled from the input feature dimension M.

[0114] For example, the feature processing unit can be viewed as an intelligent filter. Faced with thousands of original feature dimensions in a multimodal temporal feature matrix, related technologies may use fully connected layers for direct mapping, potentially leading to a quadratic increase in the number of trainable parameters with increasing feature dimension M. The feature processing unit, through channel-independent causal convolution operations performed by the convolution processing subunit and cross-attention computation mechanisms based on latent query vectors performed by the computation processing subunit, utilizes a predetermined number of latent query vectors as leverage to selectively extract and aggregate the most informative content from massive features. This helps ensure that the number of trainable parameters of the feature processing unit remains relatively small regardless of how much the feature dimension M increases. Therefore, the feature processing unit is particularly suitable for online adaptive radiotherapy-triggered decision-making scenarios with a limited number of training samples.

[0115] In some alternative embodiments, the shared projection vector includes a key projection vector and a value projection vector, wherein the key projection vector and the value projection vector have the same dimension and are independent of the number of feature channels.

[0116] For example, when mapping scalar feature vectors to key and value matrices, the mapping processing subunit can configure a set of shared projection vectors. The key projection vector can be used to map each scalar element in the scalar feature vector to a vector representation of the corresponding row in the key matrix, and the value projection vector can be used to map each scalar element to a vector representation of the corresponding row in the value matrix. The dimensions of the key and value projection vectors can be preset, for example, to 8, 16, or 32 dimensions. This setting depends on the requirements of subsequent cross-attention calculations and computational resource limitations, but is completely independent of the number of feature channels contained in the multimodal temporal feature matrix. Regardless of whether the feature dimension in the multimodal temporal feature matrix is ​​1000, 2000, or higher, the dimensions of the key and value projection vectors remain unchanged, and the key and value projection vectors themselves are shared across all feature channels. For the scalar element corresponding to the first feature channel and the scalar element corresponding to the 1000th feature channel, the mapping processing subunit uses the exact same key projection vector for mapping operations to obtain the key vector of the corresponding dimension. Since the dimensions of the key projection vector and the value projection vector are independent of the number of feature channels and are shared among all feature channels, the number of trainable parameters in this mapping step is strictly controlled to be the sum of the dimensions of the two projection vectors. This has no relation to the original feature dimensions, which helps to control the total size of trainable parameters while maintaining the model's expressive power, and is beneficial for adapting to small sample training scenarios.

[0117] Optionally, the key projection vector can refer to a shared vector that maps scalar feature values ​​to keys in the attention mechanism. The role of the key projection vector is to generate a vector representation for each feature to be queried, determining the visibility or matching degree of that feature under various queries. The value projection vector can refer to a shared vector that maps scalar feature values ​​to values ​​in the attention mechanism. The role of the value projection vector is to generate a vector representation for each feature to be weighted and aggregated, determining what specific numerical information is provided to the bottleneck layer when that feature is selected.

[0118] Optionally, having the same dimension for the key projection vector and the value projection vector means that both vectors have the same dimension. This can be a basic requirement for scaling dot product attention calculations, allowing the query vector to match the key vector dimension, and enabling similarity calculations. Being independent of the number of feature channels means that the number of parameters in the key projection vector and the value projection vector is fixed and does not increase with the number of input features M. This is beneficial for handling high-dimensional features and avoiding parameter explosion.

[0119] In some optional embodiments, the text processing unit includes: an extraction processing subunit for extracting a time series sequence of at least one target indicator from a multimodal time series feature matrix; a sequence processing subunit for calculating the trend slope and variance of the time series sequence of each target indicator; a semantic processing subunit for mapping the trend slope of each target indicator to a trend semantic description and mapping the variance of each target indicator to a stability semantic description; a text processing subunit for filling the trend semantic description and stability semantic description into a preset prompt word template to generate statistically enhanced prompt text; and an embedding processing subunit for inputting the statistically enhanced prompt text into the embedding layer of a large language model to obtain a text embedding sequence, wherein the text embedding sequence is a vector sequence.

[0120] For example, the extraction processing subunit can be used to extract the time series sequence of at least one target indicator from the multimodal time series feature matrix. The target indicator may include clinical indicators closely related to target location and dose distribution, such as bladder volume, rectal volume, and target coverage. The extraction processing subunit can locate the feature channel corresponding to each target indicator from the multimodal time series feature matrix and extract the value of that feature channel at each time step within a preset time window, thus forming the time series sequence of the target indicator. The sequence processing subunit can be used to calculate the trend slope and variance of the time series sequence of each target indicator. The trend slope can be calculated using linear fitting and is used to characterize the overall direction and rate of change of the target indicator within the time window. The variance can be used to characterize the dispersion and stability level of the target indicator within the time window. The semantic processing subunit can be used to map the trend slope of each target indicator to a trend semantic description and the variance of each target indicator to a stability semantic description. The semantic processing subunit can be configured with preset mapping rules, such as mapping the trend slope of different value ranges to trend semantic descriptions like "sharply rising," "gradually rising," "basically stable," "gradually falling," or "sharply falling," and mapping the fluctuation variance of different value ranges to stability semantic descriptions like "extremely unstable," "relatively volatile," or "relatively stable." The text processing subunit can be used to fill the trend and stability semantic descriptions into preset prompt word templates to generate statistically enhanced prompt text. The prompt word templates can contain structured paragraphs such as patient treatment trend analysis and current dose assessment. The filled prompt text can describe the changes of the target indicator within a time window in natural language. The embedding processing subunit can be used to input the statistically enhanced prompt text into the embedding layer of the large language model to obtain a text embedding sequence, where the text embedding sequence is a vector sequence. The embedding layer of the large language model can convert each word in the statistically enhanced prompt text into a corresponding vector representation, forming a text embedding sequence for subsequent decision-making units to perform fusion processing.

[0121] For example, the text processing unit can process target indicators in a differentiated manner according to different clinical scenarios and treatment stages. For instance, for prostate cancer patients at different radiotherapy stages, the extraction processing subunit can extract different combinations of target indicators. In the early stages of treatment, the extraction processing subunit can focus on extracting indicators reflecting baseline status, such as bladder volume and rectal volume; in the middle stages of treatment, it can focus on extracting indicators reflecting cumulative effects, such as target coverage and dose parameters of organs at risk; and in the later stages of treatment, it can focus on extracting indicators reflecting the risk of adverse reactions, such as rectal wall thickness and bladder wall edema. The sequence processing subunit can use different statistical calculation methods for different indicators. For indicators exhibiting periodic fluctuations, such as target location affected by respiratory motion, the sequence processing subunit can first perform periodic decomposition on the time series and then calculate the trend slope after removing the periodic component. For time series with missing or outlier values, the sequence processing subunit can first perform data interpolation or outlier removal before calculating the trend slope and variance of fluctuation. The semantic processing subunit can configure multi-level mapping rules based on clinical guidelines and expert experience. For example, for the target indicator of rectal volume, in the early stages of treatment when the cumulative dose is low, the same increase can be represented as "gradual increase"; in the later stages of treatment when the cumulative dose is close to the constraint limit, the same increase can be represented as "significant increase, requiring attention to overdose risk". The text processing subunit can select an appropriate cue word template based on the treatment stage. In the early stages of treatment, a template focusing on baseline change monitoring can be selected; in the middle stages of treatment, a template focusing on dose assessment can be selected; and in the later stages of treatment, a template focusing on warning of adverse reactions can be selected. The text processing subunit fills the selected cue word template with trend semantic descriptions and stability semantic descriptions to generate statistically enhanced cue text adapted to the current treatment stage. The embedding processing subunit can convert the statistically enhanced cue text into a corresponding text embedding sequence based on the word segmentation rules and embedding dimensions of different large language models. For example, for large language models using different vocabulary lists, the embedding processing subunit can use the corresponding word segmenter to segment the statistically enhanced cue text, and then input the segmentation results into the embedding layer of the corresponding large language model to obtain a text embedding sequence.

[0122] Optionally, the time series of the target indicator can refer to a sequence consisting of the values ​​of a specific target indicator at various time points within a preset time window. For example, if the target indicator is bladder volume, its corresponding time series could be [150cc, 145cc, 140cc, 138cc, 135cc], representing the values ​​of the past 5 measurements.

[0123] Optionally, the trend slope quantifies whether the target indicator rises, falls, or remains stable within a time window. For example, a straight line can be fitted using the least squares method, and the slope of this line is the trend slope. A positive slope indicates an upward trend, a negative slope indicates a downward trend, and the absolute value indicates the degree of change. Variance can quantify the stability of the target indicator within a time window. For example, the standard deviation or coefficient of variation of the series can be calculated. The larger the variance, the more unstable the indicator and the more volatile its fluctuations.

[0124] Optionally, the preset prompt template can refer to a pre-designed natural language text frame with blank spaces. This prompt template can specify the organization and description logic of the information, ensuring consistency in format for each generated prompt text. For example, a template could be: "In the past [L] times, [Indicator A] exhibits [trend description], its fluctuations are described as [stability description], and its current value is [Value]".

[0125] Optionally, the embedding layer can refer to a component of a large language model, located at the very front of the model, mapping each word in the input to a dense vector called a word embedding. A text embedding sequence can refer to the sequence of vectors output by the embedding layer, arranged in the order of the input text. ,in, It can refer to the number of lexical units in a text. It can refer to the dimension of word vectors.

[0126] In some optional embodiments, the target metrics include at least one of bladder volume, rectal volume, and target coverage.

[0127] For example, when extracting the time series of target indicators from the multimodal temporal feature matrix, the extraction processing subunit can select appropriate indicator combinations based on the clinical focus of prostate cancer radiotherapy. Bladder volume characterizes changes in bladder fullness; a decrease in bladder volume may lead to displacement of the prostate target area. Rectal volume characterizes changes in the state of rectal contents; an expansion of rectal volume may cause the prostate target area to shift forward. Target coverage characterizes the degree of agreement between the planned dose and the actual dose received; a decrease in target coverage may indicate the need for re-optimization of the radiotherapy plan.

[0128] Optionally, the extraction processing subunit can simultaneously extract bladder volume, rectal volume, and target coverage as target indicators to comprehensively assess the necessity of triggering online adaptive radiotherapy on the same day.

[0129] Optionally, for patient groups where rectal pneumoperitoneum is common, the extraction and processing subunit can focus on extracting rectal volume as the target indicator; for patient groups with poor bladder function, bladder volume can be focused on extracting bladder volume as the target indicator. The extraction and processing subunit locates the feature channel corresponding to each target indicator from the multimodal temporal feature matrix, and extracts the value of the feature channel at each time step within a preset time window to form the temporal sequence of the target indicator, which is then used by subsequent subunits for statistical analysis and semantic mapping.

[0130] In one optional embodiment, the semantic processing subunit includes: a first trend semantic processing module, configured to map a trend semantic description to first trend semantic information when the trend slope is greater than a first preset threshold, wherein the first trend semantic information describes that the target indicator is rising and the rising rate is greater than a preset rate; a second trend semantic processing module, configured to map a trend semantic description to second trend semantic information when the trend slope is greater than a second preset threshold and less than or equal to a first preset threshold, wherein the second trend semantic information describes that the target indicator is rising and the rising rate is less than or equal to a preset rate; and a third trend semantic processing module, configured to map a trend semantic description to second trend semantic information when the trend slope is greater than or equal to a third preset threshold and less than or equal to a first preset threshold. Under the condition of a second preset threshold, the trend semantic description is mapped to third trend semantic information, which is used to describe that the target indicator is maintained within the target range; the fourth trend semantic processing module is used to map the trend semantic description to fourth trend semantic information when the trend slope is greater than or equal to the fourth preset threshold and less than the third preset threshold, which is used to describe that the target indicator is decreasing and the decreasing speed is less than or equal to a preset speed; the fifth trend semantic processing module is used to map the trend semantic description to fifth trend semantic information when the trend slope is less than the fourth preset threshold, which is used to describe that the target indicator is decreasing and the decreasing speed is greater than a preset speed.

[0131] For example, for the target indicator of rectal volume, when the calculated trend slope is greater than a first preset threshold, the first trend semantic processing module can map its trend semantic description as "sharply increasing". When the trend slope is between the second and first preset thresholds, the second trend semantic processing module can map it as "gradually increasing". When the trend slope is between the third and second preset thresholds, the third trend semantic processing module can map it as "basically stable". When the trend slope is between the fourth and third preset thresholds, the fourth trend semantic processing module can map it as "gradually decreasing". When the trend slope is less than the fourth preset threshold, the fifth trend semantic processing module can map it as "sharply decreasing". Through the collaborative work of the above multiple trend semantic processing modules, the semantic processing subunit can discretize the trend slope of continuously measured values ​​into multiple levels with clinical semantic meaning, so that the generated statistically enhanced prompt text can accurately describe the changing trend of the target indicator in natural language, making it easier for clinicians to understand and judge.

[0132] In an optional embodiment, the semantic processing subunit further includes: a coefficient processing module, configured to determine the coefficient of variation based on the variance and mean; a first stability semantic processing module, configured to map the stability semantic description to first stability semantic information when the coefficient of variation is greater than a first variation threshold, the first stability semantic information describing that the fluctuation amplitude of the target indicator is greater than a preset fluctuation threshold; a second stability semantic processing module, configured to map the stability semantic description to second stability semantic information when the coefficient of variation is greater than a second variation threshold and less than or equal to the first variation threshold, the second stability semantic information describing that the fluctuation amplitude of the target indicator is less than or equal to the preset fluctuation threshold, and the fluctuation amplitude of the target indicator is not maintained within a preset range; and a third stability semantic processing module, configured to map the stability semantic description to third stability semantic information when the coefficient of variation is less than or equal to the second variation threshold, the third stability semantic information describing that the fluctuation amplitude of the target indicator is maintained within a preset range.

[0133] Optionally, the coefficient of variation can be used to eliminate the influence of different indicator dimensions, facilitating stability comparisons of target indicators at different scales. For the target indicator of bladder volume, when the calculated coefficient of variation is greater than the first variation threshold, the first stability semantic processing module can map its stability semantic description as "extremely unstable." When the coefficient of variation is between the second and first variation thresholds, the second stability semantic processing module can map it as "relatively volatile." When the coefficient of variation is less than or equal to the second variation threshold, the third stability semantic processing module can map it as "relatively stable." Through the collaborative work of the above multiple stability semantic processing modules, the semantic processing subunit can discretize the coefficient of variation into multiple levels with clinical semantic meaning, enabling the generated statistically enhanced prompt text to accurately describe the stability level of the target indicator within a time window in natural language form. This helps clinicians assess the degree of fluctuation in organ filling status and its impact on radiotherapy planning.

[0134] like Figure 5 As shown, the text processing unit can transform the statistical trends of numerical features into medical semantic descriptions that can be understood by a large language model. The processing flow of the text processing unit can include statistical calculation, semantic mapping, template filling, and embedding generation.

[0135] During statistical calculations, the text processing unit can receive raw feature values ​​from the multimodal time-series feature matrix as input. The extraction processing subunit is used to extract the time-series sequence of at least one target indicator from the multimodal time-series feature matrix. The extraction processing subunit can select core indicators of clinical interest as target indicators, such as bladder volume, rectal volume, target area coverage (D95%), bladder V40, and rectal V50. The sequence processing subunit can calculate statistics for the time-series sequence of each target indicator, which can include three core statistics: trend slope, used to characterize the direction and rate of change of the target indicator within the time window; coefficient of variation, used to characterize the fluctuation stability of the target indicator; and current state value, i.e., the target indicator value of the current fraction. The sequence processing subunit can calculate the trend slope and variance of the time-series sequence of each target indicator. For the j-th target indicator, the sequence processing subunit calculates its time-series statistics within a preset time window L:

[0136] The trend slope can be fitted using the least squares method to fit a linear trend, as shown in formula (7):

[0137] Formula (7);

[0138] in, It can refer to the value of the j-th target indicator at time step t. Used to refer to the time step index within a time window. Used to refer to the index of all time steps within a time window. The arithmetic mean, Used to refer to Each target indicator within the time window The arithmetic mean of all values ​​within the range. It can refer to the first Each target indicator within the preset time window The slope of the trend within.

[0139] The variance of fluctuation can be calculated by standard deviation or coefficient of variation, as shown in formula (8):

[0140] Formula (8);

[0141] in, It can refer to the first Each target indicator within the preset time window The variance within the range, It can refer to the first The deviation of the indicator value at each time step from the window mean. The current state value can refer to the target indicator value at the current time step. .

[0142] During the semantic mapping process, the semantic processing subunit can transform numerical statistics into clinical semantic descriptions using a semantic mapping function. For example, the semantic processing subunit can map the trend slope of each target indicator to a trend semantic description, and the variance of each target indicator to a stability semantic description. The semantic processing subunit can also map numerical statistics to Chinese clinical semantic descriptions. The current state value can be retained as numerical information for subsequent template filling.

[0143] For example, for the trend slope, a mapping function can be defined as shown in formula (9):

[0144] Formula (9);

[0145] in, It can refer to a mapping function used to convert numerical trend slopes into discrete Chinese clinical semantic descriptions. It can represent the trend slope calculated by the sequence processing subunit.

[0146] For the variance of fluctuation, a mapping function can be defined, as shown in formula (10):

[0147] Formula (10);

[0148] in, It can represent a mapping function used to convert a volatility measure based on the coefficient of variation into a discrete Chinese clinical semantic description. CV can refer to the coefficient of variation determined by the coefficient processing module based on the variance of the volatility and the mean.

[0149] During the prompt word template population process, the text processing subunit can populate the preset prompt word template with semantically meaningful statistical descriptions such as trend semantic descriptions, stability semantic descriptions, and current state values. A template example is as follows:

[0150] [Analysis of Patient Treatment Stages]

[0151] In the past [L] fractions: bladder volume shows a [trend description] (slope = [Slope]), fluctuations are described as [stability description], and the current volume is [Value] cc. Rectal volume shows a [trend description] (slope = [Slope]), fluctuations are described as [stability description], and the current volume is [Value] cc. Prostate target area centroid displacement shows a [trend description], and the current displacement relative to the planned CT is [X,Y,Z] mm.

[0152] [Current Predicted Dosage Assessment]

[0153] Target coverage: D95% = [Value] Gy (Prescription dose [Prescription] Gy). Bladder dose constraint: V40 = [Value]% (Limit]%). Rectal dose constraint: V50 = [Value]% (Limit]%).

[0154] Based on the above information, please determine whether it is necessary to trigger an online adaptive rescheduling today.

[0155] During the embedding generation process, the embedding processing subunit can input the padded statistically enhanced prompt text into the word segmenter and embedding layer of the large language model. The word segmenter can segment the prompt text into a sequence of tokens, and the embedding layer can map each token to a corresponding vector representation to obtain the text embedding sequence.

[0156] Formula (11);

[0157] in, It can refer to the embedding layer of a large language model. This can refer to the statistical enhancement prompt text entered. This can refer to the number of lexical units in the prompt text. It can refer to the embedding dimension of a large language model.

[0158] While large language models possess reasoning capabilities, they may not be sensitive enough to purely numerical sequences. For example, given the numerical sequence "150, 140, 130, 120", a large language model might not directly understand its clinical significance. However, through text processing, the numerical sequence can be transformed into statistically enhanced cue text such as "bladder volume has drastically decreased and is highly unstable over the past four fractions." The large language model can then utilize its pre-trained medical knowledge to understand that this change may lead to a significant shift in the relative position of the prostate, thereby affecting target dose coverage. The text processing unit can then translate the measurement data into professional language used by clinicians.

[0159] In one optional embodiment, the decision unit includes: a weighted processing subunit, used to weight the low-dimensional feature matrix using gating parameters to obtain a weighted numerical embedding sequence, wherein the gating parameters are used to dynamically adjust the weights of the low-dimensional feature matrix in the adaptive multimodal fusion process according to the image data quality; a concatenation processing subunit, used to concatenate the weighted numerical embedding sequence and the text embedding sequence along the sequence dimension to obtain a multimodal fusion sequence; an input processing subunit, used to input the multimodal fusion sequence into a large language model, wherein a hybrid expert structure is embedded in the large language model; an activation processing subunit, used to activate at least one expert network according to the multimodal fusion sequence through a gating network of the hybrid expert structure; and an output processing subunit, used to process the multimodal fusion sequence through at least one expert network to output the trigger decision result for online adaptive radiotherapy and the explanatory text corresponding to the trigger decision result.

[0160] Optionally, the gating parameter can be a learnable scalar parameter, jointly optimized with other parameters of the entire decision unit during model training. When cone-beam CT images contain metal artifacts or motion blur, the weighted processing subunit can be adjusted to a smaller gating parameter value during training, thereby reducing the contribution of numerical features in the fusion process. The concatenation processing subunit can concatenate in the order of numerical embedding followed by text embedding to form a unified sequence containing both types of feature information. The large language model can be a pre-trained Transformer architecture model, with its feedforward network layer replaced by a hybrid expert module. The gating network can calculate the relevance score between the multimodal fusion sequence and each expert network, and obtain the activation weight of each expert network through a normalization function, activating one or more expert networks with the highest weights. The trigger decision result can be a binary classification label, and the explanatory text can be a structured text containing decision basis and clinical recommendations output based on the large language model's generation capabilities.

[0161] Optionally, the decision-making unit can employ different fusion strategies and expert network configurations based on the complexity of the clinical scenario. For example, the weighted processing subunit can use multi-dimensional gating parameters to weight the low-dimensional feature matrix. The weighted processing subunit can assign different gating parameters to different feature dimensions in the low-dimensional feature matrix, giving differentiated weights to numerical features of different dimensions during the fusion process. For texture feature dimensions that are highly correlated with image quality, gating parameters with higher correlation to image quality can be assigned; for clinical baseline feature dimensions that are less correlated with image quality, relatively constant gating parameters can be assigned. When splicing the weighted numerical embedding sequence and the text embedding sequence, the concatenation processing subunit can insert a special delimiter vector between the two types of sequences to assist the large language model in distinguishing the semantic origins of different parts of the sequence. The input processing subunit can input the multimodal fusion sequence into the large language model embedded with a hybrid expert structure. The hybrid expert structure can include a first expert network, a second expert network, a third expert network, and a fourth expert network. The first expert network can handle scenarios related to bladder changes, the second expert network can handle scenarios related to rectal changes, the third expert network can handle scenarios involving multi-organ coupling changes, and the fourth expert network can handle basic scenarios with no significant changes. The activation processing subunit can calculate the correlation between the multimodal fusion sequence and each expert network through a gating network, and employ sparse activation, activating only the two expert networks with the highest correlation, while excluding the remaining expert networks to reduce computational overhead during inference. The output processing subunit can include a label output module and a text output module. The label output module can process the feature representations output by at least one expert network through a classification layer, outputting binary labels indicating whether online adaptive radiotherapy is triggered or not. The text output module can generate explanatory text in an autoregressive manner based on the generative capabilities of a large language model. When generating explanatory text, the text output module can call the parameters of the expert networks activated by the activation processing subunit to match the style and content of the generated explanatory text with the characteristics of the current clinical scenario. For example, when the activated expert network is an expert network that handles scenarios related to rectal changes, the explanatory text generated by the text output module can focus on the analysis of the impact of rectal volume changes on target location and dose distribution.

[0162] like Figure 6 As shown, for example, the decision-making unit can fuse numerical features and textual semantic features, and perform reasoning through a hybrid expert model. The processing flow of the decision-making unit may include multimodal fusion, expert routing, and weighted output.

[0163] In the adaptive multimodal fusion process, the decision unit receives the numerical embeddings output by the feature processing unit and the text embeddings output by the text processing unit as input. The weighting processing subunit can weight the low-dimensional feature matrix using gating parameters to obtain a weighted numerical embedding sequence. The decision unit can introduce a learnable adaptive gating scalar. The low-dimensional feature matrix is ​​weighted. The concatenation subunit is used to concatenate the weighted numerical embedding sequence and the text embedding sequence along the sequence dimension to obtain the multimodal fusion sequence.

[0164] Formula (12);

[0165] in, This refers to the low-dimensional feature matrix (e.g., K=64) output by the feature processing unit. Used to refer to the text embedding sequence generated by the text processing unit. Used to refer to the number of lexical units in statistically enhanced prompt text. Used to refer to the embedding dimension of a large language model. It can represent operation symbols that concatenate along the length dimension of a sequence. This can refer to the gating parameter passed through the weighted processing subunit. It is a learnable scalar used to dynamically adjust the weights of the low-dimensional feature matrix in the adaptive multimodal fusion process based on the image data quality. For example, the gating parameter can be initialized to 0.5. This can represent the multimodal fusion sequence obtained after splicing, where the sequence length of the multimodal fusion sequence is equal to... and The sum, the vector dimension at each position is 1. .

[0166] When the quality of cone-beam computed tomography (CBCT) images is poor (e.g., the presence of metal artifacts, motion blur, etc.), the reliability of numerical features may decrease. In this case, the decision unit can automatically lower the gating parameters through training. The value of the gating parameter should reduce reliance on numerical features and rely more on textual features described based on historical trends. Conversely, when image quality is good, the gating parameter... The values ​​are relatively high, and numerical characteristics play a dominant role.

[0167] In the expert routing process, lightweight hybrid expert inference can be performed. The input processing subunit can input the multimodal fusion sequence into the large language model, which embeds a hybrid expert structure. The decision unit can embed the hybrid expert structure between layers of the large language model and fine-tune it using low-rank adaptation. In the feedforward network layer of the Transformer, a hybrid expert module can be used instead. The hybrid expert layer can include a gating network and multiple expert networks. The gating network, as a routing mechanism, dynamically calculates the activation weights of each expert network based on the features of the input multimodal fusion sequence. Figure 6 The example demonstrates three expert networks: Expert 1 specializes in handling clinical scenarios related to bladder distension changes, Expert 2 specializes in handling clinical scenarios related to rectal pneumoperitoneum changes, and Expert N is responsible for handling other types of anatomical variation scenarios. The gating network analyzes the input through a trainable gating weight matrix and outputs weight assignments for each expert network.

[0168] During the weighted output process, the activation processing subunit can activate at least one expert network based on the multimodal fusion sequence through a gated network of the hybrid expert structure. For the input hidden state h of the current layer, the output of the hybrid expert module can be expressed as:

[0169] Formula (13);

[0170] in, This represents the input hiding state of the current layer. It can represent the total number of expert networks contained in a hybrid expert structure. The forward propagation function of the k-th expert network can be represented by a two-layer, multilayer perceptron. This can represent the hidden state of the k-th expert network for the input. The processed output vector has the same dimensions as the input. For a gating network, the output is the weight assignment for each expert, which can refer to the gating network's weight assignment based on the input hidden state. The output expert weight assignment vector. It can refer to the output hiding state of the current layer.

[0171] Gated networks can be represented as:

[0172] Formula (14);

[0173] in, It can represent the trainable gating weight matrix of a gating network, that is, the trainable gating weights. It can refer to the original expert score vector obtained by multiplying the gating weight matrix by the input hidden state.

[0174] The activated expert networks specialize in processing the input multimodal fusion sequence. The activation processing subunit performs a weighted sum of the outputs of each expert network and their corresponding weights to obtain the output of the hybrid expert layer. This weighted output is then passed to the subsequent large language model layer for further inference processing. Finally, the output processing subunit generates the triggering decision result for online adaptive radiotherapy and the corresponding explanatory text.

[0175] In some alternative embodiments, the expert network includes a first expert network and a second expert network, wherein the first expert network is used to handle scenarios related to bladder changes and the second expert network is used to handle scenarios related to rectal changes.

[0176] Optionally, the hybrid expert architecture can be configured with multiple functionally specialized expert networks, each implemented using a multilayer perceptron structure. During pre-training or fine-tuning, the first expert network can be exposed to training samples where bladder volume changes are the dominant factor, such as target displacement due to insufficient bladder filling or target compression due to overfilling. Through this specialized training, the first expert network can learn the correlation pattern between bladder changes and dosimetric consequences. During training, the second expert network can be exposed to training samples where rectal contents changes are the dominant factor, such as target displacement due to rectal gas accumulation or target deformation due to rectal fecal accumulation, thereby learning the influence of rectal changes on target localization and dose distribution. During the inference phase, when the activation processing subunit determines through the gating network that the current input multimodal fusion sequence closely matches the features of a bladder change-related scenario, it can assign a higher activation weight to the first expert network; conversely, when it determines that the current input closely matches the features of a rectal change-related scenario, it can assign a higher activation weight to the second expert network. By designing a division of labor among experts, the hybrid expert structure can enable each expert network to focus on learning decision-making patterns for specific types of clinical scenarios under limited training sample conditions, which helps improve the inference accuracy of the model under different anatomical changes.

[0177] In application scenarios, different expert networks can specialize in handling different clinical situations. For example, the first expert network can specifically handle situations where bladder distension leads to under-dose of the target area, while the second expert network can specifically handle situations where rectal pneumothorax leads to over-dose of organs at risk. The anatomical patterns of prostate cancer patients are diverse. Some patients primarily present with posterior displacement of the target area due to insufficient bladder distension, while others have anterior displacement due to rectal pneumothorax. Using a single model to handle all situations can easily lead to overlooking some aspects. A hybrid expert structure is similar to a multidisciplinary consultation: the first expert network excels at handling bladder-related issues, while the second expert network excels at handling rectal-related issues. For new cases, the gating network can automatically determine which expert network should be consulted first. This helps, with limited data, allow each expert network to focus on learning decision-making patterns specific to the scenario, thereby improving the model's specialization and accuracy.

[0178] In some optional embodiments, the output processing subunit includes: a label output module for outputting binary labels, which are used to indicate whether online adaptive radiotherapy is triggered or not; and a text output module for outputting structured text including decision-making basis, target indicator analysis and clinical recommendations based on the generation capabilities of a large language model.

[0179] Optionally, the label output module can be used to output binary labels, which indicate whether online adaptive radiotherapy is triggered or not. The label output module can receive feature representations output by at least one expert network, input these feature representations into a classification layer for processing, and map the processing results to binary values ​​of 0 or 1 through an activation function in the classification layer. Label "1" can be used to indicate triggering online adaptive radiotherapy, and label "0" can be used to indicate not triggering online adaptive radiotherapy. The text output module can be used to generate structured text based on the large language model's generation capabilities, outputting decision-making basis, target indicator analysis, and clinical recommendations. The text output module can utilize the autoregressive generation capabilities of the large language model, using a multimodal fusion sequence as input context, to generate explanatory text word-by-word. During the generation process, the text output module can call the parameters of the expert network activated by the activation processing subunit to match the content of the generated explanatory text with the features of the current clinical scenario. The explanatory text generated by the text output module can include a decision-making basis section explaining the main reasons for the current triggering decision; a target indicator analysis section describing the changing trends and potential impacts of target indicators such as bladder volume, rectal volume, or target coverage; and a clinical recommendation section providing operational suggestions for subsequent treatment. For example, when the binary label output by the label output module is "1", the text output module can generate the conclusion "Re-planning is recommended," and further generate structured text content such as "A sharp upward trend in rectal volume was detected in the past three fractions, leading to prostate forward displacement; the current prediction plan shows that rectal V50 has exceeded the safety limit," providing clinicians with complete decision support information.

[0180] For example, the processing flow of the output processing subunit may include binary classification decision-making and interpretation generation. In the binary classification decision-making process, the label output module outputs binary classification labels indicating whether or not online adaptive radiotherapy (ART) is triggered. The output processing subunit can output the binary classification result indicating whether ART is triggered. In the interpretation generation process, the text output module, based on the generation capabilities of a large language model, outputs structured text including decision justification, target indicator analysis, and clinical recommendations. The output processing subunit can output an example interpretation text, such as: "Recommendation to trigger replanning. Reason: Monitoring showed a sharp increase in rectal volume due to gas accumulation in the past three fractions, causing the prostate to shift forward. The current predicted plan shows that rectal V50 has exceeded the safety limit by 5%." In this way, the decision-making unit can achieve a closed loop from data to decision to evidence. Clinicians not only receive a binary classification answer but also evidence supporting clinical logic, which helps enhance the credibility of AI-assisted decision-making.

[0181] The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. In the above embodiments of this application, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0183] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0184] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A smart triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM, characterized in that, include: The acquisition unit is used to acquire the multimodal temporal feature matrix of the target object within a preset time window, wherein the multimodal temporal feature matrix includes radiomics features, dosimetry features, geometric features and clinical features related to prostate cancer; The feature processing unit is used to perform channel-independent temporal feature extraction and sparse bottleneck aggregation on the multimodal temporal feature matrix to obtain a low-dimensional feature matrix. The number of rows in the low-dimensional feature matrix is ​​independent of the original feature dimension of the multimodal temporal feature matrix. The sparse bottleneck aggregation is used to selectively extract a preset number of target features from the multimodal temporal feature matrix. The text processing unit is used to generate statistically enhanced prompt text based on the changing trend of the target indicators in the multimodal temporal feature matrix, and convert the statistically enhanced prompt text into a text embedding sequence, wherein the statistically enhanced prompt text represents a semantically rich natural language description that transforms the statistical trend of numerical features into a preset template. The decision unit is used to adaptively fuse the low-dimensional feature matrix with the text embedding sequence, and to perform reasoning through a pre-trained large language model to output the trigger decision result of online adaptive radiotherapy and the explanatory text corresponding to the trigger decision result; The feature processing unit includes: The convolution processing subunit is used to perform causal convolution operation on each feature channel in the multimodal temporal feature matrix to obtain a convolutional feature matrix. In the multimodal temporal feature matrix, all feature channels share the same convolution kernel. In the causal convolution operation, the output of the current time step depends on the input of the current time step and the historical time steps. An aggregation processing subunit is used to perform temporal aggregation on the convolutional feature matrix along the time dimension to obtain a scalar feature vector. The mapping processing subunit is used to map the scalar feature vector into a key matrix and a value matrix through a shared projection vector, wherein each row of the key matrix and the value matrix corresponds to a mapping representation of a feature channel; The computational processing subunit is used to perform cross-attention calculation on the key matrix and the value matrix using a preset potential query vector to obtain the bottleneck feature matrix; The linear transformation processing subunit is used to perform a linear transformation on the bottleneck feature matrix through an alignment matrix, so that the vector dimension of the transformed features is consistent with the word vector dimension of the large language model, thereby obtaining the low-dimensional feature matrix. The text processing unit includes: An extraction processing subunit is used to extract the time series sequence of at least one target index from the multimodal time series feature matrix; A sequence processing subunit is used to calculate the trend slope and volatility variance of the time series sequence of each of the target indicators; A semantic processing subunit is used to map the trend slope of each target indicator to a trend semantic description and to map the volatility variance of each target indicator to a stability semantic description. The text processing subunit is used to fill the trend semantic description and the stability semantic description into a preset prompt word template to generate the statistically enhanced prompt text; An embedding processing subunit is used to input the statistically enhanced prompt text into the embedding layer of a large language model to obtain the text embedding sequence, wherein the text embedding sequence is a vector sequence.

2. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM as described in claim 1, characterized in that, The shared projection vector includes a key projection vector and a value projection vector, wherein the key projection vector and the value projection vector have the same dimension and are independent of the number of feature channels.

3. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM according to claim 1, characterized in that, The target indicators include at least one of bladder volume, rectal volume, and target coverage.

4. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM as described in claim 1, characterized in that, The semantic processing subunit includes: The first trend semantic processing module is used to map the trend semantic description into first trend semantic information when the trend slope is greater than a first preset threshold. The first trend semantic information is used to describe that the target indicator is rising and the rising rate is greater than a preset rate. The second trend semantic processing module is used to map the trend semantic description into second trend semantic information when the trend slope is greater than a second preset threshold and less than or equal to the first preset threshold. The second trend semantic information is used to describe that the target indicator is rising and the rising rate is less than or equal to the preset rate. The third trend semantic processing module is used to map the trend semantic description into third trend semantic information when the trend slope is greater than or equal to a third preset threshold and less than or equal to a second preset threshold. The third trend semantic information is used to describe that the target indicator is maintained within the target range. The fourth trend semantic processing module is used to map the trend semantic description into fourth trend semantic information when the trend slope is greater than or equal to a fourth preset threshold and less than the third preset threshold. The fourth trend semantic information is used to describe that the target indicator is declining and the decline rate is less than or equal to the preset rate. The fifth trend semantic processing module is used to map the trend semantic description into fifth trend semantic information when the trend slope is less than the fourth preset threshold. The fifth trend semantic information is used to describe that the target indicator is declining and the decline rate is greater than the preset rate.

5. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM according to claim 1, characterized in that, The semantic processing subunit further includes: The coefficient processing module is used to determine the coefficient of variation based on the fluctuation variance and the mean. The first stability semantic processing module is used to map the stability semantic description to first stability semantic information when the coefficient of variation is greater than the first variation threshold. The first stability semantic information is used to describe that the fluctuation amplitude of the target indicator is greater than the preset fluctuation threshold. The second stability semantic processing module is used to map the stability semantic description to second stability semantic information when the coefficient of variation is greater than the second variation threshold and less than or equal to the first variation threshold. The second stability semantic information is used to describe that the fluctuation range of the target indicator is less than or equal to the preset fluctuation threshold and the fluctuation range of the target indicator is not maintained within the preset range. The third stability semantic processing module is used to map the stability semantic description into third stability semantic information when the coefficient of variation is less than or equal to the second variation threshold. The third stability semantic information is used to describe the fluctuation range of the target indicator as being maintained within the preset range.

6. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM according to claim 1, characterized in that, The decision-making unit includes: The weighted processing subunit is used to weight the low-dimensional feature matrix through a gating parameter to obtain a weighted numerical embedding sequence. The gating parameter is used to dynamically adjust the weight of the low-dimensional feature matrix in the adaptive multimodal fusion process according to the image data quality. The splicing processing subunit is used to splice the weighted numerical embedding sequence and the text embedding sequence along the sequence dimension to obtain a multimodal fusion sequence; An input processing subunit is used to input the multimodal fusion sequence into the large language model, wherein a hybrid expert structure is embedded in the large language model; An activation processing subunit is configured to activate at least one expert network according to the multimodal fusion sequence through the gated network of the hybrid expert structure. The output processing subunit is used to process the multimodal fusion sequence through the at least one expert network and output the triggering decision result of online adaptive radiotherapy and the explanatory text corresponding to the triggering decision result.

7. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM according to claim 6, characterized in that, The expert network includes a first expert network and a second expert network. The first expert network is used to handle scenarios related to bladder changes, and the second expert network is used to handle scenarios related to rectal changes.

8. The intelligent triggering decision-making device for online adaptive radiotherapy of prostate cancer based on LLM according to claim 6, characterized in that, The output processing subunit includes: The label output module is used to output binary labels, which are used to indicate whether online adaptive radiotherapy is triggered or not. The text output module is used to generate structured text, including decision-making basis, target indicator analysis and clinical recommendations, based on the generation capabilities of the large language model.

Images