An eye fundus diagnosis and treatment suggestion generation system and method based on a large language model

Abstract

The application discloses an eye fundus diagnosis and treatment suggestion generation system and method based on a large language model, and belongs to the technical field of artificial intelligence medical auxiliary decision-making. The method comprises the following steps: receiving an eye fundus image set and a patient clinical text record, generating a lesion semantic feature map set carrying spatial position coding through lesion semantic anchoring processing, and performing time-sequenced medical history state reconstruction processing on the clinical text record to generate a medical history dynamic feature sequence with time continuity; inputting the lesion semantic feature map set and the medical history dynamic feature sequence into a cross-modal query routing layer, and generating a query routing weight matrix through bidirectional attention mapping calculation; performing dynamic retrieval and enhancement fusion on the weight matrix and a diagnosis and treatment rule vector in a pre-constructed diagnosis and treatment knowledge base to generate an enhanced reasoning context; inputting the enhanced reasoning context into a large language model decoder fine-tuned by instructions, and generating a diagnosis and treatment suggestion text containing diagnosis basis, risk stratification and follow-up plan through autoregressive generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-assisted medical decision-making technology, specifically to a system and method for generating fundus diagnosis and treatment suggestions based on a large language model. Background Technology

[0002] The diagnosis and treatment of fundus diseases relies on the comprehensive analysis of multimodal information. Existing generative auxiliary systems based on large language models attempt to combine fundus images with clinical text, but they generally remain at the level of shallow semantic encoding of single-frame images and static medical records, lacking the ability to model the deep correlation between the spatial distribution of lesions and the temporal evolution of medical history. The feature fusion method, which involves independently processing image and text branches and then simply splicing them together, fails to capture the temporal progression of lesion locations and the interaction between medical records at different time points. This results in a lack of explanation of the continuity of the disease in the generated suggestions, making it difficult to reflect the diagnostic and treatment logic of the dynamic evolution of the disease. When using large language models to generate diagnostic and treatment text, most existing solutions directly map visual and medical history features into a prompt context input decoder, lacking a structured retrieval and controllable fusion mechanism for specialized diagnostic and treatment knowledge. Although large language models possess rich parametric knowledge, in fine-grained specialized scenarios, they are prone to generating content inconsistent with standard diagnostic and treatment guidelines, leading to problems such as incomplete evidence chains or inaccurate risk assessments. Static knowledge injection methods cannot dynamically select applicable diagnostic and treatment rules based on the lesion-medical history correlation characteristics of each case, resulting in insufficient professional consistency of the generated results and limiting their practical application value in clinical auxiliary scenarios.

[0003] To address the aforementioned shortcomings, this method focuses on two core issues. First, it addresses how to construct a bidirectional cross-modal interaction mechanism between the spatial distribution of lesions and the temporal nodes of medical history, enabling automatic alignment and weight allocation of the locational lesion semantics on the image side and the temporal medical history status on the text side at a fine-grained level. Second, it utilizes this cross-modal alignment result to drive the dynamic retrieval and fusion of the diagnostic and treatment knowledge base, allowing the large language model to adaptively reference normative rules highly relevant to the current case characteristics when generating diagnostic and treatment suggestions, thereby improving the professionalism and credibility of the generated content. Summary of the Invention

[0004] This invention provides a system and method for generating fundus diagnosis and treatment suggestions based on a large language model. It aims to achieve a refined fusion of spatial features of fundus lesions and time series of clinical history by introducing cross-modal query routing and dynamic retrieval enhancement mechanisms from a knowledge base. Based on this, and combined with the dynamic injection of diagnosis and treatment rules, it generates diagnosis and treatment suggestion texts with sufficient diagnostic basis, reasonable risk stratification, and clear follow-up plans, thus meeting the requirements of spatiotemporal semantic consistency and professional standardization for clinical auxiliary decision-making.

[0005] The objective of this invention can be achieved through the following technical solutions: This invention provides a method and system for generating fundus diagnosis and treatment recommendations based on a large language model. By integrating multimodal fundus images with patient clinical text records, it achieves the automatic generation of diagnosis and treatment recommendation texts with diagnostic basis, risk stratification, and follow-up plans.

[0006] In one technical solution of the present invention, the method includes: responding to a received set of fundus images and a patient's clinical text record, calling a pre-trained fundus multimodal feature extraction network to perform lesion semantic anchoring processing on the fundus image set, generating a set of lesion semantic feature maps carrying spatial location codes; performing temporal history state reconstruction processing on the patient's clinical text record, generating a dynamic feature sequence of medical history with temporal continuity; inputting the set of lesion semantic feature maps and the dynamic feature sequence of medical history into a cross-modal query routing layer, wherein the cross-modal query routing layer generates a query routing weight matrix by calculating the bidirectional attention mapping between the spatial distribution of lesions and the time nodes of medical history; performing dynamic retrieval enhancement fusion of the query routing weight matrix and the diagnostic rule vectors in a pre-constructed diagnostic knowledge base to generate an enhanced reasoning context; inputting the enhanced reasoning context into a large language model decoder that has been fine-tuned by instructions, and autoregressively generating a diagnostic recommendation text containing diagnostic basis, risk stratification, and follow-up plan. Therefore, the correlation between lesion location information in images and the evolution of medical history is explicitly modeled, and the reasoning ability of the large language model is enhanced by external diagnosis and treatment knowledge retrieval, thereby improving the accuracy and clinical interpretability of the generated diagnosis and treatment suggestions.

[0007] As a preferred technical solution of the present invention, the process of calling a pre-trained fundus multimodal feature extraction network to perform lesion semantic anchoring processing on the fundus image set and generating a set of lesion semantic feature maps carrying spatial location codes includes: inputting each frame of fundus image in the fundus image set into the lesion candidate region generation subnetwork of the fundus multimodal feature extraction network, and extracting the lesion candidate region bounding box and the corresponding region confidence score through a sliding window anchoring mechanism; for each lesion candidate region bounding box, calling the semantic segmentation head of the fundus multimodal feature extraction network to perform pixel-by-pixel semantic classification on the pixels inside the lesion candidate region bounding box to generate a lesion type probability map; assigning a lesion semantic label to the lesion candidate region bounding box according to the lesion type corresponding to the maximum probability value in the lesion type probability map; jointly encoding the geometric center coordinates of the lesion candidate region bounding box and its corresponding lesion semantic label into the spatial location code, and superimposing the spatial location code with the lesion type probability map to generate the set of lesion semantic feature maps carrying spatial location codes. Preferably, the lesion candidate region generation subnetwork adopts a region proposal network based on anchor boxes, and uses a non-maximum suppression algorithm to deduplicate candidate region bounding boxes with overlap exceeding a preset threshold, so as to reduce redundant candidate regions while ensuring lesion recall rate and improve the efficiency of lesion semantic anchoring.

[0008] As another preferred technical solution of the present invention, the process of performing time-series medical history state reconstruction processing on the patient's clinical text record to generate a dynamic feature sequence of medical history with temporal continuity includes: extracting multiple time point visit record fields from the patient's clinical text record, each visit record field containing a visit timestamp, chief complaint text, past medical history text, and medication record text; arranging the visit record fields into an original medical history time sequence according to the chronological order of the visit timestamps; and calling a pre-trained medical history feature encoder to process each visit record field in the original medical history time sequence. The chief complaint text, past medical history text, and medication record text are semantically vectorized to generate corresponding chief complaint semantic vectors, past medical history semantic vectors, and medication semantic vectors. The chief complaint semantic vectors, past medical history semantic vectors, and medication semantic vectors corresponding to the same medical record field are horizontally concatenated, and the positional code of the medical record timestamp is appended to the concatenated vector to generate the medical history state vector for that medical record field. The medical history state vectors of all medical record fields are vertically stacked according to the order of the medical record timestamps to generate the time-continuous dynamic feature sequence of medical history. This process transforms discrete clinical text information into a structured feature representation on a continuous timeline, enabling the dynamic evolution of medical history states over time to be captured by subsequent models.

[0009] In a further preferred embodiment of the present invention, the step of inputting the lesion semantic feature map set and the medical history dynamic feature sequence into the cross-modal query routing layer to generate a query routing weight matrix includes: flattening each lesion semantic feature map in the lesion semantic feature map set into a one-dimensional lesion feature vector, and arranging all one-dimensional lesion feature vectors in spatial coordinate order to construct a lesion feature matrix; wherein, for the i-th lesion semantic feature map in the lesion semantic feature map set, its height is denoted as H, its width as W, and its number of channels as C, and it is converted into a one-dimensional lesion feature vector of length H×W×C through a spatial dimension flattening operation; stacking the one-dimensional lesion feature vectors corresponding to all N lesion semantic feature maps in spatial coordinate order to construct a lesion feature matrix F of dimension N×D, D=H×W×C, wherein the spatial coordinate order is defined as the pixel-by-pixel scanning order from left to right and from top to bottom, and then expanding it sequentially according to the channel dimension. Each medical history state vector in the dynamic medical history feature sequence is treated as a row to construct a medical history feature matrix. The matrix product between the lesion feature matrix and the transpose of the medical history feature matrix is ​​calculated to obtain the original cross-modal association matrix. Row dimension normalization is performed on each row of the original cross-modal association matrix to generate a first attention weight matrix representing the association strength between each lesion feature and all medical history states. Column dimension normalization is performed on each column of the original cross-modal association matrix to generate a second attention weight matrix representing the association strength between each medical history state and all lesion features. The first attention weight matrix and the second attention weight matrix are multiplied element-wise to generate the query routing weight matrix. The query routing weight matrix obtained through bidirectional attention mapping can simultaneously reflect the contribution of the spatial distribution of lesions to the medical history time nodes and the degree of attention paid by the medical history time nodes to the lesion location, thereby providing a refined cross-modal routing basis for subsequent knowledge retrieval.

[0010] As a preferred implementation, the process of dynamically retrieving and enhancing the fusion of the query routing weight matrix with the diagnostic rule vectors in the pre-built diagnostic knowledge base to generate an enhanced inference context includes: extracting all diagnostic rule vectors from the pre-built diagnostic knowledge base, where each diagnostic rule vector corresponds to a rule text containing a mapping relationship between disease diagnostic criteria and treatment recommendations; calculating the weighted sum of the lesion feature matrix and the medical history feature matrix based on the weight distribution in the query routing weight matrix to generate a comprehensive query vector. Specifically, let the lesion feature matrix be F with dimension N×D, the medical history feature matrix be H with dimension M×D, and the query routing weight matrix be W with dimension N×M; first, calculate the lesion weighted query vector. as follows: in: This represents the value of the element in the i-th row and j-th column of the query routing weight matrix W. This represents the one-dimensional lesion feature vector corresponding to the semantic feature map of the i-th lesion in the lesion feature matrix F. Simultaneously, the medical history weighted query vector is calculated. as follows: in This represents the j-th medical history state vector in the medical history feature matrix H. The lesion-weighted query vector and the medical history-weighted query vector are concatenated to generate the comprehensive query vector. : Wherein, Concat(·,·) represents the concatenation operation of vectors along the feature dimension. The cosine similarity between the comprehensive query vector and each of the treatment rule vectors is calculated, and the top few treatment rule vectors are selected as candidate retrieval rule vectors based on their cosine similarity from high to low. The candidate retrieval rule vectors are then fused with the comprehensive query vector through cross-attention fusion. Specifically, the comprehensive query vector is used as the query term, and the candidate retrieval rule vectors are used as the key term to calculate the attention-weighted rule enhancement vector. The rule enhancement vector is then residually concatenated with the comprehensive query vector to generate the enhanced inference context. Preferably, each rule text in the pre-constructed treatment knowledge base is pre-mapped to the treatment rule vector through a rule encoder trained by a contrastive learning objective, ensuring that rule vectors of similar treatment scenarios are close to each other in the embedding space, thus improving retrieval accuracy. This approach uses the combined representation of images and medical history as a query, adaptively retrieves the most relevant diagnostic and treatment rules from an external knowledge base, and deeply integrates the query and rules through a cross-attention mechanism to generate a more informative and logically complete reasoning context, thereby enhancing the factual consistency and professionalism of the diagnostic and treatment recommendations generated by the large language model.

[0011] In another preferred embodiment of the present invention, the process of inputting the enhanced inference context into a large language model decoder finely tuned by instructions to autoregressively generate a treatment recommendation text containing diagnostic criteria, risk stratification, and follow-up plan includes: concatenating a preset system role cue vector at the front end of the enhanced inference context, the system role cue vector constraining the large language model decoder to generate output according to the tone of an ophthalmologist; concatenating a preset output format constraint vector at the back end of the enhanced inference context, the output format constraint vector instructing the large language model decoder to sequentially generate a diagnostic criteria paragraph, a risk stratification paragraph, and a follow-up plan paragraph; inputting the concatenated complete input sequence into the self-attention calculation layer of the large language model decoder, predicting the probability distribution of the next token position by position through a causal masking mechanism; in each generation step, calculating the generation probability of the next token based on the currently generated token sequence and the complete input sequence, and selecting the token with the highest probability as the output token of the current step; repeating the position-by-position prediction step until a preset end token is generated or the maximum generation length is reached, and combining all generated output tokens in the output order to form the treatment recommendation text. Preferably, the autoregressive generation process employs a cluster search strategy, retaining the K candidate sequences with the highest probability in each generation step, and selecting the sequence with the highest overall log probability as the final output treatment suggestion text at the end of generation. This achieves a balance between generation quality and diversity, ensuring that the output treatment suggestions are both consistent with medical logic and fully cover the required content.

[0012] This invention also includes a fundus diagnosis and treatment suggestion generation system based on a large language model. The system includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the aforementioned fundus diagnosis and treatment suggestion generation method based on a large language model. This system can automatically and comprehensively utilize multimodal patient data in ophthalmological clinical settings to efficiently generate structured and accurate diagnosis and treatment suggestions, assisting doctors in decision-making.

[0013] The beneficial effects of this invention are: The cross-modal query routing layer generates a query routing weight matrix by establishing a bidirectional attention mapping between the spatial distribution of lesions and the time nodes of medical history. Specifically, the semantic feature map of lesions carrying spatial location encoding is flattened into a lesion feature matrix, and the dynamic feature sequence of medical history with temporal continuity is constructed into a medical history feature matrix. By calculating the original cross-correlation matrix between the two modalities and performing normalization processing in the row and column dimensions respectively, two sets of attention weights are obtained, representing the correlation strength between lesion features and all medical history states, and the correlation strength between medical history states and all lesion features, respectively. The final query routing weight matrix is ​​obtained by multiplying them element-wise. This structure enables each lesion region to adaptively focus on the medical history state at the time node that is semantically related to it, while the medical history state at each time node can also accurately focus on the lesion spatial location with causal relationship, thus forming a bidirectional aligned representation of the temporal dynamics of lesion evolution and spatial morphological changes. This provides a spatiotemporal coupling fusion basis for subsequent reasoning, improving the shortcomings of traditional methods where cross-modal interaction only stays at the global semantic level, and enabling the generation of diagnosis and treatment suggestions to more accurately capture the key nodes of disease progression and the corresponding lesion features. The dynamic retrieval enhancement fusion process utilizes a query routing weight matrix to weightedly fuse the lesion feature matrix and the medical history feature matrix into a comprehensive query vector, achieving compression from the interaction matrix to a unified query representation. This comprehensive query vector simultaneously carries weighted details of lesion spatial information and medical history temporal information. It is then used as a query item in a pre-built diagnostic knowledge base to calculate the cosine similarity with each diagnostic rule vector, filtering out candidate rule vectors that highly match the cross-modal features of the current case. Further, using the comprehensive query vector as the query and the candidate rule vectors as the keys, cross-attention fusion is performed, supplemented by residual connections to generate rule enhancement vectors, forming an enhanced inference context. This mechanism deeply couples the triggering conditions of knowledge base retrieval with the cross-modal alignment results, ensuring that each retrieved applicable rule inherently corresponds to the joint features of lesions and medical history, avoiding noise introduced by irrelevant rules. When the large language model decoder generates text based on the enhanced inference context through autoregression, it can dynamically rely on professional rules to constrain the generation process, effectively reducing content output that does not conform to the clinical pathway of fundus specialists, improving the coherence of diagnostic criteria, the gradient of risk stratification, and the executability of follow-up plans. Attached Figure Description

[0014] The invention will now be further described with reference to the accompanying drawings.

[0015] Figure 1 This is a flowchart of a method for generating fundus diagnosis and treatment suggestions based on a large language model; Figure 2 This is a flowchart of the diagnostic suggestion text generation process based on enhanced reasoning context and cue constraints; Figure 3 This is a schematic diagram of the structure of a fundus diagnosis and treatment suggestion generation system based on a large language model. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] See Figure 1 This invention provides a method for generating fundus diagnosis and treatment suggestions based on a large language model. This method jointly analyzes multimodal fundus images and patient clinical text records. First, it extracts a set of semantic feature maps carrying spatial location codes from the fundus image set through lesion semantic anchoring processing. Simultaneously, it performs temporal history state reconstruction processing on the patient's clinical text records to generate a dynamic feature sequence of medical history with temporal continuity. Then, the lesion semantic feature map set and the dynamic feature sequence of medical history are input into a cross-modal query routing layer. By calculating the bidirectional attention mapping between the spatial distribution of lesions and the time nodes of medical history, a query routing weight matrix is ​​generated. This query routing weight matrix is ​​then dynamically retrieved and enhanced with treatment rule vectors from a pre-constructed diagnosis and treatment knowledge base to generate an enhanced inference context. Finally, the enhanced inference context is input into a large language model decoder that has been fine-tuned by instructions, and autoregressively generates a diagnosis and treatment suggestion text containing diagnostic criteria, risk stratification, and follow-up plans.

[0018] In practice, a pre-trained fundus multimodal feature extraction network is invoked to perform lesion semantic anchoring processing on the fundus image set. The fundus multimodal feature extraction network consists of two parts: a lesion candidate region generation subnetwork and a semantic segmentation head. The lesion candidate region generation subnetwork adopts a region proposal network based on anchor boxes, and the semantic segmentation head is a fully convolutional subnetwork for pixel-by-pixel classification.

[0019] The lesion candidate region generation subnetwork takes the original fundus image as input. First, it extracts a shared feature map from the fundus image through a feature extraction backbone network composed of multiple stacked convolutional and pooling layers. At each spatial location of the shared feature map, the lesion candidate region generation subnetwork generates a set of anchor boxes with preset scales and aspect ratios using a sliding window anchoring mechanism. The anchor box scales are set to three levels: 32×32, 64×64, and 128×128, and the aspect ratios are set to three ratios: 0.5, 1.0, and 2.0. A total of nine anchor boxes are generated for each spatial location. Above the shared feature map, the lesion candidate region generation subnetwork includes an intermediate convolutional layer with a 3×3 kernel and 256 output channels for further feature transformation. The output of the intermediate convolutional layer is fed into a classification branch and a regression branch. The classification branch is implemented using a 1×1 convolutional layer with 18 output channels, corresponding to the confidence scores for both foreground and background categories for each anchor box. The regression branch is also implemented using a 1×1 convolutional layer with 36 output channels, corresponding to the bounding box coordinate offset for each anchor box. By using the outputs of the classification and regression branches, the bounding boxes of all anchor boxes and their corresponding lesion candidate regions, along with their region confidence scores, can be obtained.

[0020] After obtaining the bounding boxes and region confidence scores of candidate lesion regions, the lesion candidate region generation subnetwork uses a non-maximum suppression algorithm (NMS) to deduplicate redundant candidate lesion region bounding boxes. The NMS algorithm sorts the candidate lesion region bounding boxes from highest to lowest region confidence score, selects the bounding box with the highest score, and calculates the intersection-union ratio (IU) of this bounding box with the remaining bounding boxes. When the IU exceeds a preset threshold, the bounding box with excessive overlap is removed. The preset threshold is set to 0.5. This setting effectively suppresses highly overlapping duplicate bounding boxes while retaining sufficient lesion region proposals, reducing the computational load of subsequent semantic segmentation and avoiding repeated processing of the same lesion. After processing by the NMS algorithm, a simplified set of candidate lesion region bounding boxes is obtained.

[0021] For each candidate lesion region bounding box retained after non-maximum suppression screening, the corresponding image region on the original fundus image is obtained and resized to a uniform size of 128×128 pixels. The uniformly sized image region is then fed into the semantic segmentation head. The semantic segmentation head contains an encoder-decoder structure consisting of multiple convolutional layers and upsampling layers. The encoder progressively reduces the spatial resolution of the feature map and increases the number of channels, while the decoder restores the spatial resolution through upsampling operations. Finally, at the output layer, a convolutional layer with a 1×1 kernel and a number of output channels, along with a Softmax activation function, generates a length [value missing] for each pixel within the bounding box. for The probability vector, This represents the number of predefined lesion categories, including fundus lesion types such as microaneurysms, hemorrhages, hard exudates, soft exudates, and neovascularization. The probability vectors of all pixels within the bounding box are aggregated to form a semantic category probability map. The spatial dimension of the semantic category probability map is 128×128, and the channel dimension is... The sum of the probability values ​​of all channels at each spatial location is 1.

[0022] Based on the lesion type probability map output by the semantic segmentation head, lesion semantic labels are assigned to the bounding boxes of lesion candidate regions. Specifically, among all pixels and all categories in the lesion type probability map, a globally maximum probability value is found, and the lesion category corresponding to this globally maximum probability value is determined as the overall lesion semantic label for the bounding box of the lesion candidate region. Simultaneously, the geometric center coordinates of the lesion candidate region bounding boxes are calculated. These coordinates are obtained by the arithmetic mean of the coordinates of the top-left and bottom-right vertices of the bounding box; that is, the horizontal center value is twice the average of one-quarter of the bounding box width, and the vertical center value is the average of the bounding box height. The canonical representation is as follows: , The values ​​of the geometric center coordinates are normalized. The normalization method is to normalize the values ​​of the geometric center coordinates by... Divide by the width of the fundus image, Divide by the height of the fundus image to obtain the normalized geometric center coordinates.

[0023] The normalized geometric center coordinates and lesion semantic labels are jointly encoded into a spatial location code. The joint encoding process is as follows: First, the lesion semantic label is mapped into a high-dimensional embedding vector with a dimension of 64. The mapping matrix consists of learnable embedding layer parameters. Then, the normalized x-coordinate and y-coordinate are converted into two 64-dimensional vectors respectively using a sinusoidal location encoding function. The sinusoidal location encoding function is expressed as follows: in: For scalar position values, the x-coordinate of the spatial position encoding, The value is the normalized x-coordinate value. For the ordinate, The value is the normalized ordinate value. ; This represents the dimension index, ranging from 0 to 31. The total dimension for location encoding is set to 64. The position encoding vector is represented by the first... sinusoidal components in each dimension Indicates the first The cosine components in each dimension are used to add the embedding vector of the lesion semantic label to the horizontal coordinate position encoding vector and the vertical coordinate position encoding vector element by element to obtain a 64-dimensional fusion encoding vector, which is the spatial position encoding.

[0024] Next, the spatial location code and the lesion type probability map are superimposed to generate a semantic feature map of the lesion carrying the spatial location code. The feature map superposition operation uses a channel-dimensional concatenation method: first, the spatial location code is expanded to the same spatial resolution as the lesion type probability map, i.e., the 64-dimensional spatial location code vector is copied 128×128 times to form a spatial location code feature map with dimensions of 128×128×64; then, the lesion type probability map and the spatial location code feature map are concatenated along the channel dimension, resulting in a feature map with dimensions of 128×128×64. +64). The stitched feature map is the lesion semantic feature map carrying spatial location encoding. The above operation is performed on all lesion candidate region bounding boxes extracted from each frame of fundus image in the fundus image set. All the obtained lesion semantic feature maps carrying spatial location encoding are summarized to form the lesion semantic feature map set.

[0025] The pre-training process of the fundus multimodal feature extraction network was completed on a dataset containing multi-sample fundus image annotations, including lesion bounding box coordinates and pixel-wise lesion categories. During training, the lesion candidate region generation sub-network adopted a multi-task loss function, which consisted of a weighted sum of classification and regression losses. The classification loss used binary cross-entropy loss, and the regression loss used smoothed L1 loss, with a weight ratio of 1:1. The semantic segmentation head was optimized using pixel-wise cross-entropy loss. The entire network was trained end-to-end using a stochastic gradient descent optimizer, with an initial learning rate of 0.001, momentum parameter of 0.9, and weight decay coefficient of 0.0005. After training, the network parameters were fixed for lesion semantic anchoring processing during the inference phase.

[0026] In practice, multiple time-point medical record fields are extracted from the patient's clinical text records. These records are stored in a structured electronic medical record format, containing complete text information from each of the patient's visits. The electronic medical record database is searched using the visit date or admission date as the key, and each visit record is read sequentially, including the visit timestamp, chief complaint text, past medical history text, and medication record text. The retrieved content is then used as components of the corresponding medical record fields. The visit timestamp is accurate to the date and represented as a string in "YYYY-MM-DD" format. The chief complaint text is a free text paragraph describing the patient's main symptoms and discomfort during the current visit. The past medical history text is a text paragraph recording the patient's confirmed historical disease diagnoses and surgical history. The medication record text is a text paragraph recording the names, dosages, and frequencies of the patient's current and previously used medications. All the fields of the patient's clinical text records extracted form a list arranged in chronological order of the visit timestamps. Each element in the list is a quadruple, which contains the visit timestamp, chief complaint text, past medical history text, and medication record text in sequence.

[0027] After extracting the list of medical record fields, the fields are arranged into a raw medical history time series according to the chronological order of the visit timestamps. Specifically, the list of medical record fields is sorted in ascending order of visit timestamps, with earlier dates placed at the beginning of the sequence and later dates at the end, forming an ordered linear sequence structure. The t-th position in the raw medical history time series corresponds to the medical record field of the t-th visit, where t is a positive integer counting from 1.

[0028] The pre-trained medical history feature encoder performs semantic vectorization on the chief complaint text, past medical history text, and medication record text in each field of the original medical history time series, generating corresponding semantic vectors for the chief complaint, past medical history, and medication. The medical history feature encoder employs a bidirectional encoder based on the Transformer architecture, containing six Transformer encoding layers. Each encoding layer consists of a multi-head self-attention sublayer and a feedforward fully connected sublayer, with residual connections and layer normalization operations following each sublayer. The number of attention heads in the multi-head self-attention sublayer is set to 8, the hidden layer dimension of the feedforward fully connected sublayer is set to 2048, and the word embedding dimension of the medical history feature encoder is set to 512.

[0029] In the semantic vectorization of the chief complaint text, the chief complaint text is fed into the medical history feature encoder. First, each word in the chief complaint text is mapped to a 512-dimensional word embedding vector through a word embedding layer. A corresponding positional encoding vector is then appended to the word embedding vector, which is calculated using a sinusoidal positional encoding function. The word embedding sequence with the added positional encoding is then input into the six encoding layers of the medical history feature encoder for layer-by-layer encoding. At the output of the last encoding layer, the vector corresponding to the position of the [CLS] special classification marker inserted at the beginning of the chief complaint text is taken as the chief complaint semantic vector, which has a dimension of 512.

[0030] In the semantic vectorization of past medical history texts, the past medical history texts are fed into the same medical history feature encoder as the chief complaint text vectorization process. The [CLS] marker is inserted at the beginning of the past medical history text. After word embedding and position encoding, the texts are encoded by the 6-layer encoding layer of the medical history feature encoder. The vector corresponding to the position of the [CLS] marker in the output sequence is taken as the past medical history semantic vector. The dimension of the past medical history semantic vector is 512.

[0031] In the semantic vectorization processing of medication record text, the medication record text is fed into the same medical history feature encoder as input. Similarly, a [CLS] marker is inserted at the beginning of the text. After word embedding and position encoding, it is encoded by the medical history feature encoder. The vector corresponding to the position of the [CLS] marker is taken as the medication semantic vector. The dimension of the medication semantic vector is 512.

[0032] The pre-training process of the medical history feature encoder was performed on a large medical text corpus, using a masked language model task. Specific training settings included: randomly masking 15% of the word tokens in the input text, with 80% of the masked words replaced by a special mask [MASK], 10% replaced by random words, and 10% remaining unchanged. The medical history feature encoder outputs all position vectors except for the [CLS] marker locations. These vectors are projected onto the vocabulary dimension through a linear layer, and the original words at the masked locations are predicted using a Softmax function. The loss function used was cross-entropy loss, the optimizer was the Adam optimizer, the learning rate was set to 0.0001, the batch size was set to 32, and the training epochs were set to 10. After pre-training, the parameters of the medical history feature encoder were fixed and saved for extracting semantic vectors from the text.

[0033] After obtaining the chief complaint semantic vector, past medical history semantic vector, and medication semantic vector corresponding to the same medical record field, the three semantic vectors are horizontally concatenated. The horizontal concatenation operation is performed in the order of chief complaint semantic vector, past medical history semantic vector, and medication semantic vector, resulting in a 1536-dimensional intermediate vector. A positional code for the medical visit timestamp is appended to the concatenated vector. The implementation method for appending the positional code is as follows: the medical visit timestamp is converted into a cumulative number of days calculated from a base date, which is set to the earliest medical visit date among all the patient's medical records. Let the cumulative number of days be... ,Will As the position value, a 512-dimensional time-position coding vector is calculated using a sinusoidal position coding function, which is expressed as follows: in: This represents the cumulative number of days corresponding to the timestamp of the medical visit, and the value range is a non-negative integer starting from 0; This is the dimension index, and its value ranges from 0 to 255 (integers). The time position encoding vector is represented by the first... The values ​​of the sinusoidal components in each dimension The time position encoding vector is represented by the first... The cosine component values ​​in each dimension are calculated. The calculated 512-dimensional time location encoding vector is concatenated to the end of the 1536-dimensional intermediate vector to generate a 2048-dimensional vector, which is the medical history status vector of the corresponding medical record field.

[0034] After generating the medical history status vectors corresponding to each field of all medical records, the medical history status vectors of all medical record fields are stacked vertically according to the order of the medical visit timestamps. The vertical stacking operation places the medical history status vector corresponding to the first medical visit in the original medical history time series in the first row, the second medical visit in the second row, and so on until the last medical visit in the last row, thus forming a two-dimensional matrix. The number of rows in this matrix equals the total number of medical visit fields in the original medical history time series, and the number of columns equals 2048. This two-dimensional matrix is ​​the dynamic feature sequence of medical history with temporal continuity. Each row of the dynamic feature sequence corresponds to a medical history status at a continuous point in time, and the time interval between rows is determined by the difference in medical visit timestamps between adjacent medical visit records, thus completely preserving the temporal evolution information during the development of the medical history.

[0035] In practice, the semantic feature map set of lesions and the dynamic feature sequence of medical history are input into the cross-modal query routing layer. The role of the cross-modal query routing layer is to perform bidirectional attention mapping between the spatial distribution of lesions and the time nodes of medical history, generate a query routing weight matrix, and then dynamically retrieve and enhance the fusion from the pre-built diagnosis and treatment knowledge base based on the query routing weight matrix to generate an enhanced reasoning context.

[0036] The cross-modal query routing layer first flattens and constructs a matrix from the lesion semantic feature map set. For the i-th lesion semantic feature map in the lesion semantic feature map set, the height of the lesion semantic feature map is defined as H, the width as W, and the number of channels as C. In the flattening operation, according to the definition of spatial coordinate order, that is, the pixel-by-pixel scanning order from left to right and from top to bottom, and on this basis, it is expanded sequentially according to the channel dimension, transforming the three-dimensional feature map of dimension H×W×C into a one-dimensional lesion feature vector. The length of the one-dimensional lesion feature vector is D=H×W×C. After performing the flattening operation, all N lesion semantic feature maps in the lesion semantic feature map set correspond to N one-dimensional lesion feature vectors. These N one-dimensional lesion feature vectors are stacked row by row in spatial coordinate order to construct a two-dimensional matrix of dimension N×D, denoted as the lesion feature matrix F. Each row of the lesion feature matrix F corresponds to a one-dimensional lesion feature vector of a lesion semantic feature map, and the row index i ranges from 1 to N.

[0037] Simultaneously, the cross-modal query routing layer constructs a matrix from the dynamic medical history feature sequence. The dynamic medical history feature sequence is a two-dimensional matrix, where each row represents a medical history state vector, and the dimension of the medical history state vector is 2048. The dynamic medical history feature sequence is directly used as the medical history feature matrix, denoted as the medical history feature matrix H. The number of rows in the medical history feature matrix H equals the total number of fields in the medical records M, and the number of columns equals the dimension of the medical history state vector. =2048. The dimension of the medical history feature matrix H is M× Since it will interact with the lesion feature matrix later, before calculation, the vector dimension of the medical history feature matrix H is mapped to the same dimension D as the lesion feature vector through a linear projection layer. The mapped medical history feature matrix is ​​still denoted as H, with a dimension of M×D. The weight matrix of the linear projection layer is a... A ×D matrix, biased into a D-dimensional vector.

[0038] After obtaining the lesion feature matrix F and the medical history feature matrix H, the cross-modal query routing layer calculates the matrix product between the transposes of the lesion feature matrix F and the medical history feature matrix H. The transpose of the medical history feature matrix is ​​denoted as H^T, with dimensions D×M. Multiplying F and H^T, where F has dimensions N×D and H^T has dimensions D×M, results in an N×M matrix, called the original cross-modal correlation matrix, denoted as... Original cross-modal correlation matrix The element value in the i-th row and j-th column represents the initial association strength between the i-th lesion feature and the j-th medical history state.

[0039] For the original cross-modal correlation matrix Each row undergoes row-dimensional normalization. The normalization method involves calculating the Softmax function along the row direction of the matrix, ensuring that the sum of all column elements in each row is 1. The matrix after row-dimensional normalization is called the first attention weight matrix, denoted as . First attention weight matrix The elements in the expression represent the association weight of each lesion feature with all medical history states, that is, the distribution of attention given to lesion features by medical history states from the perspective of lesions.

[0040] For the original cross-modal correlation matrix Each column undergoes column-dimensional normalization. The normalization method involves calculating the Softmax function along the column direction of the matrix, ensuring that the sum of all row elements in each column is 1. The matrix after column-dimensional normalization is called the second attention weight matrix, denoted as . Second attention weight matrix The elements in the table represent the association weight of each medical history state with all lesion features, that is, the distribution of attention of lesion features to medical history state from the perspective of medical history state.

[0041] The first attention weight matrix With the second attention weight matrix Element-wise multiplication is performed to generate the query routing weight matrix W. Element-wise multiplication is defined as multiplying the elements at the same positions in the two matrices, resulting in a new matrix with dimensions N×M. Elements in the query routing weight matrix W... It integrates attention information from both the lesion-to-medical-history and medical-history-to-lesion directions to comprehensively characterize the cross-modal association strength between the i-th lesion feature and the j-th medical history state.

[0042] After generating the query routing weight matrix W, all diagnostic rule vectors are extracted from a pre-built diagnostic knowledge base. The pre-built knowledge base contains several rule texts, each a natural language description of the mapping relationship between disease diagnostic criteria and treatment recommendations. The acquisition of diagnostic rule vectors relies on a rule encoder, which employs a Transformer-based encoder structure with four Transformer encoding layers, each with four multi-head self-attention heads, a hidden layer dimension of 256, and an output vector dimension of 256. The rule encoder is trained by contrastive learning objectives, with training data consisting of pairs of positive and negative rule texts. Positive rule texts are semantically similar diagnostic rule pairs, while negative rule texts are randomly sampled or semantically unrelated diagnostic rule pairs. During training, a pair of rule texts are fed into the rule encoder, each yielding a 256-dimensional rule vector. The distance between the vectors is calculated using cosine similarity. The contrastive learning loss function is InfoNCE loss, with a temperature coefficient set to 0.07, a batch size of 64, an Adam optimizer, and a learning rate of 0.00005. After training, the rule encoder parameters are fixed, and each rule text in the diagnostic knowledge base is encoded to obtain the corresponding diagnostic rule vector. All diagnostic rule vectors are stored to form a rule vector library.

[0043] In the dynamic retrieval enhancement and fusion stage, the comprehensive query vector is first calculated based on the query routing weight matrix W. The comprehensive query vector is obtained by concatenating the lesion-weighted query vector and the medical history-weighted query vector. Let the lesion feature matrix F have dimensions N×D, the medical history feature matrix H have dimensions M×D, and the query routing weight matrix W have dimensions N×M. (Lesion-weighted query vector) The calculation formula is: in: Represents the query route weight matrix The Middle Line 1 The element values ​​of the column, The index is an integer from 1 to N, corresponding to the lesion feature matrix. A one-dimensional lesion feature vector from the semantic feature map of each lesion. The integer index is from 1 to M, corresponding to the first element of the medical history feature matrix. A vector of medical history status; Representation of lesion feature matrix The Middle The length of the corresponding row is A one-dimensional lesion feature vector. Lesion weighted query vector. This is the sum of all lesion feature vectors multiplied by their corresponding query route weights, with dimensions of [missing information]. .

[0044] The calculation method for the medical history weighted query vector q_h is as follows: in: Representing the medical history feature matrix The Middle The length of the corresponding row is The medical history status vector. The medical history weighted query vector. This is the sum of all medical history state vectors multiplied by their corresponding query route weights, with dimensions [missing information]. .

[0045] Weighted query vector of lesions Weighted query vector with medical history Vectors are concatenated along the feature dimensions to generate a comprehensive query vector. The splicing operation is expressed as: in: This means concatenating two vectors end-to-end along the feature dimension. The dimension is .

[0046] Calculate the comprehensive query vector The cosine similarity is calculated between the query vector and each treatment rule vector in the rule vector library. The cosine similarity is calculated by dividing the inner product of the query vector and the treatment rule vector by the product of the magnitudes of the two vectors. Based on the cosine similarity, the top K treatment rule vectors are selected as candidate retrieval rule vectors, sorted from highest to lowest. The value of K is set to 5, chosen to maintain computational efficiency while covering multiple potentially related treatment rules.

[0047] Candidate retrieval rule vectors and comprehensive query vectors are fused across attention levels. The cross-attention fusion process uses the comprehensive query vector... The query term (Query) uses a matrix of K candidate retrieval rule vectors as the key and value terms. A scaling dot product attention mechanism is employed: the query and key are multiplied by a matrix, and then divided by a scaling factor, which is set to a specific scaling factor. ,in The dimension of the Key vector is 256. The Softmax function is applied to the result to obtain the attention weights, which are then multiplied by the Value to obtain the rule enhancement vector. The rule enhancement vector has the same dimension as the query vector, which is 2D.

[0048] Integrating rule-enhanced vectors with comprehensive query vectors Residual joins are performed, which involves adding vectors element by element to generate an enhanced inference context. This enhanced inference context is 2D and integrates lesion features, medical history features, and rule information retrieved from the diagnostic knowledge base.

[0049] In some implementations, the parameters of the linear projection layer and the rule encoder in the cross-modal query routing layer are jointly trained. The training objective is the negative log-likelihood loss of the downstream diagnosis and treatment suggestion generation task. The entire network is fine-tuned in an end-to-end manner. During fine-tuning, fundus images and medical record data with labeled diagnosis and treatment suggestion text are used. The optimizer is AdamW, the learning rate is set to 0.00002, and the batch size is set to 8.

[0050] See Figure 2 In the specific implementation, the enhanced inference context is input to the large language model decoder, which has been fine-tuned by instructions. The large language model decoder adopts a Transformer-based autoregressive decoder architecture, containing 24 Transformer decoding layers. Each decoding layer consists of a masked multi-head self-attention sub-layer, a cross-attention sub-layer, and a feedforward fully connected sub-layer. Each sub-layer is followed by residual connections and layer normalization operations. The number of attention heads in the masked multi-head self-attention sub-layer is set to 16, the hidden layer dimension of the feedforward fully connected sub-layer is set to 4096, the model word embedding dimension is set to 1024, and the vocabulary size is set to 50,000.

[0051] Before the augmented inference context is input to the large language model decoder, a pre-defined system role cue vector is concatenated at the front end of the augmented inference context. The system role cue vector is obtained through a learnable cue embedding matrix, which is a parameter matrix of dimension L×1024, where L is the fixed length of the system role cue, set to 16 vector positions. The cue embedding matrix is ​​updated through backpropagation during the training phase, ensuring that each 1024-dimensional vector in the matrix represents a specific role-constrained semantic unit. The role of the system role cue vector is defined as constraining the large language model decoder to generate output according to the tone of an ophthalmologist. The parameters of the system role cue vector are obtained through supervised fine-tuning on ophthalmology diagnosis dialogues and medical record generation corpora.

[0052] A pre-defined output format constraint vector is concatenated at the back end of the enhanced inference context. This output format constraint vector is also obtained through a learnable format embedding matrix with dimensions R×1024, where R is the fixed length of the output format constraint, set to 32 vector positions. The output format constraint vector instructs the large language model decoder to sequentially generate the diagnostic basis paragraph, risk stratification paragraph, and follow-up plan paragraph. The output format constraint vector is learned through instruction fine-tuning during the training phase. Each target output text in the training data is organized in the order of diagnostic basis, risk stratification, and follow-up plan, and paragraph boundaries are marked in the text using special delimiters. During training, the corresponding delimiter positions are aligned and matched with the output format constraint vector.

[0053] The enhanced inference context, system role cue vector, and output format constraint vector are concatenated into a complete input sequence in sequence. The concatenation order is as follows: the system role cue vector is placed at the beginning of the sequence, followed by the enhanced inference context vector, and finally the output format constraint vector. The total length of the complete input sequence is L+1+R. Since the enhanced inference context is a vector with a dimension of 2D, this vector is treated as a single position in the sequence length dimension. However, before concatenation, a linear projection layer is used to map the dimension of the enhanced inference context vector from 2D to 1024 dimensions, aligning it with the model's word embedding dimension. The weight matrix of the linear projection layer has a dimension of (2D)×1024, and the bias is a 1024-dimensional vector.

[0054] The concatenated complete input sequence is fed into the self-attention computation layer of the large language model decoder. The large language model decoder employs a causal masking mechanism for autoregressive generation. This mechanism is implemented using an upper triangular mask matrix, where all elements at the upper triangular positions are set to negative infinity. This ensures that the t-th generation step can only focus on the first t-1 generated tokens of the complete input sequence and the complete input sequence itself, ignoring subsequent ungenerated tokens. In each generation step, the large language model decoder performs a layer-by-layer forward computation through 24 decoding layers based on the currently generated token sequence and the complete input sequence. At the output of the last decoding layer, a 1024-dimensional hidden state vector is generated. This hidden state vector is mapped to a 50000-dimensional vocabulary space through a linear output projection layer. The weight matrix of the linear output projection layer has a dimension of 1024 × 50000, and the bias is a 50000-dimensional vector. The Softmax function is applied to the vocabulary space vector to calculate the probability distribution of the next token. in Indicates the first The output token of each generation step; Indicates the first The sequence of all tokens generated before the step; Represents the complete input sequence; This indicates that the last layer of the large language model decoder is in the first... The 1024-dimensional hidden state vector output by the step; This represents the weight matrix of the linear output projection layer, with a dimension of 50000×1024; This represents the bias vector of the linear output projection layer, with a dimension of 50,000. The output of the Softmax function is a 50,000-dimensional probability vector, where each element corresponds to the generation probability of a token in the vocabulary.

[0055] In the first generation step, the generated token sequence is empty, and the input is only the complete input sequence. The large language model decoder predicts the probability distribution of the first output token based on the complete input sequence and selects the token with the highest probability as the first output token. The first output token is then added to the generated token sequence. In each subsequent generation step, the generated token sequence and the complete input sequence are input together into the large language model decoder to calculate the probability distribution of the next token and select the token with the highest probability.

[0056] Repeat the position-by-position prediction steps described above until the generated token is the preset end-of-line token or the generated sequence length reaches the maximum generation length. The maximum generation length is set to 512, which covers the standard treatment recommendation text length requirement that includes the diagnostic criteria, risk stratification, and follow-up plan. When the generation process meets the termination condition, stop generation, and combine all the output tokens generated during the autoregressive generation process into a complete text in the output order. This text is the treatment recommendation text.

[0057] A cluster search strategy is employed in the autoregressive generation process. Specifically, in each generation step, instead of retaining a single token selection, the K candidate sequences with the highest probabilities are retained, where K is set to 4. Setting K to 4 strikes a balance between computational efficiency and decoding diversity. In the first generation step, the four tokens with the highest probabilities are selected from the vocabulary as the beginnings of four candidate sequences. In the second generation step, these four candidate sequences are input into the large language model decoder to obtain their respective next token probability distributions. The log probability of the expanded candidate sequences is calculated, and the four candidate sequences with the highest overall log probability are retained from all expanded sequences. This process continues, retaining four candidate sequences in each generation step until a candidate sequence generates an end-of-session token or the maximum generation length is reached. After all candidate sequences have been generated, the sequence with the highest overall log probability is selected as the final output diagnostic suggestion text.

[0058] The fine-tuning process of the large language model decoder is performed on a dataset containing fundus image features, medical history features, and corresponding treatment suggestion texts. Each sample in the training data includes enhanced inference context and labeled treatment suggestion text, formatted according to paragraphs outlining diagnostic criteria, risk stratification, and follow-up plans. During training, the enhanced inference context, corresponding system role cue vectors, and output format constraint vectors are concatenated into a complete input sequence, which is then input into the large language model decoder. The cross-entropy loss function is used to calculate the loss between the model's predicted token sequence and the labeled token sequence. The optimizer is AdamW, with a learning rate of 0.00002, a weight decay coefficient of 0.01, a batch size of 8, and 5 training epochs. Through instruction fine-tuning, the large language model decoder learns to follow system role cue cues and output format constraints to generate compliant treatment suggestion text.

[0059] In practical implementation, a fundus diagnosis and treatment suggestion generation system based on a large language model includes a memory, a processor, and a computer program stored in the memory and running on the processor. The memory uses a non-volatile storage medium to store computer program instructions, pre-trained model parameters, a pre-built diagnosis and treatment knowledge base, and intermediate data generated during system operation. The processor uses a central processing unit or a graphics processing unit to execute the computer program in the memory.

[0060] In practice, when the processor executes the computer program, it implements each step of the method for generating fundus diagnosis and treatment suggestions based on a large language model. The computer program is logically divided into multiple functional modules, each corresponding to one of the implemented method steps.

[0061] In specific implementation, please refer to Figure 3The fundus image set and the patient's clinical text record are obtained through the system input interface. The fundus image set contains multiple frames of fundus images, and the patient's clinical text record contains structured electronic medical record data from each visit. After receiving the fundus image set and the patient's clinical text record, the processor calls the lesion semantic anchoring processing module to perform lesion semantic anchoring processing.

[0062] The lesion semantic anchoring processing module consists of a fundus multimodal feature extraction network, which includes a lesion candidate region generation subnetwork and a semantic segmentation head. The lesion candidate region generation subnetwork employs an anchor-boundary-based region proposal network, comprising a feature extraction backbone network, an intermediate convolutional layer, a classification branch, and a regression branch. The feature extraction backbone network, composed of multiple stacked convolutional and pooling layers, is used to extract shared feature maps from fundus images. The intermediate convolutional layer has a 3×3 kernel size and 256 output channels, used to transform the shared feature maps. The classification branch is implemented by a 1×1 convolutional layer with 18 output channels, corresponding to the foreground and background confidence scores for each of the nine anchor bounding boxes at each spatial location. The regression branch is implemented by a 1×1 convolutional layer with 36 output channels, corresponding to the bounding box coordinate offset of each anchor bounding box. The anchor box dimensions are set to 32×32, 64×64, and 128×128, with aspect ratios of 0.5, 1.0, and 2.0, respectively. The cross-union threshold in the non-maximum suppression algorithm is set to 0.5.

[0063] The semantic segmentation head is a fully convolutional subnetwork with an encoder-decoder structure. The encoder progressively reduces spatial resolution and increases the number of channels, while the decoder restores spatial resolution through upsampling. The final output is a probability map with the number of channels equal to the total number of lesion categories (CS). The output of the semantic segmentation head is concatenated with the spatial location encoding by channel dimension. The spatial location encoding is generated by jointly encoding the normalized geometric center coordinates of the lesion candidate region bounding box and the lesion semantic label. This joint encoding is achieved through a learnable embedding layer and a sinusoidal location encoding function. In the expression of the sinusoidal location encoding function, the dimension index parameter ranges from 0 to 31, and the total dimension of the location encoding is 64. The embedding dimension of the lesion semantic label is 64. The normalized geometric center coordinates are converted into 64-dimensional vectors by the sinusoidal location encoding function, and the three are added element-wise to obtain the spatial location encoding.

[0064] In practical implementation, the processor invokes the medical history state reconstruction module to perform temporal medical history state reconstruction processing on the patient's clinical text records. The medical history state reconstruction module extracts multiple time-point visit record fields from the patient's clinical text records and arranges them in chronological order according to the visit timestamps to form the original medical history time series. The medical history state reconstruction module then invokes a pre-trained medical history feature encoder. This encoder is a bidirectional encoder based on the Transformer architecture, consisting of six stacked Transformer encoding layers. Each layer contains a multi-head self-attention sublayer and a feedforward fully connected sublayer. The number of multi-head self-attention heads is 8, the feedforward fully connected hidden layer dimension is 2048, and the word embedding dimension is 512. The medical history feature encoder is pre-trained on a large medical text corpus using a masked language model task. During training, the masking ratio is 15%, the optimizer is Adam, the learning rate is 0.0001, the batch size is 32, and the training epochs are 10. The medical history feature encoder encodes the chief complaint text, past medical history text, and medication record text separately. The vector corresponding to the [CLS] marker is taken as the semantic vector. The three semantic vectors are concatenated horizontally and the sinusoidal position encoding of the visit timestamp is added to generate the medical history state vector. All medical history state vectors are stacked vertically to form the dynamic feature sequence of medical history.

[0065] In the specific implementation, the processor calls the instruction-fine-tuned large language model decoder for autoregressive generation. The large language model decoder is a 24-layer Transformer decoder with 16 attention heads, 4096 hidden layer dimensions, 1024 word embedding dimensions, and a vocabulary size of 50,000. The enhanced inference context front-end concatenates system role cue vectors, and the back-end concatenates output format constraint vectors. The system role cue vector consists of 16 learnable vectors of 1024 dimensions, and the output format constraint vector consists of 32 learnable vectors of 1024 dimensions. The concatenated complete input sequence is input to the large language model decoder. The decoder predicts the token probability distribution position by position using a causal masking mechanism, employing a bundle search strategy with a bundle width K set to 4. At each generation step, the four candidate sequences with the highest probabilities are selected and retained. Generation stops when an end-of-sequence token is generated or the maximum generation length of 512 is reached. The sequence with the highest overall logarithmic probability in the entire sequence is output as the diagnostic suggestion text.

[0066] In practice, the parameters of the fundus multimodal feature extraction network, medical history feature encoder, linear projection layer, rule encoder, large language model decoder, cue embedding matrix, and format embedding matrix stored in memory are all obtained through end-to-end joint fine-tuning. Joint fine-tuning is performed on a dataset containing fundus images, clinical text, and annotated treatment suggestion text, using the AdamW optimizer with a learning rate of 0.00002, weight decay of 0.01, and a batch size of 8. The processor loads these pre-trained and fine-tuned parameters when executing the computer program, implementing all steps of the fundus treatment suggestion generation method.

[0067] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for generating fundus diagnosis and treatment suggestions based on a large language model, characterized in that, The method includes: In response to the received set of fundus images and the patient's clinical text record, a pre-trained fundus multimodal feature extraction network is invoked to perform lesion semantic anchoring processing on the fundus image set, generating a set of lesion semantic feature maps carrying spatial location encoding; The patient's clinical text records are subjected to temporal history state reconstruction processing to generate a dynamic feature sequence of medical history with temporal continuity; The semantic feature map set of lesions and the dynamic feature sequence of medical history are input into the cross-modal query routing layer. The cross-modal query routing layer generates a query routing weight matrix by calculating the bidirectional attention mapping between the spatial distribution of lesions and the time nodes of medical history. The query routing weight matrix is ​​dynamically retrieved and enhanced by fusing it with the diagnostic rule vectors in the pre-built diagnostic knowledge base to generate an enhanced reasoning context; The enhanced reasoning context is input into a large language model decoder that has been fine-tuned by instructions, and autoregressive generation generates a text of treatment recommendations that includes diagnostic criteria, risk stratification, and follow-up plans.

2. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 1, characterized in that, The step involves calling a pre-trained fundus multimodal feature extraction network to perform lesion semantic anchoring processing on the fundus image set, generating a set of lesion semantic feature maps carrying spatial location encoding, including: Each frame of fundus image in the fundus image set is input into the lesion candidate region generation subnetwork of the fundus multimodal feature extraction network, and the lesion candidate region bounding box and corresponding region confidence score are extracted through the sliding window anchor point mechanism. For each candidate lesion region bounding box, the semantic segmentation head of the fundus multimodal feature extraction network is invoked to perform pixel-by-pixel semantic classification on the pixels inside the candidate lesion region bounding box, generating a lesion type probability map. Based on the lesion type corresponding to the maximum probability value in the lesion type probability map, assign a semantic label to the bounding box of the candidate lesion region. The geometric center coordinates of the bounding box of the lesion candidate region and its corresponding semantic label are jointly encoded into the spatial location code, and the spatial location code is superimposed with the lesion type probability map to generate the lesion semantic feature map set carrying the spatial location code.

3. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 2, characterized in that, The lesion candidate region generation subnetwork adopts an anchor-boundary-based region proposal network, and uses a non-maximum suppression algorithm to deduplicate candidate region bounding boxes whose overlap exceeds a preset threshold.

4. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 1, characterized in that, The patient's clinical text records are subjected to temporal history state reconstruction processing to generate a dynamic feature sequence of medical history with temporal continuity, including: Multiple time point medical record fields are extracted from the patient's clinical text records. Each medical record field includes a medical record timestamp, chief complaint text, past medical history text, and medication record text. Arrange the medical record fields into an original medical history time sequence according to the chronological order of the consultation timestamps; The pre-trained medical history feature encoder is invoked to perform semantic vectorization processing on the chief complaint text, past medical history text and medication record text in each field of the original medical history time series, generating corresponding chief complaint semantic vector, past medical history semantic vector and medication semantic vector; The chief complaint semantic vector, past medical history semantic vector, and medication semantic vector corresponding to the same medical record field are horizontally concatenated, and the position code of the medical record timestamp is added to the concatenated vector to generate the medical history status vector of the medical record field. The medical history status vectors of all medical record fields are stacked vertically in the order of the medical visit timestamps to generate the dynamic feature sequence of medical history with temporal continuity.

5. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 1, characterized in that, The semantic feature map set of lesions and the dynamic feature sequence of medical history are input into the cross-modal query routing layer. The cross-modal query routing layer generates a query routing weight matrix by calculating the bidirectional attention mapping between the spatial distribution of lesions and the time nodes of medical history, including: Each lesion semantic feature map in the lesion semantic feature map set is flattened into a one-dimensional lesion feature vector, and all one-dimensional lesion feature vectors are arranged in spatial coordinate order to construct a lesion feature matrix. Each medical history state vector in the dynamic medical history feature sequence is treated as a row to construct a medical history feature matrix; Calculate the matrix product between the transpose of the lesion feature matrix and the medical history feature matrix to obtain the original cross-modal correlation matrix; Perform row dimension normalization on each row of the original cross-modal association matrix to generate a first attention weight matrix that characterizes the association strength between each lesion feature and all medical history states. Perform column dimension normalization on each column of the original cross-modal association matrix to generate a second attention weight matrix that characterizes the association strength between each medical history state and all lesion features; The query routing weight matrix is ​​generated by multiplying the first attention weight matrix and the second attention weight matrix element by element.

6. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 5, characterized in that, The query routing weight matrix is ​​dynamically retrieved and enhanced by fusing it with the diagnostic rule vectors in the pre-built diagnostic knowledge base to generate an enhanced reasoning context, including: Extract all diagnosis and treatment rule vectors from the pre-built diagnosis and treatment knowledge base. Each diagnosis and treatment rule vector corresponds to a rule text containing the mapping relationship between disease diagnosis criteria and treatment recommendations. Based on the weight distribution in the query routing weight matrix, calculate the weighted sum of the lesion feature matrix and the medical history feature matrix to generate a comprehensive query vector; Calculate the cosine similarity between the comprehensive query vector and each of the treatment rule vectors, and sort them from high to low according to the cosine similarity, selecting the top several treatment rule vectors as candidate retrieval rule vectors; The candidate retrieval rule vector and the comprehensive query vector are fused across attention. Specifically, the comprehensive query vector is used as the query item and the candidate retrieval rule vector is used as the key item to calculate the attention-weighted rule enhancement vector. The rule enhancement vector and the comprehensive query vector are joined by a residual to generate the enhanced inference context.

7. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 6, characterized in that, Each rule text in the pre-built diagnostic knowledge base is pre-mapped into a diagnostic rule vector by a rule encoder trained with a contrastive learning objective.

8. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 1, characterized in that, The enhanced inference context is input into a large language model decoder that has been fine-tuned by instructions, and autoregressive methods generate treatment recommendation text containing diagnostic criteria, risk stratification, and follow-up plans, including: A preset system role cue vector is concatenated at the front end of the enhanced reasoning context. The system role cue vector is used to constrain the large language model decoder to generate output according to the tone of an ophthalmologist. A preset output format constraint vector is concatenated at the back end of the enhanced reasoning context. The output format constraint vector is used to instruct the large language model decoder to sequentially generate the diagnostic basis paragraph, the risk stratification paragraph, and the follow-up plan paragraph. The concatenated complete input sequence is input into the self-attention computation layer of the large language model decoder, and the probability distribution of the next token is predicted position by position through the causal masking mechanism; In each generation step, the generation probability of the next token is calculated based on the currently generated token sequence and the complete input sequence, and the token with the highest probability is selected as the output token of the current step. Repeat the position-by-position prediction step until a preset end-of-line token is generated or the maximum generation length is reached. Combine all generated output tokens in the output order to form the diagnosis and treatment suggestion text.

9. The method for generating fundus diagnosis and treatment suggestions based on a large language model according to claim 6, characterized in that, The autoregressive generation process employs a cluster search strategy, retaining the K candidate sequences with the highest probabilities in each generation step, and selecting the sequence with the highest overall logarithmic probability as the final output diagnostic and treatment suggestion text at the end of the generation process.

10. A fundus diagnosis and treatment suggestion generation system based on a large language model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for generating fundus diagnosis and treatment suggestions based on a large language model as described in any one of claims 1 to 9.

Images