Eye fundus fluorescence contrast report generation method and system based on visual large model

Abstract

The invention is suitable for the technical field of medical image processing, and provides a fundus fluorescence radiography report generation method based on a visual large model, and the method comprises the steps: pre-training a fundus fluorescence radiography visual large model, extracting the image features of a fundus fluorescence radiography sequence image through the fundus fluorescence radiography visual large model, and constructing a pseudo time sequence curve; based on the pseudo time sequence curve, key point detection is carried out, a plurality of phase periods are divided, and a phase weight is set for each phase period; calculating a comprehensive score for each frame of the pseudo time sequence curve based on the phase weight in combination with time novelty; based on the comprehensive score, combining a preset target key frame number to obtain a key frame sequence; based on the key frame sequence, a multi-view sequence image report generation model is combined, a Chinese diagnosis report is generated, the time sequence information and phase characteristics of the fundus fluorescence contrast sequence can be fully mined, the diagnosis report conforming to clinical specifications is generated, and the limitation of the prior art is effectively overcome.

Description

Technical Field

[0001] This application belongs to the field of medical image processing technology, and in particular relates to a method and system for generating fundus fluorescein angiography reports based on a large visual model. Background Technology

[0002] Fluorescein fundus angiography (FFA) is one of the main diagnostic techniques for fundus diseases. It involves injecting a fluorescent dye into the body and using a light beam of a specific wavelength to excite the dye, thus imaging the retinal structure, the only part of the body where blood vessels can be directly observed. Using an instrument with special lens filters, the flow of the dye in the retinal vascular network can be tracked. By analyzing the sequence of images obtained from fundus fluorescein angiography, doctors can better understand the fundus structure, describe the type of disease, and assess the effectiveness and safety of treatment. In this process, doctors usually need to select representative images for summarization and analysis. However, manual selection is costly and has limitations such as subjectivity, high labor intensity, and low efficiency.

[0003] In recent years, artificial intelligence has been introduced into existing technologies for screening fundus fluorescein angiography (FFA) and generating image reports. For example, representative screening based on K-means or comprehensive selection based on facility location function (FMAC) can improve the diversity and global coverage of screening sequences to a certain extent. However, the lack of temporal structure or phase knowledge constraints during screening leads to redundancy in the screening results, affecting the generation of subsequent image reports. Similarly, in image report generation, image-to-text generation based on CNN-RNN / attention mechanisms or end-to-end methods based on Transformer encoding-decoding are used. However, neither method considers the temporal continuity and phase differences unique to FFA sequences when generating image reports, resulting in the loss of important information.

[0004] Therefore, there is an urgent need for a method that can fully extract the temporal information and phase characteristics of sequence images and generate diagnostic reports that conform to clinical standards, in order to overcome the limitations of the existing technologies mentioned above. Summary of the Invention

[0005] This application provides a method and system for generating fundus fluorescein angiography reports based on a large visual model, which can solve one of the problems of the prior art mentioned above.

[0006] In a first aspect, embodiments of this application provide a method for generating fundus fluorescein angiography reports based on a large visual model, including: A pre-trained large-scale visual model for fundus fluorescein angiography was used to extract image features from fundus fluorescein angiography sequence images and construct pseudo-time-series curves. Based on the pseudo-time series curve, key point detection is performed, multiple period periods are divided, and a period weight is set for each period period. Based on the phase weights and combined with temporal novelty, a comprehensive score is calculated for each frame of the pseudo-time series curve. Based on the comprehensive score and the preset target number of key frames, a key frame sequence is obtained; Based on the keyframe sequence, and combined with the multi-view sequence image report generation model, a Chinese diagnostic report is generated.

[0007] Furthermore, the large-scale visual model of fundus fluorescein angiography includes a student network, a teacher network, a projection head module, a distillation alignment module, and a mask modeling module; The pre-trained fundus fluorescein angiography visual model includes: Construct a sample dataset and generate an unlabeled dataset based on the sample dataset; In the projection head module, the high-dimensional semantic features output by the student network and the teacher network are mapped to the contrast space for distillation loss calculation. In the distillation alignment module, two different random augmented views are generated for each unlabeled image from the unlabeled dataset, and then input into the student network and the teacher network respectively for distillation alignment. In the mask modeling module, the output features of the teacher network are used as the supervision target, information missing training is performed on the student network, and mask modeling loss is calculated. Based on distillation loss and mask modeling loss, a comprehensive loss function is generated, and the training of the fundus fluorescein angiography visual model is guided by the comprehensive loss function.

[0008] Furthermore, in the mask modeling module, using the teacher network as the supervised target, information-deficient training is performed on the student network, and mask modeling loss is calculated, including: In the student network, the input image is randomly partially occluded, and feature recovery prediction is performed based on the unoccluded part to obtain the predicted feature representation; In the teacher network, a complete input image is used as input, and a global feature representation is output as the supervision target. Based on the predicted feature representation and the global feature representation, and combined with the mask modeling loss function, the mask modeling loss is calculated.

[0009] Furthermore, the step of extracting image features from the fundus fluorescein angiography sequence images and constructing a pseudo-time-series curve using the large visual model of fundus fluorescein angiography includes: Using the aforementioned large-scale visual model of fundus fluorescein angiography, the first image feature vector of each fundus fluorescein angiography sequence image is extracted to generate the first image feature set. An intensity index is calculated for each of the first image feature vectors to generate an intensity sequence. The intensity sequence is then smoothed and normalized to generate a pseudo-time series curve.

[0010] Furthermore, based on the pseudo-time series curve, key point detection is performed to divide multiple phase time periods, including: Based on a height threshold and a minimum peak distance, the main peak value is determined on the pseudo-time series curve. The height threshold is the median of the pseudo-time series curve, and the minimum peak distance is a first preset proportion of the sequence length of the pseudo-time series curve. A trough point is determined before the main peak, and a significant rise point is located after the trough point. The significant rise point is determined as the arterial phase start point. The trough point is the point where the intensity index is lower than the second preset ratio, and the significant rise point is the point where the rise judgment threshold is exceeded for the first time. All frames prior to the start of the arterial phase are divided into the pre-angiographic phase. Centered on the main peak, a preset proportion of the sequence length of the pseudo-time curve is taken before and after it as the arteriovenous phase; The period from the start of the arterial phase to the start of the arteriovenous phase is defined as the arterial phase. The point from the end of the arteriovenous phase to the third preset proportion of the sequence length of the pseudo-time curve is divided into the venous phase; All frames following the venous phase are classified as late stage.

[0011] Furthermore, the step of calculating a comprehensive score for each frame of the pseudo-time series curve based on the phase weights and temporal novelty includes: For each frame of the pseudo-temporal curve, the Euclidean distance of the intensity indices of adjacent frames is calculated and normalized to obtain the temporal novelty of each frame. The phase weight is determined based on the phase time period in which the corresponding frame is located; For each frame of the pseudo-time series curve, a comprehensive score is calculated based on the temporal novelty and the phase weight.

[0012] Furthermore, obtaining the keyframe sequence based on the comprehensive score and a preset target number of keyframes includes: Set the target number of keyframes for the keyframe sequence; The first and last frames of the pseudo-timing curve are selected as mandatory frames, and the other frames of the pseudo-timing curve are selected as candidate frames. In the candidate frames, they are traversed in descending order according to the comprehensive score, and filtered in combination with a preset filtering mechanism. If the target number of key frames is reached or the candidate frames have been traversed, the filtering stops. If the target number of key frames is not met after the filtering is stopped, then frame supplementation is performed based on the facility location function until the target number of key frames is met or there are no available candidate frames.

[0013] Furthermore, the multi-view sequence image report generation model includes a large-scale visual model of fundus fluorescein angiography, a multi-view sequence visual projector, and a text decoder; The process of generating a Chinese diagnostic report based on the keyframe sequence and combined with a multi-view sequence image report generation model includes: The keyframe sequence is input into the fundus fluorescein angiography visual model to extract visual features, output the second image feature vector, and generate the second image feature set. In the multi-view visual projector, each of the second image feature vectors is projected onto the text semantic space, visual-language modality alignment is performed, and multi-level feature fusion is carried out to generate a sequence representation; The sequence representation is input into the text decoder, and combined with the cross-entropy loss function, a Chinese diagnostic report is generated word by word.

[0014] Furthermore, the multi-view sequence visual projector includes a linear projection layer, a visual fusion module, a local enhancement module, a global view module, and a position encoding module; The step of projecting each of the second image feature vectors onto the text semantic space, performing visual-linguistic modality alignment, and performing multi-level feature fusion to generate a sequence representation includes: In the linear projection layer, each of the second image feature vectors is projected onto the text latent space to generate a projection feature set; In the visual fusion module, a multi-head self-attention mechanism is used to fuse the multi-view information of each projection feature in the projection feature set to generate a multi-view feature set. In the local enhancement module, one-dimensional convolution is performed on each view feature of the multi-view feature set to obtain a convolution feature set; In the global view module, multiple convolutional features from the convolutional feature set are aggregated into a global feature vector of fixed length. In the location encoding module, the global feature vector is concatenated with the projected feature set, and combined with the location encoding matrix to output a sequence representation.

[0015] Secondly, embodiments of this application provide a fundus fluorescein angiography report generation system based on a large visual model, including: First processing module: used for pre-training a large visual model of fundus fluorescein angiography, extracting image features from fundus fluorescein angiography sequence images through the large visual model of fundus fluorescein angiography, and constructing pseudo-time series curves; The second processing module is used to perform key point detection based on the pseudo-time series curve, divide the time into multiple phase periods, and set phase weights for each phase period. The third processing module calculates a comprehensive score for each frame of the pseudo-time series curve based on the phase weights and temporal novelty. The fourth processing module is used to obtain a keyframe sequence based on the comprehensive score and a preset target number of keyframes. The fifth processing module is used to generate a Chinese diagnostic report based on the keyframe sequence and in conjunction with the multi-view sequence image report generation model.

[0016] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for generating fundus fluorescein angiography reports based on a large visual model.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium, including a computer program stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the above-described method for generating fundus fluorescein angiography reports based on a large visual model.

[0018] The beneficial effects of the embodiments in this application compared with the prior art are: This application discloses a method for generating fundus fluorescein angiography reports based on a large visual model. A large visual model for fundus fluorescein angiography is pre-trained to extract image features from fundus fluorescein angiography sequences. Considering the temporal sequence of fundus fluorescein angiography, a pseudo-temporal curve is generated. Then, based on the pseudo-temporal curve, keyframes are selected through keypoint detection and phase segmentation, combined with a DIP scoring mechanism that combines phase weights and temporal novelty, achieving multi-objective optimization under diversity constraints. By selecting these keyframes, the number of images requiring processing is significantly reduced, thus significantly reducing the computational burden while maintaining the integrity of diagnostic information. Furthermore, in generating diagnostic reports, an architecture combining a large-scale visual model of fundus fluorescein angiography, a multi-view sequence visual projector, and a text decoder is adopted to construct a multi-view sequence image report generation model. Through unified feature extraction from the large-scale visual model of fundus fluorescein angiography, combined with the hierarchical fusion mechanism of the multi-view sequence visual projector, and based on the Transformer text decoder, accurate modeling of fundus fluorescein angiography sequences and high-quality diagnostic report generation are achieved. Moreover, it can fully explore the temporal information and phase characteristics of fundus fluorescein angiography sequences to generate diagnostic reports that conform to clinical standards, effectively overcoming the limitations of existing technologies. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for generating fundus fluorescein angiography reports based on a large visual model, according to an embodiment of the present invention. Figure 2 yes Figure 1 The flowchart illustrating the keyframe sequence generation process in the embodiment shown is as follows; Figure 3 This is a schematic diagram of the structure of a fundus fluorescein angiography report generation system based on a large visual model, provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] Please see Figure 1 As shown, this invention is a method for generating fundus fluorescein angiography reports based on a large visual model, comprising the following steps: S100, Pre-trained fundus fluorescein angiography visual model: Using the fundus fluorescein angiography visual model, image features of fundus fluorescein angiography sequence images are extracted to construct pseudo-time-series curves; In some embodiments, the large visual model of fundus fluorescein angiography includes a student network, a teacher network, a projection head module, a distillation alignment module, and a mask modeling module; In this application, the Fundus Fluorescence Imaging Visual Large Model (FFA-VFM) uses Vision Transformer (ViT) as the backbone network and implements teacher-student self-distillation learning based on the DINOv2 self-supervised learning framework. It can obtain highly semantic and strongly generalized feature representations from large-scale fundus fluorescein imaging data under unlabeled conditions, providing a unified visual feature foundation for subsequent keyframe screening and report generation. In a preferred embodiment, the Fundus Fluorescence Imaging Visual Large Model includes a student network, a teacher network, a projection head module, a distillation alignment module, and a mask modeling module.

[0028] Specifically, the student network adopts the Vision Transformer-Base (ViT-B / 14) structure, which contains 12 Transformer blocks. Each layer consists of a multi-head self-attention (MSA) layer and a feed-forward network (FFN). It takes enhanced fundus fluorescein angiography sequence images as input and outputs the global and local features of the images. During training, it aligns with the teacher network through contrastive loss to learn robust feature representations. The global features are 768-dimensional high-dimensional semantic features, and the local features include 16×16 patches, that is, the student network divides the image into 16×16 image patches, and the features of each image patch are 768-dimensional high-dimensional semantic features.

[0029] More specifically, the structure of the teacher network is the same as that of the student network. During training, the teacher network does not participate in backpropagation; that is, its parameters are not updated through gradient descent, but only through the moving average EMA. Therefore, the teacher network provides a more stable and smoother target distribution to guide the training of the student network. In one embodiment, the parameter update rule of the teacher network is as follows: ,in, Indicates the first The parameters of the teacher network after the next iteration Indicates the first The parameters of the student network after the next iteration. The momentum coefficient is preferably m=0.996.

[0030] Thus, the teacher network learns from the student network but also guides the evolution of the student network. The EMA mechanism of the teacher network provides a smooth optimization path and learns robust features that are invariant to data augmentation through contrastive loss, thereby improving the reliability of the final large-scale visual model of fundus fluorescein angiography.

[0031] Furthermore, the projection head module is positioned after the teacher network and the student network. It consists of several layers of linear transformations and nonlinear activation functions. In one embodiment, its structure specifically includes two linear layers, an intermediate GELU activation function, and a layer normalization operation. The output dimension is 256. Its main function is to map the high-dimensional semantic features output by the teacher network and the student network to the contrast space for the self-supervised distillation loss calculation of the subsequent distillation alignment module. Understandably, the distillation alignment module is used to perform global semantic alignment through distillation loss.

[0032] Furthermore, to enhance the model's local structure modeling capability, a mask modeling module is set up. During training, the input image of the student network is randomly partially masked, and feature recovery prediction is performed on the unmasked part. Meanwhile, the complete input image is input into the teacher network, and its output of the global features of the complete view is used as the supervision target to minimize the prediction error of the student network at the corresponding masked position, thereby achieving local structure learning.

[0033] In summary, during the training process, the distillation alignment module and the mask modeling module enable the large-scale visual model of fundus fluorescein angiography to take into account both global semantic alignment and local structural learning, thereby improving the reliability of the model.

[0034] In some embodiments, the pre-trained fundus fluorescein angiography visual large model includes: Construct a sample dataset and generate an unlabeled dataset based on the sample dataset; In the projection head module, the high-dimensional semantic features output by the student network and the teacher network are mapped to the contrast space for distillation loss calculation. In the distillation alignment module, two different random augmented views are generated for each unlabeled image from the unlabeled dataset, and then input into the student network and the teacher network respectively for distillation alignment. In the mask modeling module, the output features of the teacher network are used as the supervision target, information missing training is performed on the student network, and mask modeling loss is calculated. Based on distillation loss and mask modeling loss, a comprehensive loss function is generated, and the training of the fundus fluorescein angiography visual model is guided by the comprehensive loss function.

[0035] Specifically, a sample dataset is constructed, where each sample consists of a set of fundus fluorescein angiography sequence images and a corresponding structured report in Chinese and English. All fundus fluorescein angiography sequence images in the sample are uniformly resized to 224×224 pixels and normalized. Then, they are divided into training, validation, and test sets according to the proportion. In the subsequent training process, all fundus fluorescein angiography sequence images in the training set and other publicly available fundus fluorescein angiography datasets, such as 2023 APTOS and MM-Retinal, are used to construct a large-scale unlabeled dataset for the self-supervised pre-training of the subsequent fundus fluorescein angiography visual model. The training objective is to enable the model to learn a more discriminative, generalizable, and semantically consistent visual representation of fundus fluorescein angiography images.

[0036] Furthermore, during training, a batch of unlabeled images is sampled from the unlabeled dataset for training. For each unlabeled image in the same batch, two different random augmented views are generated: a first view and a second view. The first view is input into the student network and normalized using Softmax to output the target distribution. Similarly, the second view is input into the teacher network and normalized using Softmax to output a stable target distribution. Then, the distillation loss is calculated using the Kullback–Leibler divergence (KL Divergence). Specifically, ,in, This represents the output of the i-th unlabeled image in the training batch through the teacher network. Similarly, This represents the output of the i-th unlabeled image in the training batch through the student network. It is worth noting that during the training process, the results of the distillation loss are only backpropagated to the student network for parameter updates, while the parameters of the teacher network are updated through EMA.

[0037] In some embodiments, the step of using the teacher network as the supervised target, performing information-deficient training on the student network, and calculating the mask modeling loss in the mask modeling module includes: In the student network, the input image is randomly partially occluded, and feature recovery prediction is performed based on the unoccluded part to obtain the predicted feature representation; In the teacher network, a complete input image is used as input, and a global feature representation is output as the supervision target. Based on the predicted feature representation and the global feature representation, and combined with the mask modeling loss function, the mask modeling loss is calculated.

[0038] Furthermore, during training, the input image of the student network is divided into multiple image blocks, and some image blocks are randomly occluded. Feature recovery is performed based on the unoccluded image blocks to obtain the predicted feature representation. The teacher network is then fed the complete input image, and the final output is the global feature representation of the corresponding student network, which serves as the supervision target for the predicted feature representation to measure the prediction error of the predicted feature representation.

[0039] In a preferred embodiment, the random occlusion ratio of the input image to the student network is 40%–60%. During the occlusion process, each image patch is removed or replaced with a learnable mask marker, such as [MASK], to simulate the information loss of the input image. The image patch features at the occlusion location are predicted by combining the unoccluded image patches, and finally a predicted feature representation is generated. Finally, the predicted feature representation at the occlusion location is compared with the teacher network. At the same time, the mask modeling loss function is combined to minimize the prediction error of the student network at the occlusion location and quantify the mask modeling loss.

[0040] More specifically, the mask modeling loss function is as follows: ,in, This represents the set of occluded image patches. and These represent the teacher network and the student network respectively in the [number]th [year]. Image features output from each image patch location This is the cross-entropy loss function.

[0041] Specifically, the mask modeling mechanism described above enables the model to not only learn global semantic alignment, but also effectively capture local details such as fundus vascular texture and leakage edges, significantly improving the model's performance in fundus fluorescence angiography analysis tasks.

[0042] Furthermore, combining the distillation loss and mask modeling loss mentioned above, a comprehensive loss function is generated. Specifically, the comprehensive loss function is as follows: ,in, This represents a hyperparameter used to balance the contributions of different loss terms.

[0043] In some embodiments, the step of extracting image features from fundus fluorescein angiography sequence images and constructing pseudo-time-series curves using the large visual model of fundus fluorescein angiography includes: Using the aforementioned large-scale visual model of fundus fluorescein angiography, the first image feature vector of each fundus fluorescein angiography sequence image is extracted to generate the first image feature set. An intensity index is calculated for each of the first image feature vectors to generate an intensity sequence. The intensity sequence is then smoothed and normalized to generate a pseudo-time series curve.

[0044] In this application, the pre-trained fundus fluorescein angiography visual model is used to extract image features from a set of fundus fluorescein angiography sequence images. Considering the temporal nature of fundus fluorescein angiography, a pseudo-temporal curve is generated, providing a unified, transferable visual coding foundation with rich semantic information for subsequent keyframe screening and report generation.

[0045] Specifically, a sequence of images acquired via fundus fluorescein angiography, arranged in the order of acquisition but without reliable timestamps, is obtained. The 768-dimensional global feature vector of each frame in the sequence of images was extracted using a large visual model of fundus fluorescein angiography, which is the first image feature vector, thus obtaining the corresponding first image feature set. ,in .

[0046] Furthermore, a pseudo-temporal curve is constructed based on this first image feature set to simulate the dynamic changes in the imaging process. Specifically, the L2 norm of the feature vector of the first image in each frame is calculated to measure the feature intensity, and the intensity index of each frame is as follows: The corresponding intensity sequence is obtained from this, and Savitzky-Golay filtering is applied to the intensity sequence to smooth it. Then, the smoothed curve is normalized by min-max to obtain a pseudo-time series curve in the range of [0,1], which is convenient for subsequent phase division.

[0047] S200. Based on the pseudo-time series curve, perform key point detection, divide the time into multiple phase periods, and set phase weights for each phase period. In this application, key point detection is performed on the pseudo-time curve obtained in step S100 above to delineate phase boundaries, which are then used to calculate a comprehensive score for each frame on the pseudo-time curve, thereby selecting key frame sequences. Specifically, phase is introduced in the key frame sequence selection process, so that when selecting each fundus fluorescein angiography sequence image, the temporal information and phase characteristics of the sequence image can be fully explored, thereby generating a diagnostic report that conforms to clinical standards.

[0048] In some embodiments, step S200 above includes: Based on a height threshold and a minimum peak distance, the main peak value is determined on the pseudo-time series curve. The height threshold is the median of the pseudo-time series curve, and the minimum peak distance is a first preset proportion of the sequence length of the pseudo-time series curve. A trough point is determined before the main peak, and a significant rise point is located after the trough point. The significant rise point is determined as the arterial phase start point. The trough point is the point where the intensity index is lower than the second preset ratio, and the significant rise point is the point where the rise judgment threshold is exceeded for the first time. All frames prior to the start of the arterial phase are divided into the pre-angiographic phase. Centered on the main peak, a preset proportion of the sequence length of the pseudo-time curve is taken before and after it as the arteriovenous phase; The period from the start of the arterial phase to the start of the arteriovenous phase is defined as the arterial phase. The point from the end of the arteriovenous phase to the third preset proportion of the sequence length of the pseudo-time curve is divided into the venous phase; All frames following the venous phase are classified as late stage.

[0049] In this embodiment, key points include the main peak, trough, and significant rise point. The pseudo-timeline curve includes the pre-angiography phase, arterial phase, arteriovenous phase, venous phase, and late phase. The main peak is used to determine the arteriovenous phase of the pseudo-timeline curve, while the main peak and trough are combined to determine the significant rise point, which corresponds to the start of the arterial phase. The pre-angiography phase is located before the arterial phase, the arterial phase is located between the pre-angiography phase and the arteriovenous phase, the venous phase is located after the arteriovenous phase, and the late phase is located after the venous phase.

[0050] Specifically, for determining the dominant peak, a height threshold and a minimum peak distance are determined based on the pseudo-time series curve. The height threshold is used to filter out peaks with smaller values, while the minimum peak distance avoids detecting peaks that are too close together. In a preferred embodiment, the height threshold corresponds to the median of each intensity index of the pseudo-time series curve, and the minimum peak distance is based on a first preset ratio combined with the sequence length of the pseudo-time series curve. For example, in one embodiment, the first preset ratio is 5%, then the minimum peak distance is the sequence length × 5%. The minimum peak distance represents the minimum distance between two adjacent peaks. Thus, the sequence length can correspond to the number of points in the pseudo-time series curve, i.e., the number of fundus fluorescein angiography sequence images. Specifically, in the pseudo-time series curve, multiple local peaks are initially determined based on the height threshold, and then the final dominant peak is selected from these multiple local peaks by combining the minimum peak distance. In one embodiment, when selecting the dominant peak using the minimum peak distance, multiple dominant peaks may occur. In this case, the intensity index values ​​of each dominant peak are compared, and the dominant peak with the largest value is taken as the unique dominant peak for determining subsequent phases.

[0051] More specifically, for a significant rise point, it corresponds to the start of the arterial phase, which is located before the arteriovenous phase. Therefore, the significant rise point is determined before the main peak. Before determining the significant rise point, it is necessary to screen out the trough points. The trough point is the point where the intensity index is lower than the main peak by a second preset proportion. In a preferred embodiment, the second preset proportion is 30%. It is worth noting that if there are multiple trough points in the pseudo-time series curve where the intensity index is lower than the main peak by the second preset proportion, the trough point closest to the main peak on the time axis is selected as the unique trough point. Based on this unique trough point, the point where the intensity significantly rises from the pre-angiography stage to the arterial stage is found. That is, the point where the intensity first rises above the rise judgment threshold is selected as the significant rise point, and the start of the arterial phase is determined accordingly. In a preferred embodiment, the rise judgment threshold = baseline mean + 1.5 times the standard deviation, where the baseline mean is the average value of the intensity index of the pseudo-time series curve, and the standard deviation is the standard deviation of the intensity index of the pseudo-time series curve.

[0052] More specifically, the phase boundaries are divided based on the aforementioned main peak value and the arterial phase start point. The pre-angiography phase includes all frames before the arterial phase start point; the arterial phase extends from the arterial phase start point to the arteriovenous phase start point; the arteriovenous phase is defined by taking 8%–12% of the sequence length before and after the main peak value as a window; the venous phase extends from the end of the arteriovenous phase to 60% of the sequence length; and the late phase includes all frames after the venous phase.

[0053] Furthermore, different phase weights are assigned to different time periods. In a preferred embodiment, the weights for the pre-angiography phase, arterial phase, arteriovenous phase, venous phase, and late phase are {0.3, 1.5, 1.8, 1.2, 1.0}, respectively. The determination of these phase weights is an empirical weight design that combines the importance of medical phases and experimental parameter tuning. Specifically, the arteriovenous phase is the most critical for lesion filling and leakage, so it is assigned the highest weight of 1.8; the arterial phase shows early filling and is given the second highest weight of 1.5; the venous phase and late phase contain some supplementary information and are given medium weights, such as 1.2 and 1.0; the pre-angiography phase is mostly background or baseline images and is given the lowest weight, such as 0.3. This setup allows frames near the arteriovenous phase to have a moderate advantage in the overall score, while preserving temporal novelty, i.e. the contribution of feature changes to local dynamic information. Finally, when calculating the overall score of each frame, the phase weight is determined by combining the phase time period to which the frame belongs. The overall score is calculated by combining the phase weight, which is conducive to fully mining the phase features in the pseudo-time series curve.

[0054] S300. Based on the phase weights and combined with temporal novelty, calculate a comprehensive score for each frame of the pseudo-time series curve. In some embodiments, step S300 above includes: For each frame of the pseudo-temporal curve, the Euclidean distance of the intensity indices of adjacent frames is calculated and normalized to obtain the temporal novelty of each frame. The phase weight is determined based on the phase time period in which the corresponding frame is located; For each frame of the pseudo-time series curve, a comprehensive score is calculated based on the temporal novelty and the phase weight.

[0055] In this embodiment, for each frame in the pseudo-timing curve Calculate the Euclidean distance between the intensity indices of adjacent frames. Specifically, Then, it is normalized. This achieves the final temporal novelty.

[0056] Furthermore, based on the phase period corresponding to the frame, phase weights are determined, and combined with temporal novelty, a comprehensive score is calculated. Specifically, ,in, This represents the phase weight corresponding to the i-th frame. Indicates the temporal novelty of the i-th frame. This represents a hyperparameter used to balance the contribution of temporal novelty.

[0057] S400. Based on the comprehensive score and combined with the preset target number of key frames, a key frame sequence is obtained; In some embodiments, step S400 above includes: Set the target number of keyframes for the keyframe sequence; The first and last frames of the pseudo-timing curve are selected as mandatory frames, and the other frames of the pseudo-timing curve are selected as candidate frames. In the candidate frames, they are traversed in descending order according to the comprehensive score, and filtered in combination with a preset filtering mechanism. If the target number of key frames is reached or the candidate frames have been traversed, the filtering stops. If the target number of key frames is not met after the filtering is stopped, then frame supplementation is performed based on the facility location function until the target number of key frames is met or there are no available candidate frames.

[0058] In this embodiment, a target number of keyframes K is set for the keyframe sequence. Based on this target number of keyframes K, a multi-stage optimization selection method is used to screen keyframes on the pseudo-timeline curve. Specifically, the first and last frames on the pseudo-timeline curve are retained as mandatory frames to provide the start and end context for the keyframe sequence. For other candidate frames, a greedy selection method is used for screening. Specifically, each candidate frame is traversed in descending order according to its comprehensive score. If the maximum value of the cosine similarity between the candidate frame and each of the selected frames is lower than the threshold τ, it can be selected and added to the keyframe sequence. In a preferred embodiment, the threshold τ = 0.95. During the traversal, if the number of frames added to the keyframe sequence has reached the target number of keyframes K, the traversal can be terminated.

[0059] In addition, the traversal is terminated after all candidate frames have been traversed. In this case, if the number of key frame sequences added after the traversal is terminated still does not reach the target number of key frames K, then frame supplementation is performed based on the facility position function until the target number of key frames is met or there are no available candidate frames, and the filtered sequence image S, i.e. the above key frame sequence, is obtained.

[0060] In this application, based on pseudo-time series curves, key frame selection is achieved through key point detection and phase segmentation, combined with the DIP scoring mechanism of phase weight and temporal novelty, under diversity constraints. By filtering the key frames, the number of images that need to be processed is greatly reduced, and the computational burden is significantly reduced while maintaining the integrity of diagnostic information.

[0061] S500. Based on the keyframe sequence, combined with the multi-view sequence image report generation model, a Chinese diagnostic report is generated.

[0062] In some embodiments, the multi-view sequence image report generation model includes a large visual model of fundus fluorescein angiography, a multi-view sequence visual projector, and a text decoder; In this embodiment, the multi-view sequence image report generation model uses the keyframe sequence selected from the pseudo-time-series curve in steps S100-S400 as input to generate a Chinese diagnostic report. Furthermore, the multi-view sequence image report generation model includes a large-scale fundus fluorescein angiography visual model, a multi-view sequence visual projector, and a text decoder. The large-scale fundus fluorescein angiography visual model pre-trained in step S100 extracts visual features from the keyframe sequence, then fuses the features of each keyframe to generate a sequence representation, which is then input into the text decoder to generate the Chinese diagnostic report. In some embodiments, step S500 includes: The keyframe sequence is input into the fundus fluorescein angiography visual model to extract visual features, output the second image feature vector, and generate the second image feature set. In the multi-view visual projector, each of the second image feature vectors is projected onto the text semantic space, visual-language modality alignment is performed, and multi-level feature fusion is carried out to generate a sequence representation; The sequence representation is input into the text decoder, and combined with the cross-entropy loss function, a Chinese diagnostic report is generated word by word.

[0063] Specifically, the keyframe sequence is input into the fundus fluorescein angiography visual large model, and its input shape is [batch size, number of views, 3, 224, 224]. The batch size is the number of keyframe sequences processed by the model each time due to hardware resource limitations, such as 8, 16, 32, 64. In a preferred embodiment, the batch size is 16, and the number of views is the number of views in each keyframe sequence. In addition, the numbers 3, 224, and 224 represent the three color channels (RGB) of the image, the image width of 224, and the image height of 224, respectively.

[0064] More specifically, after visual feature extraction by the fundus fluorescein angiography visual model, the second image feature vector of each key frame is output, thereby generating the second image feature set, the output shape of which is [batch size, number of views, 768]. During the report generation training process, the fundus fluorescein angiography visual model can be frozen to stabilize optimization and avoid overfitting.

[0065] Furthermore, the feature vectors of each second image are projected into the text semantic space to achieve visual-language modality alignment, and multi-level feature fusion is completed along the time / view dimension to obtain a sequence representation that contains both global context and preserves local changes, which is used for the generation of subsequent Chinese diagnostic reports.

[0066] In some embodiments, the multi-view sequence visual projector includes a linear projection layer, a visual fusion module, a local enhancement module, a global view module, and a position encoding module; The step of projecting each of the second image feature vectors onto the text semantic space, performing visual-linguistic modality alignment, and performing multi-level feature fusion to generate a sequence representation includes: In the linear projection layer, each of the second image feature vectors is projected onto the text latent space to generate a projection feature set; In the visual fusion module, a multi-head self-attention mechanism is used to fuse the multi-view information of each projection feature in the projection feature set to generate a multi-view feature set. In the local enhancement module, one-dimensional convolution is performed on each view feature of the multi-view feature set to obtain a convolution feature set; In the global view module, multiple convolutional features from the convolutional feature set are aggregated into a global feature vector of fixed length. In the location encoding module, the global feature vector is concatenated with the projected feature set, and combined with the location encoding matrix to output a sequence representation.

[0067] Specifically, in the linear projection layer, the high-dimensional features output after visual feature extraction from the large visual model of fundus fluorescein angiography are projected onto the text latent space. ,in Let i be the second image feature vector of the i-th frame. Let be the projection matrix. For bias terms, Given the projection feature vector of the i-th frame, the final projection feature set is generated. ].

[0068] Furthermore, in the visual fusion module, based on the aforementioned projected feature set, a multi-head self-attention mechanism is used to fuse information features from multiple perspectives, generating a multi-view feature set. Specifically, ,in , , ; The feature matrix for all viewpoints in the projection feature set. , and These represent three learnable weight matrices. The similarity between "Q" and "K" is calculated, and then... The similarity score is converted into a weight, and then the weights are summed in combination with V to obtain a new representation for each frame, namely the viewpoint feature.

[0069] Furthermore, one-dimensional depthwise separable convolution is applied to the viewpoint features obtained after multi-view fusion to enhance local continuity. Specifically, For each perspective, This represents the feature of the i-th viewpoint after local convolution enhancement, where Conv1D_{depthwise} is used to represent a one-dimensional depthwise convolution. This represents a local sequence window centered at viewpoint i, while k represents the kernel size, which can be configured as needed while keeping the sequence length constant. This represents the learnable weights of pointwise convolution. It is worth noting that the sequence length remains unchanged during the convolution process.

[0070] Furthermore, in the global view module, the convolutional features are aggregated to generate a sequence-level global representation. This involves aggregating multiple convolutional features from the convolutional feature set into a fixed-length global feature vector. Specifically... ,in, This represents the aggregated global feature vector. For attention weights of each perspective, through Perform attention weight calculation. Let w denote the transpose of the learnable weight vector w. This represents the features obtained after multi-view fusion and local convolution enhancement.

[0071] Furthermore, a learned positional encoding method is employed, concatenating the global vector with the sequence vector to obtain the conditional context for the decoder. Specifically, ,in The location encoding matrix has an output shape of [batch size, number of views + 1, 512].

[0072] Furthermore, in the file decoder, as described above... Decoding is performed using conditional context. Specifically, the text decoder, based on the Transformer architecture, implements autoregressive generation from visual semantic features to diagnostic report text. Teacher-driven generation is employed during the training phase, while word-by-word generation occurs progressively during the inference phase. With the large-scale visual model of fundus fluorescein angiography frozen, the cross-entropy loss function is used to optimize the remaining parameters. The cross-entropy loss function is specifically... ,in, Indicates the length of the target sequence; Indicates the size of the vocabulary; Indicates the first The position The true label of each word; The model represents the first time. The position The predicted probability of each word is used to generate a Chinese diagnostic report.

[0073] Specifically, the aforementioned multi-view sequence image report generation model adopts an architecture of fundus fluorescein angiography visual large model + multi-view sequence visual projector + text decoder. Through unified feature extraction of the fundus fluorescein angiography visual large model, combined with the hierarchical fusion mechanism of the multi-view sequence visual projector, and based on the Transformer text decoder, it achieves accurate modeling of fundus fluorescein angiography sequences and generation of high-quality diagnostic reports. It can also fully explore the temporal information and phase characteristics of fundus fluorescein angiography sequences to generate diagnostic reports that meet clinical standards, effectively overcoming the limitations of existing technologies.

[0074] Please see Figure 3 As shown, the present invention also provides a fundus fluorescein angiography report generation system based on a large visual model, the system comprising: First processing module 201: used for pre-training a large visual model of fundus fluorescein angiography, extracting image features of fundus fluorescein angiography sequence images through the large visual model of fundus fluorescein angiography, and constructing pseudo-time series curves; The second processing module 202 is used to perform key point detection based on the pseudo-time series curve, divide the time into multiple phase periods, and set phase weights for each phase period. Third processing module 203: Based on the phase weights and combined with temporal novelty, calculates a comprehensive score for each frame of the pseudo-time series curve; Fourth processing module 204: used to obtain a key frame sequence based on the comprehensive score and a preset target number of key frames; The fifth processing module 205 is used to generate a Chinese diagnostic report based on the keyframe sequence and in conjunction with the multi-view sequence image report generation model.

[0075] It is understandable that, such as Figure 1 The content of the illustrated embodiment of the fundus fluorescein angiography report generation method based on a large visual model is applicable to the embodiment of the fundus fluorescein angiography report generation system based on a large visual model. The specific functions implemented by the embodiment of the fundus fluorescein angiography report generation system based on a large visual model are the same as those shown in the illustration. Figure 1 The illustrated method for generating fundus fluorescein angiography reports based on a large visual model is identical to the one shown, and achieves the same beneficial effects. Figure 1 The beneficial effects achieved by the illustrated embodiment of the fundus fluorescein angiography report generation method based on a large visual model are also the same.

[0076] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0078] Please see Figure 4As shown, this embodiment of the invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, it implements the method for generating fundus fluorescein angiography reports based on a large visual model as described in any of the above methods.

[0079] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0080] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0081] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0082] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for generating fundus fluorescein angiography reports based on a large visual model as described in any of the above methods.

[0083] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0084] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating fundus fluorescein angiography reports based on a large visual model, characterized in that, include: A pre-trained large-scale visual model for fundus fluorescein angiography was used to extract image features from fundus fluorescein angiography sequence images and construct pseudo-time-series curves. Based on the pseudo-time series curve, key point detection is performed, multiple period periods are divided, and a period weight is set for each period period. Based on the phase weights and combined with temporal novelty, a comprehensive score is calculated for each frame of the pseudo-time series curve. Based on the comprehensive score and the preset target number of key frames, a key frame sequence is obtained; Based on the keyframe sequence, and combined with the multi-view sequence image report generation model, a Chinese diagnostic report is generated.

2. The method as described in claim 1, characterized in that, The large-scale visual model of fundus fluorescein angiography includes a student network, a teacher network, a projection head module, a distillation alignment module, and a mask modeling module; The pre-trained fundus fluorescein angiography visual model includes: Construct a sample dataset and generate an unlabeled dataset based on the sample dataset; In the projection head module, the high-dimensional semantic features output by the student network and the teacher network are mapped to the contrast space for distillation loss calculation. In the distillation alignment module, two different random augmented views are generated for each unlabeled image from the unlabeled dataset, and then input into the student network and the teacher network respectively for distillation alignment. In the mask modeling module, the output features of the teacher network are used as the supervision target, information missing training is performed on the student network, and mask modeling loss is calculated. Based on distillation loss and mask modeling loss, a comprehensive loss function is generated, and the training of the fundus fluorescein angiography visual model is guided by the comprehensive loss function.

3. The method as described in claim 2, characterized in that, In the mask modeling module, the teacher network is used as the supervised target, and information-deficient training is performed on the student network, along with mask modeling loss calculation, including: In the student network, the input image is randomly partially occluded, and feature recovery prediction is performed based on the unoccluded part to obtain the predicted feature representation; In the teacher network, a complete input image is used as input, and a global feature representation is output as the supervision target. Based on the predicted feature representation and the global feature representation, and combined with the mask modeling loss function, the mask modeling loss is calculated.

4. The method as described in claim 1, characterized in that, The step of extracting image features from fundus fluorescein angiography sequence images and constructing pseudo-time-series curves using the large visual model of fundus fluorescein angiography includes: Using the aforementioned large-scale visual model of fundus fluorescein angiography, the first image feature vector of each fundus fluorescein angiography sequence image is extracted to generate the first image feature set. An intensity index is calculated for each of the first image feature vectors to generate an intensity sequence. The intensity sequence is then smoothed and normalized to generate a pseudo-time series curve.

5. The method as described in claim 1, characterized in that, The process involves key point detection based on the pseudo-time series curve, dividing the data into multiple phase periods, including: Based on a height threshold and a minimum peak distance, the main peak value is determined on the pseudo-time series curve. The height threshold is the median of the pseudo-time series curve, and the minimum peak distance is a first preset proportion of the sequence length of the pseudo-time series curve. A trough point is determined before the main peak, and a significant rise point is located after the trough point. The significant rise point is determined as the arterial phase start point. The trough point is the point where the intensity index is lower than the second preset ratio, and the significant rise point is the point where the rise judgment threshold is exceeded for the first time. All frames prior to the start of the arterial phase are divided into the pre-angiographic phase. Centered on the main peak, a preset proportion of the sequence length of the pseudo-time curve is taken before and after it as the arteriovenous phase; The period from the start of the arterial phase to the start of the arteriovenous phase is defined as the arterial phase. The point from the end of the arteriovenous phase to the third preset proportion of the sequence length of the pseudo-time curve is divided into the venous phase; All frames following the venous phase are classified as late stage.

6. The method as described in claim 1, characterized in that, The calculation of a comprehensive score for each frame of the pseudo-time series curve based on the phase weights and temporal novelty includes: For each frame of the pseudo-temporal curve, the Euclidean distance of the intensity indices of adjacent frames is calculated and normalized to obtain the temporal novelty of each frame. The phase weight is determined based on the phase time period in which the corresponding frame is located; For each frame of the pseudo-time series curve, a comprehensive score is calculated based on the temporal novelty and the phase weight.

7. The method as described in claim 1, characterized in that, The process of obtaining a keyframe sequence based on the comprehensive score and a preset target number of keyframes includes: Set the target number of keyframes for the keyframe sequence; The first and last frames of the pseudo-timing curve are selected as mandatory frames, and the other frames of the pseudo-timing curve are selected as candidate frames. In the candidate frames, they are traversed in descending order according to the comprehensive score, and filtered in combination with a preset filtering mechanism. If the target number of key frames is reached or the candidate frames have been traversed, the filtering stops. If the target number of key frames is not met after the filtering is stopped, then frame supplementation is performed based on the facility location function until the target number of key frames is met or there are no available candidate frames.

8. The method as described in claim 1, characterized in that, The multi-view sequence image report generation model includes a large visual model of fundus fluorescein angiography, a multi-view sequence visual projector, and a text decoder. The process of generating a Chinese diagnostic report based on the keyframe sequence and combined with a multi-view sequence image report generation model includes: The keyframe sequence is input into the fundus fluorescein angiography visual model to extract visual features, output the second image feature vector, and generate the second image feature set. In the multi-view visual projector, each of the second image feature vectors is projected onto the text semantic space, visual-language modality alignment is performed, and multi-level feature fusion is carried out to generate a sequence representation; The sequence representation is input into the text decoder, and combined with the cross-entropy loss function, a Chinese diagnostic report is generated word by word.

9. The method as described in claim 8, characterized in that, The multi-view sequence visual projector includes a linear projection layer, a visual fusion module, a local enhancement module, a global view module, and a position encoding module; The step of projecting each of the second image feature vectors onto the text semantic space, performing visual-linguistic modality alignment, and performing multi-level feature fusion to generate a sequence representation includes: In the linear projection layer, each of the second image feature vectors is projected onto the text latent space to generate a projection feature set; In the visual fusion module, a multi-head self-attention mechanism is used to fuse the multi-view information of each projection feature in the projection feature set to generate a multi-view feature set. In the local enhancement module, one-dimensional convolution is performed on each view feature of the multi-view feature set to obtain a convolution feature set; In the global view module, multiple convolutional features from the convolutional feature set are aggregated into a global feature vector of fixed length. In the location encoding module, the global feature vector is concatenated with the projected feature set, and combined with the location encoding matrix to output a sequence representation.

10. A fundus fluorescein angiography report generation system based on a large visual model, characterized in that, include: First processing module: used for pre-training a large visual model of fundus fluorescein angiography, extracting image features from fundus fluorescein angiography sequence images through the large visual model of fundus fluorescein angiography, and constructing pseudo-time series curves; The second processing module is used to perform key point detection based on the pseudo-time series curve, divide the time into multiple phase periods, and set phase weights for each phase period. The third processing module calculates a comprehensive score for each frame of the pseudo-time series curve based on the phase weights and temporal novelty. The fourth processing module is used to obtain a keyframe sequence based on the comprehensive score and a preset target number of keyframes. The fifth processing module is used to generate a Chinese diagnostic report based on the keyframe sequence and in conjunction with the multi-view sequence image report generation model.

Images