Neonatal fundus image description generation method and imaging method

Abstract

The present invention discloses a method for generating descriptions of neonatal fundus images and an imaging method, comprising obtaining existing neonatal fundus images and performing descriptions and preprocessing to construct a training dataset; constructing an initial model for generating descriptions of neonatal fundus images based on a feature extraction network, a feature mapping network, and a large language model, and training the model to obtain a neonatal fundus image description generation model; and automatically generating descriptions of neonatal fundus images using the obtained model. The method for generating descriptions of neonatal fundus images and the imaging method provided by the present invention combine the technical solutions of feature extraction, feature mapping, and a large language model, designing corresponding image feature extraction networks and feature mapping networks for neonatal fundus images, and employing a description generation scheme. These methods not only achieve targeted generation of descriptions and imaging of neonatal fundus images, but also achieve higher reliability and better accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing, and in particular relates to a method for generating descriptions of neonatal fundus images and an imaging method. Background Art

[0002] Neonatal fundus images are of great significance in both clinical practice and basic medical research. However, current clinical and basic medical applications require the prior interpretation of neonatal fundus images by professional medical imaging professionals or ophthalmologists. This manual interpretation process is not only time-consuming and labor-intensive, but also inefficient, with limited reliability and consistency.

[0003] Researchers have proposed methods for generating descriptions for general-purpose images. However, these methods are designed for general-purpose images, while neonatal fundus images have unique characteristics, such as sparse and thin blood vessels, a large optic nerve head, and blurred edges. These characteristics make existing methods for generating descriptions for general-purpose images unsuitable for neonatal fundus images. Summary of the Invention

[0004] One of the purposes of the present invention is to provide a method for generating descriptions of neonatal fundus images with high reliability and good accuracy.

[0005] A second object of the present invention is to provide an imaging method including the method for generating a description of a neonatal fundus image.

[0006] The method for generating a description of a neonatal fundus image provided by the present invention comprises the following steps:

[0007] S1. Obtain existing fundus images of the newborn;

[0008] S2 describes and preprocesses the neonatal fundus images obtained in step S1 to construct a training data set;

[0009] S3. Build an initial model for generating descriptions of neonatal fundus images based on an image feature extraction network, a feature mapping network, and a description generation module.

[0010] The initial model for generating descriptions of neonatal fundus images includes an image feature extraction network, a feature mapping network, and a description generation module connected in series. The image feature extraction network is used to extract features from input image data. The feature mapping network is used to map the extracted features to corresponding image descriptions. The description generation module is used to generate a description of the input image based on the obtained image features and corresponding text prompt words.

[0011] S4 using the training data set obtained in step S2, based on the likelihood function, the neonatal fundus image description generated in step S3 constructed initial model training, to obtain a neonatal fundus image description generation model;

[0012] S5. Using the neonatal fundus image description generation model obtained in step S4, automatically generate neonatal fundus image descriptions.

[0013] The step S2 specifically includes the following steps:

[0014] The following describes the various types of neonatal fundus images obtained in step S1;

[0015] The described fundus image of the newborn is preprocessed; the preprocessing includes the following steps:

[0016] For each set type, the fundus images of newborns of a type with a number of images less than a first set value are repeated several times; the fundus images of newborns of a type with a number of images greater than a second set value are discarded according to a set ratio;

[0017] The obtained neonatal fundus image is adjusted to a set resolution;

[0018] The neonatal fundus image after adjusting the resolution is divided into n square images; the neonatal fundus image after adjusting the resolution is reduced to the same size as the divided square images; finally, an original neonatal fundus image is preprocessed into n+1 images.

[0019] The step S3 includes the following steps:

[0020] An image feature extraction network is constructed based on the SigLip model to extract features from the input image data;

[0021] Building a feature mapping network based on the multi-layer perceptron model to map the extracted features to the corresponding image description;

[0022] A description generation module is constructed based on the Qwen2 model to generate a description of the input image based on the obtained image features and the corresponding text prompt words.

[0023] The image feature extraction network based on the SigLip model is constructed, which specifically includes the following steps:

[0024] The SigLip model is used to construct an image feature extraction network. The processing process of the image feature extraction network is expressed as: Where Z is the feature extraction result; X is the input image; The parameters are The processing function of the SigLip model;

[0025] An original neonatal fundus image and the corresponding preprocessed n images are collectively represented as ,in is the compressed original neonatal fundus image, is the corresponding preprocessed n images;

[0026] nn+1 images , are input into the image feature extraction network to obtain the feature extraction results for ,in for The feature extraction results of for The feature extraction results.

[0027] The feature mapping network constructed based on the multi-layer perceptron model specifically includes the following steps:

[0028] The input of the feature mapping network is the feature extraction result output by the image feature extraction network ;

[0029] Feature extraction results for the input , the feature mapping network adopts The convolutions are aligned and pooled in the spatial dimension to reduce the spatial dimension; for The j-th feature extraction result in , and ;

[0030] Map the pooled feature data to the target feature dimension and flatten the three-dimensional features into two-dimensional features to obtain the flattened image features. ;

[0031] The flattened image features Input a multilayer perceptron with two linear layers for feature mapping, which is expressed as: In the formula Feature extraction result output by the first linear layer The hidden state of is the activation function, and ; is the projection matrix of the first linear layer; is the bias term of the first linear layer; Feature extraction results output by the multi-layer perceptron Image features; is the projection matrix of the second linear layer; is the bias term of the second linear layer;

[0032] The feature extraction results obtained by the image feature extraction network Both are input into the feature mapping network to obtain the corresponding output for ; for The output of the corresponding feature mapping network, for The output of the corresponding feature mapping network;

[0033] Will get Perform stitching to obtain the features of the neonatal fundus image in the text space , and is used as input to the description generation module in the form of a token sequence.

[0034] The Qwen2 model-based description generation module specifically includes the following steps:

[0035] The Qwen2 model is used to construct the description generation module;

[0036] The input text instruction is passed through the embedding layer to get the text embedding ; Embed text and Perform splicing and generate a description of the input image using an autoregressive approach;

[0037] The autoregressive method includes the following steps:

[0038] During the generation process, the previously generated value is recursively used to generate the next value until the generation process ends;

[0039] During the generation process, the current position prediction value is expressed as , all generated values before the current position are represented as , then the generation process is expressed as: in represents the probability of generating a description, and , L is the length of the generated description, To predict the input command before the position, The generated content before the predicted position.

[0040] The training described in step S4 specifically includes the following steps:

[0041] Through efficient parameter fine-tuning training, the likelihood function of the model is maximized; the likelihood function within a batch is expressed as: In the formula For the parameters The probability of ; B is the size of the batch; is the current position prediction value of the b-th sample in the corresponding batch; is the image of the bth sample in the corresponding batch ; is the input instruction before the predicted position of the b-th sample in the corresponding batch, is the generated content before the predicted position of the b-th sample in the corresponding batch;

[0042] In order to improve the generalization ability of the model, the gradient accumulation method is used to average the gradients obtained in K batches; K is the set batch value;

[0043] The training method for efficient parameter fine-tuning specifically includes the following:

[0044] The initial model generated by describing the neonatal fundus image constructed in step S3 is used as the primary model, and the corresponding primary model parameters are expressed as ;

[0045] During the training process, by training a parameter relative to the primary model The parameter set is less than the third set value , to encode the increments of the primary model while keeping the original weights unchanged; add a trainable low-rank matrix factorization to the linear layer of the model; during training, the input data passes through the linear layer, and the gradient update only acts on the low-rank matrix, while keeping the parameters of the initial model unchanged; during the inference phase, the incremental weights obtained from training are merged into the initial model to achieve more efficient parameter updates;

[0046] The update formula of the model weight is expressed as: In the formula are the updated model parameters; For trainable parameters Encoded model weight increments;

[0047] When updating the model weights, the weights of the feature mapping network and the description generation module are updated: for the linear layer, the weight matrix is represented as W, the bias is represented as b, and the increment of the weight matrix is represented as , setting the input of the linear layer to z, then after efficient parameter fine-tuning, the forward propagation of the linear layer is expressed as: Where h is the output of the linear layer; is a scaling factor used to adjust the value of the weight increment; through this efficient parameter fine-tuning mechanism, while ensuring the efficient and adjustable model parameters, it can reduce the computing and storage costs, making the fine-tuning of large-scale models more efficient.

[0048] The present invention also provides an imaging method including the method for generating a description of a neonatal fundus image, further comprising the following steps:

[0049] S6. The neonatal fundus image description content generated in step S5 is marked on the neonatal fundus image and a secondary image is formed to obtain a neonatal fundus image with the fundus image description content.

[0050] The neonatal fundus image description generation method and imaging method provided by the present invention, by combining the technical solutions of feature extraction, feature mapping and description generation modules, design corresponding image feature extraction network and feature mapping network for neonatal fundus images, and adopt a description generation scheme, not only realizes the generation of neonatal fundus image descriptions in a targeted manner, but also has higher reliability and better accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the method flow of the generation method of the present invention.

[0052] Figure 2 Schematic diagram of the process flow of the imaging method of the present invention. DETAILED DESCRIPTION

[0053] like Figure 1 The figure shows a flow chart of the method of the present invention: The method for generating a description of a fundus image of a newborn disclosed in the present invention comprises the following steps:

[0054] S1. Obtain existing fundus images of the newborn;

[0055] In specific implementation, the preferred solution is to obtain multiple types of neonatal fundus images, including normal neonatal fundus images and various types of abnormal neonatal fundus images;

[0056] S2. Describing and preprocessing the neonatal fundus images obtained in step S1 to construct a training dataset; specifically, the steps include:

[0057] The following describes the various types of neonatal fundus images obtained in step S1;

[0058] The described fundus image of the newborn is preprocessed; the preprocessing includes the following steps:

[0059] For each set type, the fundus images of newborns of a type with a number of images less than a first set value are repeated several times; the fundus images of newborns of a type with a number of images greater than a second set value are discarded according to a set ratio;

[0060] The obtained neonatal fundus image is adjusted to a set resolution;

[0061] The fundus image of the newborn after adjusting the resolution is divided into n square images; the fundus image of the newborn after adjusting the resolution is reduced to the same size as the square image after segmentation; finally, an original fundus image of the newborn is preprocessed into n+1 images; a preferred solution is to input Resolution image, resized to resolution, and then cut into square images, and correspondingly reduce the original image to , that is, processing a newborn fundus image into 13 square images.

[0062] S3. Build an initial model for generating descriptions of neonatal fundus images based on an image feature extraction network, a feature mapping network, and a large language model.

[0063] The initial model for generating descriptions of neonatal fundus images includes an image feature extraction network, a feature mapping network, and a description generation module connected in series. The image feature extraction network is used to extract features from input image data. The feature mapping network is used to map the extracted features to corresponding image descriptions. The description generation module is used to generate a description of the input image based on the obtained image features and corresponding text prompt words.

[0064] The specific implementation includes the following steps:

[0065] An image feature extraction network is constructed based on the SigLip model to extract features from the input image data;

[0066] Building a feature mapping network based on the multi-layer perceptron model to map the extracted features to the corresponding image description;

[0067] A description generation module is constructed based on the Qwen2 model to generate a description of the input image based on the obtained image features and the corresponding text prompt words.

[0068] The construction of the image feature extraction network based on the SigLip model specifically includes the following steps:

[0069] The SigLip model is used to construct an image feature extraction network. The network processing process is expressed as: Where Z is the feature extraction result; X is the input image; The parameters are The processing function of the SigLip model;

[0070] An original neonatal fundus image and the corresponding preprocessed n images are collectively represented as ,in is the compressed original neonatal fundus image, is the corresponding preprocessed n images;

[0071] nn+1 images , are input into the image feature extraction network to obtain the feature extraction results for ,in for The feature extraction results of for The feature extraction results.

[0072] The feature mapping network constructed based on the multi-layer perceptron model specifically includes the following steps:

[0073] The input of the feature mapping network is the feature extraction result output by the image feature extraction network ;

[0074] Feature extraction results for the input , the feature mapping network adopts The convolutions are aligned and pooled in the spatial dimension to reduce the spatial dimension; is the j-th feature extraction result in v, and ;

[0075] Map the pooled feature data to the target feature dimension and flatten the three-dimensional features into two-dimensional features to obtain the flattened image features. ;

[0076] The flattened image features Input a multilayer perceptron with two linear layers for feature mapping, which is expressed as: In the formula Feature extraction result output by the first linear layer The hidden state of is the activation function, and ; is the projection matrix of the first linear layer; is the bias term of the first linear layer; Feature extraction results output by the multi-layer perceptron Image features; is the projection matrix of the second linear layer; is the bias term of the second linear layer; The activation function is used to introduce nonlinear transformations in the multi-layer perceptron to enhance the expressive power of the model;

[0077] The feature extraction results obtained by the image feature extraction network Both are input into the feature mapping network to obtain the corresponding output for ; for The output of the corresponding feature mapping network, for The output of the corresponding feature mapping network;

[0078] Will get Perform stitching to obtain the features of the neonatal fundus image in the text space , and is used as input to the description generation module in the form of a token sequence.

[0079] The Qwen2 model-based description generation module specifically includes the following steps:

[0080] The Qwen2 model is used to construct the description generation module;

[0081] The input text instruction is passed through the embedding layer to get the text embedding ; Embed text and Perform splicing and generate a description of the input image using an autoregressive approach;

[0082] The autoregressive method includes the following steps:

[0083] During the generation process, the previously generated value is recursively used to generate the next value until the generation process ends;

[0084] During the generation process, the current position prediction value is expressed as , all generated values before the current position are represented as , then the generation process is expressed as: in represents the probability of generating a description, and , L is the length of the generated description, To predict the input command before the position, The generated content before the predicted position.

[0085] S4 using the training data set obtained in step S2, based on the likelihood function, the neonatal fundus image description generated in step S3 constructed initial model training, to obtain a neonatal fundus image description generation model;

[0086] In specific implementation, the training process includes:

[0087] Through efficient parameter fine-tuning training, the likelihood function of the model is maximized; the likelihood function within a batch is expressed as: In the formula For the parameters B is the batch size, which can be set according to the hardware configuration of the system where the model is deployed; is the current position prediction value of the b-th sample in the corresponding batch; is the image of the bth sample in the corresponding batch ; is the input instruction before the predicted position of the b-th sample in the corresponding batch, is the generated content before the predicted position of the b-th sample in the corresponding batch;

[0088] In order to improve the generalization ability of the model, the gradient accumulation method is used to average the gradients obtained in K batches; K is the set batch value;

[0089] The training method for efficient parameter fine-tuning specifically includes the following:

[0090] The initial model generated by describing the neonatal fundus image constructed in step S3 is used as the primary model, and the corresponding primary model parameters are expressed as ;

[0091] During the training process, by training a parameter relative to the primary model The parameter set is less than the third set value , to encode the increments of the primary model while keeping the original weights unchanged; add a trainable low-rank matrix factorization to the linear layer of the model; during training, the input data passes through the linear layer, and the gradient update only acts on the low-rank matrix, while keeping the parameters of the initial model unchanged; during the inference phase, the incremental weights obtained from training are merged into the initial model to achieve more efficient parameter updates;

[0092] The update formula of the model weight is expressed as: In the formula are the updated model parameters; For trainable parameters Encoded model weight increments;

[0093] When updating the model weights, the weights of the feature mapping network and the description generation module are updated: for the linear layer, the weight matrix is represented as W, the bias is represented as b, and the increment of the weight matrix is represented as , setting the input of the linear layer to z, then after efficient parameter fine-tuning, the forward propagation of the linear layer is expressed as: Where h is the output of the linear layer; is a scaling factor used to adjust the value of the weight increment. This efficient parameter fine-tuning mechanism ensures efficient and adjustable model parameters while reducing computing and storage costs, making fine-tuning of large-scale models more efficient.

[0094] This adjustment method more accurately optimizes the model's ability to map and describe neonatal fundus image features without significantly increasing computational complexity. This, combined with the overall neonatal fundus image description generation method, enhances the model's performance when processing neonatal fundus images. The description generation module also ensures that the generated image descriptions are better adapted to the characteristics of neonatal fundus images, improving the accuracy and reliability of the descriptions.

[0095] S5. Using the neonatal fundus image description generation model obtained in step S4, automatically generate neonatal fundus image descriptions.

[0096] The effect of the generation method of the present invention is described below with reference to an embodiment:

[0097] Experiments were conducted on a dataset of neonatal fundus image text descriptions using the PyTorch 2.1 framework. The model's generation performance was evaluated from the following perspectives: (1) evaluation of descriptive text based on word matching; (2) evaluation of image descriptions; (3) evaluation of disease diagnosis performance based on a binary classification task; and (4) evaluation of disease diagnosis performance based on a multi-classification task. In addition, an ablation experiment was designed in this section to demonstrate the effectiveness of increasing the number of image slices in improving text generation performance. In the text evaluation, the Python Jieba library was used to segment the text and use words as the smallest unit of text evaluation.

[0098] The description text evaluation based on word matching is used to evaluate the mutual inclusion of words in the true value and the predicted value. The selected indicators include Accuracy, Precision, Recall and F1 score.

[0099] For comparison with our method, we selected two open-source models with strong performance in the medical and fundus image fields: LLaVA-Med-v1.5 and MM-Retinal. LLaVA-Med-v1.5 is a multimodal large language model derived by fine-tuning the LLaVA-v1.5 model on a biomedical dataset; MM-Retinal is an image-to-text model pre-trained on a large number of ophthalmological images.

[0100] The description text evaluation results based on phrase matching are shown in Table 1:

[0101] Table 1 Schematic diagram of description text evaluation results based on phrase matching

[0102]

[0103] As can be seen from Table 1, the various indicators of the method of the present invention are far higher than those of the existing method, which shows the effectiveness of the method of the present invention.

[0104] Then, we selected evaluation metrics widely used in image captioning tasks to evaluate the effectiveness of our method. The evaluation metrics include BLEU (Bilingual Evaluation Score), ROUGE (Recall Summary Evaluation Alternative), METEOR (Explicit Ranking Translation Evaluation Metric), and CIDER (Consistent Image Description Evaluation Metric). The comparison results are shown in Table 2.

[0105] Table 2 Schematic diagram of comparison results based on image description evaluation

[0106]

[0107] As can be seen from Table 2, compared with the existing solutions, the method of the present invention has obvious improvements in various indicators.

[0108] In terms of the evaluation of neonatal fundus image detection indicators based on binary classification, the output text is divided into "normal" and "abnormal" and its statistical indicators are calculated, including accuracy, specificity, recall rate and F1 score. The specific results are shown in Table 3:

[0109] Table 3 Schematic diagram of evaluation indicators of neonatal fundus images based on binary classification

[0110]

[0111] Table 3 shows that our method significantly outperforms existing methods in terms of F1 score, recall, and precision, lagging behind only in specificity. This is because specificity, which measures the test's ability to identify normal individuals, is too high, as the output is almost entirely "no significant abnormality." However, our method's specificity is very close, demonstrating its superiority.

[0112] In terms of the multi-classification neonatal fundus image detection index evaluation, simple word matching is used to perform classification and calculate its statistical indicators, including accuracy, specificity, recall rate, and F1 score. The specific results are shown in Table 4:

[0113] Table 4 Schematic diagram of neonatal fundus image evaluation indicators based on multi-classification

[0114]

[0115] It can be seen from Table 4 that the various indicators of the method of the present invention are far superior to other indicators.

[0116] The present invention also proposes a feature block extraction scheme; for this scheme, an ablation experiment is designed to prove that the number of image blocks (corresponding to the number of blocks in S2, that is, the number of nn) is , i.e. the total number of processed images is 13) on the description generation performance; multi-classification indicators are used to measure it; the experimental results are shown in Table 5:

[0117] Table 5 Schematic diagram of ablation experiment results

[0118]

[0119] It can be seen from Table 5 that the method of the present invention is much better than the result obtained by using only the original image or dividing the image into 4 blocks, which shows that improving the image resolution can significantly improve the accuracy of the description generated by the model.

[0120] It can be seen from Tables 1 to 5 that the various indicators of the method of the present invention are significantly higher than those of the existing method, which once again proves the effectiveness of the present invention.

[0121] like Figure 2 The figure shows a flow chart of the imaging method of the present invention: the imaging method disclosed in the present invention, including the method for generating a description of a fundus image of a newborn, comprises the following steps:

[0122] S1. Obtain existing fundus images of the newborn;

[0123] S2 describes and preprocesses the neonatal fundus images obtained in step S1 to construct a training data set;

[0124] S3. Build an initial model for generating descriptions of neonatal fundus images based on an image feature extraction network, a feature mapping network, and a large language model.

[0125] The initial model for generating descriptions of neonatal fundus images includes an image feature extraction network, a feature mapping network, and a description generation module connected in series. The image feature extraction network is used to extract features from input image data. The feature mapping network is used to map the extracted features to corresponding image descriptions. The description generation module is used to generate a description of the input image based on the obtained image features and corresponding text prompt words.

[0126] S4 using the training data set obtained in step S2, based on the likelihood function, the neonatal fundus image description generated in step S3 constructed initial model training, to obtain a neonatal fundus image description generation model;

[0127] S5. The neonatal fundus image description generation model obtained in step S4 is used to automatically generate a description of the neonatal fundus image;

[0128] S6. The neonatal fundus image description content generated in step S5 is marked on the neonatal fundus image and a secondary image is formed to obtain a neonatal fundus image with the fundus image description content.

[0129] The imaging method provided by the present invention can be directly applied to existing neonatal fundus imaging equipment (such as a neonatal fundus imaging system) or directly applied to a terminal (such as a computer); in specific application, the existing scheme is used to obtain the actual neonatal fundus image, and then the obtained data is input into the corresponding machine equipment or terminal. At this time, the machine equipment or terminal can obtain the description content of the actual neonatal fundus image according to the imaging method disclosed by the present invention, and display the description content on the original image through different types of representation (such as color), and then perform secondary imaging and output; at this time, the output image is an image with the fundus image description content, which can reflect the actual neonatal fundus image and the corresponding description content, thereby greatly facilitating the subsequent work of clinical medical staff and laboratory experimenters.

Claims

1. A method for generating descriptions of neonatal fundus images, characterized in that The steps include: S1. Obtain existing fundus images of the newborn; S2 describes and preprocesses the neonatal fundus images obtained in step S1 to construct a training data set; S3. Build an initial model for generating descriptions of neonatal fundus images based on an image feature extraction network, a feature mapping network, and a large language model. The initial model for generating descriptions of neonatal fundus images includes an image feature extraction network, a feature mapping network, and a description generation module connected in series. The image feature extraction network is used to extract features from input image data. The feature mapping network is used to map the extracted features to corresponding image descriptions. The description generation module is used to generate a description of the input image based on the obtained image features and corresponding text prompt words. The specific steps include: An image feature extraction network is constructed based on the SigLip model to extract features from the input image data; Building a feature mapping network based on the multi-layer perceptron model to map the extracted features to the corresponding image description; A description generation module is built based on the Qwen2 model to generate a description of the input image based on the obtained image features and the corresponding text prompt words; S4 using the training data set obtained in step S2, based on the likelihood function, the neonatal fundus image description generated in step S3 constructed initial model training, to obtain a neonatal fundus image description generation model; During training: Through efficient parameter fine-tuning, the likelihood function of the maximum model is maximized; the likelihood function within a batch is expressed as: In the formula For the parameters The probability of ; B is the size of the batch; is the current position prediction value of the b-th sample in the corresponding batch; is the image of the bth sample in the corresponding batch ; is the input instruction before the predicted position of the b-th sample in the corresponding batch, is the generated content before the predicted position of the b-th sample in the corresponding batch; The update formula of the model weight is expressed as: In the formula are the updated model parameters; For trainable parameters Encoded model weight increments; Generate model parameters for the initial model for the neonatal fundus image description constructed in step S3 When updating the model weights, the weights of the feature mapping network and the description generation module are updated: for the linear layer, the weight matrix is represented as W, the bias is represented as b, and the increment of the weight matrix is represented as , assuming the input of the linear layer is z, then after efficient parameter fine-tuning, the forward propagation of the linear layer is expressed as: Where h is the output of the linear layer; is the scaling factor used to adjust the value of the weight increment; S5. Using the neonatal fundus image description generation model obtained in step S4, automatically generate neonatal fundus image descriptions.

2. The method for generating a description of a neonatal fundus image according to claim 1, wherein The step S2 specifically includes the following steps: The following describes the various types of neonatal fundus images obtained in step S1; The described fundus image of the newborn is preprocessed; the preprocessing includes the following steps: For each set type, the fundus images of newborns of a type with a number of images less than a first set value are repeated several times; the fundus images of newborns of a type with a number of images greater than a second set value are discarded according to a set ratio; The obtained neonatal fundus image is adjusted to a set resolution; The neonatal fundus image after adjusting the resolution is divided into n square images; the neonatal fundus image after adjusting the resolution is reduced to the same size as the divided square images; finally, an original neonatal fundus image is preprocessed into n+1 images.

3. The method for generating a description of a neonatal fundus image according to claim 2, wherein The image feature extraction network based on the SigLip model is constructed, which specifically includes the following steps: The SigLip model is used to construct an image feature extraction network. The processing process of the image feature extraction network is expressed as: Where Z is the feature extraction result; X is the input image; The parameters are The processing function of the SigLip model; An original neonatal fundus image and the corresponding preprocessed n images are collectively represented as ,in is the compressed original neonatal fundus image, is the corresponding preprocessed n images; nn+1 images , are input into the image feature extraction network to obtain the feature extraction results for ,in for The feature extraction results of for The feature extraction results.

4. The method for generating a description of a neonatal fundus image according to claim 3, wherein The feature mapping network constructed based on the multi-layer perceptron model specifically includes the following steps: The input of the feature mapping network is the feature extraction result output by the image feature extraction network ; Feature extraction results for the input , the feature mapping network adopts The convolutions are aligned and pooled in the spatial dimension to reduce the spatial dimension; for The j-th feature extraction result in , and ; Map the pooled feature data to the target feature dimension and flatten the three-dimensional features into two-dimensional features to obtain the flattened image features. ; The flattened image features Input a multilayer perceptron with two linear layers for feature mapping, which is expressed as: In the formula Feature extraction result output by the first linear layer The hidden state of is the activation function, and ; is the projection matrix of the first linear layer; is the bias term of the first linear layer; Feature extraction results output by the multi-layer perceptron Image features; is the projection matrix of the second linear layer; is the bias term of the second linear layer; The feature extraction results obtained by the image feature extraction network Both are input into the feature mapping network to obtain the corresponding output for ; for The output of the corresponding feature mapping network, for The output of the corresponding feature mapping network; Will get Perform stitching to obtain the features of the neonatal fundus image in the text space , and is used as input to the description generation module in the form of a token sequence.

5. The method for generating descriptions of neonatal fundus images according to claim 4, wherein The Qwen2 model-based description generation module specifically includes the following steps: The Qwen2 model is used to construct the description generation module; The input text instruction is passed through the embedding layer to get the text embedding ; Embed text and Perform splicing and generate a description of the input image using an autoregressive approach; The autoregressive method includes the following steps: During the generation process, the previously generated value is recursively used to generate the next value until the generation process ends; During the generation process, the current position prediction value is expressed as , all generated values before the current position are represented as , then the generation process is expressed as: in represents the probability of generating a description, and , L is the length of the generated description, To predict the input command before the position, The generated content before the predicted position.

6. The method for generating descriptions of neonatal fundus images according to claim 5, wherein The training described in step S4 specifically includes the following steps: In order to improve the generalization ability of the model, the gradient accumulation method is used to average the gradients obtained in K batches; K is the set batch value; The training method for efficient parameter fine-tuning specifically includes the following: The initial model generated by describing the neonatal fundus image constructed in step S3 is used as the primary model, and the corresponding primary model parameters are expressed as ; During the training process, by training a parameter relative to the primary model The parameter set is less than the third set value , to encode the increments of the primary model while keeping the original weights unchanged; add a trainable low-rank matrix factorization to the linear layer of the model; during training, the input data passes through the linear layer, and the gradient update only acts on the low-rank matrix, while keeping the parameters of the initial model unchanged; During the inference phase, the trained incremental weights are merged into the initial model to achieve more efficient parameter updates.

7. An imaging method comprising the method for generating a description of a neonatal fundus image according to any one of claims 1 to 6, characterized in that The following steps are also included: S6. The neonatal fundus image description content generated in step S5 is marked on the neonatal fundus image and a secondary image is formed to obtain a neonatal fundus image with the fundus image description content.

Images