Ophthalmic image auxiliary information generation system based on multi-agent cooperation

Abstract

The application discloses an ophthalmic image auxiliary information generation system based on multi-agent cooperation and relates to the technical field of medical auxiliary diagnosis.The system comprises a multi-modal preprocessing engine, which receives fundus images and performs image enhancement and noise reduction to generate target images; a diagnosis reasoning server, which classifies the target images and determines a probability difference value according to the classification results; a multi-agent interaction engine, which updates a probability distribution vector when the probability difference value is less than a preset threshold; and a mixed retrieval knowledge base, which retrieves and returns a candidate knowledge fragment set according to the multi-modal retrieval probe.The multi-agent interaction engine also generates an auxiliary diagnosis reply based on the candidate knowledge fragment set.The system has the beneficial effect of improving the rigor and clinical credibility of ophthalmic auxiliary diagnosis results.

Description

Technical Field

[0001] This invention relates to the field of medical auxiliary diagnostic technology, and in particular to an ophthalmic image auxiliary information generation system based on multi-agent collaboration. Background Technology

[0002] Eye diseases seriously endanger visual health, and early screening and diagnosis are of great significance in preventing irreversible blindness. Currently, deep learning-based fundus image-assisted diagnostic solutions typically use convolutional neural networks to classify fundus images, obtain the probability distributions corresponding to candidate disease categories, and then directly present the classification results or probability values ​​to the user. Some solutions further combine general large language models to provide image-text consultation functions to explain the classification results or answer related questions raised by users.

[0003] However, the above-mentioned solutions have the following shortcomings: On the one hand, the imaging quality of fundus images is easily affected by factors such as shooting equipment and lighting conditions. When the image quality is poor or the lesion features are atypical, the model's discrimination probabilities for multiple candidate disease categories are often quite similar. If the discrimination result is directly output according to the conventional process, it is easy to lead to misdiagnosis or missed diagnosis. On the other hand, the classification probabilities output by deep learning models themselves lack interpretability, and general large language models lack the support of professional medical knowledge when generating explanations or consultation responses, which can easily produce content that does not match clinical reality and is difficult to meet the requirements of accuracy and reliability for auxiliary diagnosis. Summary of the Invention

[0004] In view of the above-mentioned prior art, this application is hereby made. Embodiments of this application provide an ophthalmic image-assisted information generation system based on multi-agent collaboration, which can avoid the direct output of conclusions with low confidence or no medical basis, thereby improving the rigor and clinical credibility of ophthalmic auxiliary diagnostic results.

[0005] According to one aspect of this application, an ophthalmic image-assisted information generation system based on multi-agent collaboration is provided, comprising: a multimodal preprocessing engine for receiving fundus images, performing image enhancement and noise reduction processing on the fundus images according to preprocessing parameters, and generating a target image; a diagnostic inference server for classifying the target image, outputting probability distribution vectors corresponding to candidate disease categories, and visual feature vectors corresponding to the probability distribution vectors; and determining a probability difference based on the highest and second-highest classification probabilities in the probability distribution vectors; and a multi-agent interaction engine for performing the following based on the visual feature vectors, the probability distribution vectors, and the probability difference: when the probability difference is less than a preset threshold, generating a preprocessing parameter adjustment instruction and sending it to the multimodal preprocessing engine. The system adjusts the preprocessing parameters and regenerates the target image, driving the diagnostic inference server to classify the regenerated target image and output an updated probability distribution vector. Medical keywords are extracted based on the probability distribution vector or the updated probability distribution vector, converted into text semantic vectors, and the visual feature vector and the text semantic vector are fused to generate a multimodal retrieval probe. A hybrid retrieval knowledge base is used to receive the multimodal retrieval probe, perform joint retrieval based on the multimodal retrieval probe, obtain a candidate knowledge fragment set, and return the candidate knowledge fragment set to the multi-agent interaction engine. The multi-agent interaction engine is also used to generate an auxiliary diagnostic response based on the candidate knowledge fragment set.

[0006] Compared with existing technologies, the ophthalmic image-assisted information generation system based on multi-agent collaboration according to the embodiments of this application can improve the accuracy of ambiguous cases by establishing a feedback loop triggered by probability difference. When the probability difference is less than a preset threshold, a preprocessing parameter adjustment instruction is generated to drive the multimodal preprocessing engine to perform secondary image enhancement and reclassification. Simultaneously, the system generates a multimodal retrieval probe by fusing visual feature vectors and text semantic vectors, driving a hybrid retrieval knowledge base to perform joint retrieval. This ensures that the auxiliary diagnostic response generated by the multi-agent interaction engine is based on a set of candidate knowledge fragments as evidence, avoiding the direct output of conclusions with low confidence or no medical basis, and improving the rigor and clinical credibility of ophthalmic auxiliary diagnostic results. Attached Figure Description

[0007] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0008] Figure 1 This is a block diagram of the ophthalmic image auxiliary information generation system based on multi-agent cooperation according to the present invention.

[0009] Figure 2 This is a timing diagram of the ophthalmic image auxiliary information generation system based on multi-agent cooperation of the present invention. Detailed Implementation

[0010] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0011] Example 1:

[0012] This embodiment describes a specific implementation of an ophthalmic image-assisted information generation system based on multi-agent collaboration.

[0013] Existing ophthalmic auxiliary diagnostic solutions typically rely solely on deep learning models to perform a one-time classification of fundus images and directly present the resulting probability vectors or diagnostic conclusions to the user. However, on the one hand, the imaging quality of fundus images is greatly affected by factors such as the imaging equipment, ambient lighting, and operator technique. When the image quality is poor or the lesion features are atypical, the model is prone to fuzzy discrimination situations where the probabilities of the Top-1 and Top-2 categories are similar, making it difficult to choose. If the discrimination results are directly output according to the conventional process, it is easy to cause misdiagnosis or missed diagnosis. On the other hand, the classification probability vectors output by deep learning models themselves lack interpretability and evidence-based support. Patients and primary care physicians find it difficult to understand their clinical implications and cannot determine whether the conclusions are supported by authoritative medical evidence. Both of these factors together constitute the technical problem of insufficient accuracy and reliability in auxiliary diagnosis.

[0014] The system provided in this embodiment establishes a feedback loop between the diagnostic inference server and the multi-agent interaction engine, triggered by probability differences. When the difference between the Top-1 and Top-2 classification probabilities is small, indicating insufficient model confidence in the current image, the system proactively generates preprocessing parameter adjustment instructions to drive the multi-modal preprocessing engine to perform targeted secondary enhancement and reclassification of the image, rather than directly outputting low-confidence discrimination results. This improves the discrimination accuracy of ambiguous cases with minimal additional computational overhead. After obtaining high-confidence classification results (whether the initial classification reaches the preset threshold or the reclassification occurs after secondary enhancement), the system extracts medical keywords and fuses visual feature vectors to generate a multi-modal retrieval probe. This probe drives a hybrid retrieval knowledge base to jointly retrieve authoritative knowledge fragments related to the current diagnosis. The multi-agent interaction engine can then generate evidence-based auxiliary diagnostic responses that are easy for patients and primary care physicians to understand, thus solving the problem of insufficient accuracy and reliability in conventional diagnostic methods.

[0015] Figure 1 The illustration shows an ophthalmic image auxiliary information generation system based on multi-agent collaboration according to an embodiment of this application, including a multimodal preprocessing engine, a diagnostic reasoning server, a multi-agent interaction engine, and a hybrid retrieval knowledge base.

[0016] like Figure 1 As shown, the multimodal preprocessing engine receives fundus images, performs image enhancement and noise reduction on the fundus images according to preprocessing parameters, and generates the target image.

[0017] Fundus images are two-dimensional color images obtained by capturing the retina, optic disc, macula, and blood vessels of the human eye using devices such as fundus cameras and slit lamps. They are commonly used imaging data for screening and diagnosing ophthalmic diseases. Due to differences in the model of the capturing equipment, ambient lighting conditions, and operator techniques, fundus images from different sources often vary significantly in terms of clarity, color deviation, and noise levels. If these images are directly input into the diagnostic model without processing, the true lesion features may be obscured due to image quality issues, affecting diagnostic accuracy. Therefore, the multimodal preprocessing engine performs image enhancement and noise reduction on fundus images before formal classification, generating target images of higher quality suitable for diagnostic analysis. The preprocessing parameters control the specific methods and intensities of enhancement and noise reduction. Their initial values ​​can be preset default parameters, or they can be dynamically adjusted by the multi-agent interaction engine based on the classification results from the diagnostic inference server. The specific adjustment methods will be detailed below.

[0018] like Figure 1As shown, the diagnostic inference server is used to classify the target image, output the probability distribution vector corresponding to the candidate disease category, and the visual feature vector corresponding to the probability distribution vector; and determine the probability difference based on the highest and second highest classification probabilities in the probability distribution vector.

[0019] Candidate disease categories refer to a variety of ophthalmic disease types that the diagnostic inference server can identify, pre-defined by the system, such as diabetic retinopathy, glaucoma, cataracts, and age-related macular degeneration. The specific number and types of categories can be pre-configured according to the diagnostic needs of the actual application scenario, and this application does not limit this. The probability distribution vector refers to the vector composed of the probability values ​​of the target image belonging to each candidate disease category, output by the diagnostic inference server for the target image, with the sum of all elements in the vector being 1. The visual feature vector refers to the high-dimensional feature representation of the visual content of the target image extracted by the diagnostic inference server during the intermediate process of classifying the target image. This visual feature vector is used not only to generate the probability distribution vector but will also be reused in the multimodal retrieval probe generation process described below. The probability difference refers to the difference between the largest classification probability (denoted as P1, corresponding to the most likely candidate disease category in the model's view) and the second largest classification probability (denoted as P2, corresponding to the second most likely candidate disease category in the model's view) in the probability distribution vector; that is, the probability difference is equal to the difference between P1 and P2. The magnitude of the probability difference directly reflects the degree of discriminative power of the diagnostic inference server in determining the classification result. For example, if the probability distribution vector corresponding to a target image has P1=0.92 and P2=0.05, the probability difference is 0.87, indicating that the model has a clear and unique tendency to judge among the candidate disease categories, with a high degree of discriminative power. However, if the corresponding to another target image has P1=0.55 and P2=0.48, the probability difference is only 0.07, indicating that the model has difficulty choosing between the two candidate disease categories, with a low degree of discriminative power. In this case, directly outputting the category corresponding to P1 as the diagnosis result would pose a high risk of misjudgment.

[0020] like Figure 1 As shown, after the diagnostic inference server completes the classification and determines the probability difference, the multi-agent interaction engine performs dynamic image enhancement decisions based on the visual feature vector, probability distribution vector, and probability difference, taking into account whether the probability difference reaches a preset threshold—a boundary condition for the degree of discriminative control.

[0021] When the probability difference is less than the preset threshold, a preprocessing parameter adjustment instruction is generated and sent to the multimodal preprocessing engine to adjust the preprocessing parameters and regenerate the target image, driving the diagnostic inference server to classify the regenerated target image and output the updated probability distribution vector.

[0022] This scenario corresponds to a situation where the diagnostic inference server has a low degree of confidence in the current target image. In this case, this embodiment does not directly use the low-confidence classification result as the basis for diagnosis, but actively triggers a targeted secondary image enhancement process: the preprocessing parameter adjustment instruction generated by the multi-agent interaction engine is used to instruct the multimodal preprocessing engine to adjust the preprocessing parameters (the specific types of preprocessing parameter adjustment instructions and the multimodal preprocessing engine's response to them will be detailed below), so that the regenerated target image has a clearer or more prominent lesion feature visual appearance compared to the original target image, thereby driving the diagnostic inference server to reclassify based on the higher-quality target image. This feedback closed-loop mechanism is similar to a doctor actively adjusting equipment parameters (such as contrast and magnification) and re-observing an image if they are not confident enough in the lesion features of the image, rather than directly giving a diagnostic conclusion when the information is insufficient. This improves the accuracy of the judgment of ambiguous cases without increasing human intervention.

[0023] After completing the above decision-making and processing, the multi-agent interaction engine extracts medical keywords based on the probability distribution vector (corresponding to the case with high confidence in the initial classification) or the updated probability distribution vector (corresponding to the case after secondary enhancement), converts the medical keywords into text semantic vectors, and fuses the visual feature vectors and text semantic vectors to generate a multimodal retrieval probe.

[0024] This scenario corresponds to a situation where the diagnostic reasoning server already has a high degree of discriminative confidence in the current target image. In this case, there is no need to trigger image reprocessing; instead, the focus shifts to knowledge-level evidence-based verification and content preparation for the diagnostic conclusion. Medical keywords refer to keywords extracted from the names or related medical terms corresponding to the candidate disease categories that rank highly (e.g., first or top-ranked) in the probability distribution vector. For example, when the highest-ranking category in the probability distribution vector is "diabetic retinopathy," the extracted medical keywords could include "diabetic retinopathy," "microaneurysm," and "retinal hemorrhage." Text semantic vectors refer to the vector representations that characterize the semantic content of medical keywords after encoding them using a pre-defined text embedding model. Text embedding models are a conventional technique in this field, and this application does not limit their specific network structure. A multimodal retrieval probe refers to a joint vector representation obtained by fusing visual feature vectors and text semantic vectors in a preset manner (such as vector concatenation, weighted summation, or mapping through a fusion network). This joint vector carries both the visual content information of the target image and the text semantic information of the candidate disease, and is used to drive the joint retrieval process described below. Compared with using only medical keywords for retrieval, it can more comprehensively represent the semantic content of the current diagnostic request and improve the relevance of the retrieved knowledge fragments to the actual cases.

[0025] like Figure 1 As shown, the hybrid retrieval knowledge base is used to receive multimodal retrieval probes, perform joint retrieval based on multimodal retrieval probes, obtain a set of candidate knowledge fragments, and return the set of candidate knowledge fragments to the multi-agent interaction engine.

[0026] The hybrid retrieval knowledge base pre-stores knowledge fragments and their corresponding vector representations obtained by segmenting and vectorizing medical knowledge documents such as ophthalmology treatment guidelines and professional literature. The specific implementation method of joint retrieval will be explained in detail below in conjunction with the sub-steps. The candidate knowledge fragment set refers to the set of several knowledge fragments that are highly semantically relevant to the multimodal retrieval probe and are selected by joint retrieval. Each knowledge fragment can be a continuous text in a knowledge document, such as a paragraph in an ophthalmology treatment guideline about the diagnostic criteria or treatment recommendations for a specific disease.

[0027] Among them, the multi-agent interaction engine is also used to generate auxiliary diagnostic responses based on a set of candidate knowledge fragments.

[0028] Therefore, regardless of whether the diagnostic process involved image re-enhancement and reclassification triggered by insufficient confidence, the multi-agent interaction engine ultimately uses the "candidate knowledge fragment set" retrieved from the hybrid retrieval knowledge base as the factual basis when generating the auxiliary diagnostic response. This serial architecture, where "all paths ultimately converge on knowledge retrieval," avoids the possibility of a large language model directly outputting conclusions in the absence of medical evidence support, ensuring the rigor and high credibility of the auxiliary diagnostic response.

[0029] As mentioned above, when the probability difference is less than the preset threshold, the multi-agent interaction engine generates a preprocessing parameter adjustment instruction and sends it to the multimodal preprocessing engine. The following section will further explain the specific instruction types and the corresponding processing methods of the multimodal preprocessing engine.

[0030] Preprocessing parameter adjustment commands include color space switching commands and Gaussian filter kernel adjustment commands;

[0031] The multimodal preprocessing engine responds to color space switching commands:

[0032] The color space of the fundus image was switched from RGB to Lab, and contrast enhancement was performed on the a channel in the Lab color space.

[0033] The RGB color space is a commonly used color space that represents image colors using the three primary color components of red, green, and blue. The Lab color space, on the other hand, is a color space composed of a luminance component (L, representing the brightness of the image) and two chromaticity components (a and b). The a channel represents the degree of color shift in the red-green direction, and the b channel represents the degree of color shift in the yellow-blue direction. Because lesions such as retinal hemorrhage and microvascular leakage are more reddish in color due to hemoglobin deposition, this reddish information has a more concentrated and independent numerical expression in the a channel of the Lab color space. It is not coupled with luminance information. Compared to direct enhancement in the RGB color space, performing contrast enhancement processing on the a channel alone can better highlight the contrast of reddish lesions such as hemorrhage relative to the background tissue, without simultaneously amplifying luminance noise unrelated to the lesion, thereby improving the accuracy of subsequent classification.

[0034] The multimodal preprocessing engine responds to Gaussian filter kernel adjustment instructions:

[0035] Generating a heatmap of lesion probability based on fundus images;

[0036] The lesion probability heatmap refers to a two-dimensional numerical matrix with the same size as the fundus image or the size after downsampling. The value at each position in the matrix represents the probability that there is a lesion in the image region corresponding to that position. It can be obtained based on the intermediate layer feature response generated by the diagnostic inference server during the classification process, or based on the preliminary segmentation of the fundus image by a pre-trained lightweight coarse segmentation model. This application does not limit the specific generation algorithm of the lesion probability heatmap.

[0037] A Gaussian filter is used to perform noise reduction on the fundus image, and the kernel parameter of the Gaussian filter is adjusted according to the lesion probability heatmap. Specifically, the first kernel parameter is used for the regions of the fundus image with a lesion probability greater than the preset probability in the lesion probability heatmap, and the second kernel parameter is used for other regions. The first kernel parameter is smaller than the second kernel parameter.

[0038] Gaussian filtering is a common filtering method that reduces noise by weighted averaging of the neighborhood of image pixels. Its filtering strength is determined by the kernel parameter; a larger kernel parameter value results in stronger filtering and more significant noise reduction, but also leads to more severe loss of image details. Noise reduction of fundus images faces a contradiction between the need for "noise reduction" and "preservation of lesion details": applying a uniformly strong noise reduction intensity to the entire image can effectively suppress noise, but it also blurs the features of small lesions such as microvessels and microaneurysms, which is detrimental to diagnosis; conversely, applying a uniformly weak noise reduction intensity to the entire image makes it difficult to adequately suppress imaging noise. To resolve this contradiction, this embodiment employs differentiated kernel parameters for different regions of the image based on the lesion probability heatmap: for regions in the lesion probability heatmap with values ​​greater than a preset probability (i.e., a higher likelihood of lesion), a conservative filtering is performed using a smaller first kernel parameter to minimize the loss of detailed features such as microvessels and exudates within these regions; for other regions in the lesion probability heatmap with values ​​not greater than the preset probability (i.e., a lower likelihood of lesion and mostly background tissue), a more substantial second kernel parameter is used for more thorough noise reduction to improve the overall signal-to-noise ratio of the image. The numerical relationship that the first kernel parameter is smaller than the second kernel parameter is precisely the embodiment of the aforementioned differentiated filtering strategy. For example, the first kernel parameter can be a value in the range of 0.6 to 1.0, and the second kernel parameter can be a value in the range of 1.2 to 2.0. The specific values ​​can be adjusted according to the imaging quality and noise level of the fundus image, and this application does not impose any limitations on this.

[0039] The following section, in conjunction with the internal structure of the diagnostic reasoning server, further explains the aforementioned classification process and the generation methods of probability distribution vectors and visual feature vectors.

[0040] The diagnostic reasoning server consists of a backbone network and a custom classification head connected in sequence;

[0041] The backbone network is used to encode features of the target image and output a visual feature vector;

[0042] The custom classification head consists of a first dropout layer, a first fully connected layer, a batch normalization layer, an activation layer, a second dropout layer, and a second fully connected layer. The first fully connected layer is used to reduce the dimensionality of the visual feature vector, and the output of the second fully connected layer is used to generate a probability distribution vector.

[0043] The backbone network refers to the convolutional neural network used to extract features from the input image. It can be implemented using the convolutional neural network architecture commonly used in this field. This application does not limit the specific network structure of the backbone network. The high-dimensional feature vector output by the backbone network after performing layer-by-layer convolution and downsampling processing on the target image is the aforementioned visual feature vector. For example, the dimension of the visual feature vector can be 1792 dimensions. The custom classification head is a lightweight network structure specifically designed for the number of candidate disease categories, based on the visual feature vector output by the backbone network. It maps the high-dimensional visual feature vector to a probability distribution vector with the same dimension as the number of candidate disease categories. Specifically, the dropout layer randomly sets the output of some neurons to zero during training to reduce overfitting of the network to the training data; the batch normalization layer normalizes the output data of the fully connected layer to accelerate network convergence and improve training stability; the activation layer introduces a non-linear transformation into the batch-normalized data to enhance the network's feature representation ability; the first fully connected layer compresses the high-dimensional visual feature vector into lower-dimensional intermediate features, for example, compressing 1792 dimensions to 512 dimensions, enabling the network to learn a more compact and generalized disease feature representation with limited medical training data; the second fully connected layer further maps the intermediate features to a dimension equal to the number of candidate disease categories. For example, if the number of candidate disease categories is 8, the output of the second fully connected layer is an 8-dimensional vector, which, after normalization, yields the aforementioned probability distribution vector.

[0044] As mentioned earlier, the probability difference reflects the degree to which the diagnostic inference server grasps the classification results. However, this value itself is difficult for patients without professional backgrounds and some primary care physicians to directly understand its clinical meaning. Therefore, the multi-agent interaction engine also uses the following when generating auxiliary diagnostic responses:

[0045] According to the preset hierarchical mapping rules, the probability difference is mapped to a quantified diagnostic reliability score, and the diagnostic reliability score is attached to the auxiliary diagnostic response.

[0046] Among them, the hierarchical mapping rule defines the mapping relationship between multiple probability difference intervals and diagnostic reliability score values.

[0047] The tiered mapping rule predefines multiple value ranges for the probability difference, each range corresponding to a quantified diagnostic reliability score. The higher the value within a probability difference range, the higher the mapped diagnostic reliability score. For example, the tiered mapping rule can be set as follows: when the probability difference is not less than 0.5, the mapped diagnostic reliability score is 95, corresponding to "highly reliable"; when the probability difference is between 0.3 and 0.5, the mapped diagnostic reliability score is 85, corresponding to "relatively reliable"; when the probability difference is between 0.2 and 0.3, the mapped diagnostic reliability score is 75, corresponding to "generally reliable"; when the probability difference is between 0.1 and 0.2, the mapped diagnostic reliability score is 65, corresponding to "lowly reliable"; and when the probability difference is less than 0.1, the mapped diagnostic reliability score is 50, corresponding to "low reliability, manual review recommended." The specific range divisions and score values ​​mentioned above are merely illustrative examples, and those skilled in the art can adjust them according to the diagnostic accuracy requirements of actual application scenarios. By transforming abstract probability differences into intuitive quantitative scores and attaching them to the auxiliary diagnostic response, patients and primary care physicians can intuitively judge the credibility of the diagnostic results without understanding the professional meaning of probability distributions. When the diagnostic reliability score is low, they can promptly seek further review from licensed physicians, thereby improving the safety of the auxiliary diagnostic system in practical applications.

[0048] As mentioned above, the hybrid retrieval knowledge base performs joint retrieval based on multimodal retrieval probes and returns a set of candidate knowledge fragments. The following section will further explain the specific implementation of this joint retrieval.

[0049] The hybrid retrieval knowledge base performs joint retrieval based on multimodal retrieval probes to obtain a set of candidate knowledge fragments, including:

[0050] The built-in vector database is used to perform vector matching recall based on multimodal retrieval probes and keyword matching recall based on medical keywords in parallel to obtain dual-path recall results.

[0051] Vector matching recall refers to a retrieval method that calculates the similarity between a multimodal retrieval probe and the vector representations of various knowledge fragments pre-stored in a vector database, and recalls several knowledge fragments based on their similarity scores. This method can recall knowledge fragments that are semantically similar to the multimodal retrieval probe but have different literal expressions. Keyword matching recall, on the other hand, refers to a retrieval method that recalls knowledge fragments based on the literal matching degree between medical keywords and the text content of knowledge fragments. This method can recall knowledge fragments containing precise expressions of specific professional terms, complementing vector matching recall. For example, for the medical keyword "glycoretinopathy," a common abbreviation for "diabetic retinopathy," vector matching recall might recall knowledge fragments discussing "diabetic retinopathy" due to semantic similarity, while keyword matching recall is better at recalling knowledge fragments in the text that explicitly contain the abbreviation "glycoretinopathy." The recall results obtained by executing vector matching recall and keyword matching recall in parallel are called dual-path recall results.

[0052] A reciprocal sorting fusion algorithm is used to perform the first-level fusion sorting of the dual-path recall results;

[0053] The reciprocal ranking fusion algorithm is an algorithm that merges ranking results from different retrieval paths. Its basic principle is to calculate a fusion score for each knowledge fragment in the dual-path recall results based on its ranking position in its respective path. The fusion score is negatively correlated with the ranking position; that is, the higher the ranking, the higher the fusion score. The fusion scores of the same knowledge fragment in the dual-path recall results are added together to obtain the final fusion ranking score for that knowledge fragment. The dual-path recall results are then sorted from highest to lowest fusion ranking score to obtain the first-level fusion ranking result. For example, the fusion score can be calculated as the reciprocal of the sum of the ranking position and a preset smoothing constant; the higher the ranking position, the higher the corresponding fusion score.

[0054] The results after the first-level fusion and sorting are input into a preset cross-coding model for re-sorting, and a specified number of document fragments at the top are obtained as a set of candidate knowledge fragments.

[0055] Cross-coding models refer to models that pair query content (here, the text corresponding to a multimodal retrieval probe or medical keywords) with candidate knowledge fragments and input them into the same encoding network for joint encoding, directly outputting their relevance scores. Compared to vector matching recall, where the query and knowledge fragments are encoded independently before similarity is calculated, cross-coding models can capture finer-grained semantic interactions between query content and knowledge fragments more precisely, resulting in higher accuracy in relevance determination. However, the computational cost is also correspondingly higher. Therefore, this embodiment only applies it to candidate results that have been narrowed down after the first-level fusion and sorting, rather than directly performing cross-coding on all knowledge fragments in the vector database, thus achieving a balance between retrieval accuracy and computational efficiency. After re-sorting by the cross-coding model, a specified number of document fragments from the top rankings are taken as the final set of candidate knowledge fragments returned to the multi-agent interaction engine. For example, the specified number can be 5.

[0056] In addition to the core processing flow of generating auxiliary diagnostic responses based on image classification and knowledge retrieval, as described above, in order to further enhance the system's scalability and enable the system to easily access third-party diagnostic tools in addition to the built-in diagnostic capabilities, the system in this embodiment is also configured with a protocol extension gateway, which is described in detail below.

[0057] The system also includes a protocol extension gateway that communicates with the multi-agent interaction engine;

[0058] The protocol extension gateway is configured to operate based on the model context protocol. It is used to receive normalized configuration information from external input, parse the normalized configuration information to obtain the calling interface of external third-party diagnostic tools, and map and store the calling interface of external third-party diagnostic tools in the database.

[0059] The Model Context Protocol (MGP) is a standardized protocol for communication between artificial intelligence models or agents and external tools and data sources. Through a unified interface specification, external tools can be discovered and invoked by agents in a "plug-and-play" manner, without requiring the development of separate adaptation code for each tool. Standardized configuration information refers to configuration text conforming to the MGP specification, which includes information such as the name of the external third-party diagnostic tool, its calling address, transmission method, and a list of callable functions. This configuration information can be entered by system users through the management interface. After parsing the standardized configuration information, the protocol extension gateway can obtain the calling interface of the external third-party diagnostic tool—that is, the specific function entry point exposed by the external third-party diagnostic tool that can be remotely invoked—and map and store this calling interface in the database for subsequent querying and invocation.

[0060] The multi-agent interaction engine also has a built-in protocol translation layer;

[0061] The multi-agent interaction engine is also used to: bridge calls to external third-party diagnostic tools to the protocol extension gateway via the protocol translation layer, so as to dynamically discover and invoke external third-party diagnostic tools.

[0062] The protocol translation layer is used to convert and bridge the standardized tool call interface within the multi-agent interaction engine and the model context protocol interface followed by the protocol extension gateway. This allows the multi-agent interaction engine to make decisions about calling an external third-party diagnostic tool without needing to be aware of the tool's specific protocol details and interface differences. It only needs to initiate a request according to the unified internal calling method, and the protocol translation layer will automatically convert it into a call request conforming to the model context protocol specification and forward it to the protocol extension gateway for execution. The protocol extension gateway then queries the database for the external third-party diagnostic tool's call interface, completes the actual call, and returns the call result along the original path. Therefore, when the system integrates a new external third-party diagnostic tool, it only needs to configure the corresponding standardized configuration information through the management interface without modifying the core logic of the multi-agent interaction engine, achieving dynamic and low-cost expansion of diagnostic capabilities.

[0063] In addition to the aforementioned image enhancement and noise reduction processing based on probability difference triggering, the multimodal preprocessing engine can also perform independent complexity analysis on the vascular structure in fundus images to provide additional auxiliary discrimination information from the perspective of vascular morphology, as detailed below.

[0064] The multimodal preprocessing engine also includes the following when performing image enhancement and noise reduction:

[0065] The target region in the fundus image is triangulated in three dimensions to reconstruct the vascular tree model, and the fractal dimension of the vascular tree model is calculated.

[0066] If the fractal dimension value is greater than the preset normal threshold, an abnormal location warning will be generated based on the complexity of the microvessels.

[0067] The target region refers to the image area in the fundus image that contains the distribution of retinal blood vessels. Three-dimensional triangulation refers to constructing a three-dimensional mesh structure composed of multiple interconnected triangular facets based on multiple feature points extracted from the blood vessel trajectories within the target region, using commonly used triangulation algorithms. This reconstructs a vascular tree model that can characterize the spatial morphology of blood vessels. The fractal dimension is a quantitative indicator of the complexity of the branches in the vascular tree model. The larger the value, the denser and more tortuous the branches of the vascular tree model, and the higher the complexity of the blood vessel morphology. The blood vessel distribution in a healthy retina usually has a relatively regular and moderately complex branching pattern. However, during the progression of diseases such as diabetic retinopathy, neovascularization may occur in the retina, leading to an increase in the branching complexity of the vascular tree model, and thus a corresponding increase in the fractal dimension. Based on this, when the calculated fractal dimension value is greater than the preset normal threshold, it indicates that the vascular morphological complexity in the target area has exceeded the normal range. At this time, the multimodal preprocessing engine generates an abnormal location warning for microvascular complexity. This warning information can be presented as supplementary content to the auxiliary diagnostic response to indicate the possibility of abnormalities such as microvascular proliferation, providing additional morphological discrimination criteria for the early detection of diseases. For example, the preset normal threshold can be a value of about 1.6. The specific value can be calibrated according to the statistical distribution of fractal dimension values ​​of the vascular tree model of healthy groups, and this application does not limit it.

[0068] The multi-agent interaction engine generates the aforementioned auxiliary diagnostic response based on a pre-built large language model. To ensure that the behavior of the large language model during the generation process conforms to the expected role positioning, operation process and security compliance requirements, this embodiment further designs a hierarchical structure for the prompt words input to the large language model.

[0069] The multi-agent interaction engine is configured to generate auxiliary diagnostic responses based on hierarchical structured prompts; the hierarchical structured prompts include:

[0070] The core instruction layer is used to define the role attributes of the multi-agent interaction engine;

[0071] The tool invocation layer is used to guide the multi-agent interaction engine to perform the operation of generating multimodal retrieval probes;

[0072] The output constraint layer is used to constrain the generation of auxiliary diagnostic responses so that the auxiliary diagnostic responses include preset warning text information.

[0073] Hierarchical structured prompts refer to a method of organizing prompts where not all constraints are mixed into a single paragraph, but rather they are divided into multiple functional layers, written separately, and then combined before being input into the large language model. The core instruction layer defines the role attributes of the multi-agent interaction engine; for example, it can be set to act as a general ophthalmologist responsible for providing auxiliary diagnostic suggestions based on image classification results and knowledge base content. The tool invocation layer guides the multi-agent interaction engine to execute tool invocation-related operations according to a preset workflow. For example, it guides the engine to generate a multimodal retrieval probe and trigger knowledge retrieval when the probability difference reaches a preset threshold, reducing the engine's reliance on the large language model's free decision-making and making the invocation behavior more stable and predictable. The output constraint layer imposes security and compliance restrictions on the generated content. For example, it can constrain the auxiliary diagnostic response from providing a final diagnosis and require a prompt suggesting the user consult a licensed physician, ensuring that the auxiliary diagnostic response necessarily includes preset warning text information and preventing the large language model's generated content from exceeding medical safety boundaries and misleading the user. Through the above layered design, the constraints at each level are relatively independent, making them easy to maintain and adjust separately. This also reduces the risk of mutual interference between constraints at different levels, which could cause the content generated by the large language model to deviate from expectations.

[0074] Considering that in real-world applications, the same fundus image may be submitted to the system multiple times due to user uploads, page refreshes, or administrator reviews, re-executing the entire image preprocessing, classification inference, and response generation process for each submission would result in unnecessary waste of computing power and prolonged response time. Therefore, the system in this embodiment is also equipped with a pre-hash caching module.

[0075] The system is also equipped with a front-end hash cache module;

[0076] The pre-hash cache module is used to: calculate the hash value of the binary data of the fundus image before the multimodal preprocessing engine receives the fundus image, and query the database to see if there is a historical auxiliary diagnosis response that matches the hash value; if the query is successful, the historical auxiliary diagnosis response is returned directly; if the query is unsuccessful, the fundus image is transmitted to the multimodal preprocessing engine.

[0077] A hash value is a fixed-length string with unique identification meaning obtained by processing the binary data of a fundus image using a preset hash algorithm. Theoretically, different image data corresponds to different hash values, while completely identical image data will necessarily correspond to the same hash value. Before the multimodal preprocessing engine receives the fundus image, the pre-hash cache module calculates the hash value of the fundus image to be processed and uses this hash value as an index to query the database to see if a historical auxiliary diagnostic response generated and stored for the same image already exists. If the query is successful, it means that the currently submitted fundus image has been processed before. In this case, there is no need to repeat the subsequent image preprocessing, classification reasoning, and response generation processes. The historical auxiliary diagnostic response can be returned directly, thereby shortening the response time from the usual several seconds for the complete processing process to the hundreds of milliseconds required to query the database. If the query is unsuccessful, it means that the current fundus image is submitted for the first time. In this case, the fundus image is transmitted to the multimodal preprocessing engine according to the normal process, triggering the complete processing process. After the process is completed, the generated auxiliary diagnostic response is associated with the corresponding hash value and stored for possible subsequent repeated queries.

[0078] Since the multi-agent interaction engine typically requires several seconds or even longer to generate auxiliary diagnostic responses based on a large language model, if the conventional approach is to wait until all the response content is generated before returning it to the user all at once, the user will lack progress feedback and have a poor interactive experience while waiting. Therefore, this embodiment further designs the auxiliary diagnostic response generation process as a streaming output method.

[0079] The multi-agent interaction engine is configured as follows when generating auxiliary diagnostic responses:

[0080] Establish a streaming communication channel for the server to send events;

[0081] Acquire streaming output data during the interaction process, parse reply message fragments from the streaming output data, and transmit them incrementally through the streaming communication pipeline;

[0082] The incremental transmission of response message fragments is accumulated to generate auxiliary diagnostic responses, and the auxiliary diagnostic responses are persisted to the database when the end of the streaming communication pipeline transmission is detected.

[0083] Server-sending-events is a standardized communication technology based on the HTTP protocol that supports continuous data push from the server to the client. After the two parties establish a connection, the server can send incremental data to the client multiple times on the same connection without the client repeatedly initiating new requests. It is suitable for application scenarios in this embodiment where auxiliary diagnostic responses need to be generated and displayed simultaneously. In the process of generating auxiliary diagnostic responses, the multi-agent interaction engine continuously generates streaming output data in a segment-by-segment or word-by-word manner. The multi-agent interaction engine parses displayable response message fragments from the streaming output data in real time and transmits each response message fragment incrementally to the user terminal through the established streaming communication channel. This allows the user to see the generation process of the auxiliary diagnostic response in real time without waiting for all content to be generated. At the same time, the multi-agent interaction engine accumulates and splices the incrementally transmitted response message fragments on the server side to reconstruct the complete auxiliary diagnostic response content. When the data transmission in the streaming communication channel is detected to be completed, that is, when the large language model has completed the generation of the entire response, the multi-agent interaction engine persistently stores the accumulated complete auxiliary diagnostic response in the database for subsequent queries, verification, or hash matching returns of the aforementioned historical auxiliary diagnostic responses.

[0084] This concludes the complete description of the components and working principles of the ophthalmic image-assisted information generation system based on multi-agent collaboration provided in this embodiment.

[0085] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0086] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An ophthalmic image-assisted information generation system based on multi-agent collaboration, characterized in that, include: A multimodal preprocessing engine is used to receive fundus images, perform image enhancement and noise reduction processing on the fundus images according to preprocessing parameters, and generate target images; The diagnostic reasoning server is used to classify the target image, output the probability distribution vector corresponding to the candidate disease category, and the visual feature vector corresponding to the probability distribution vector; and determine the probability difference based on the highest and second highest classification probabilities in the probability distribution vector. A multi-agent interaction engine is used to execute the following based on the visual feature vector, the probability distribution vector, and the probability difference: When the probability difference is less than a preset threshold, a preprocessing parameter adjustment instruction is generated and sent to the multimodal preprocessing engine to adjust the preprocessing parameters and regenerate the target image. The diagnostic reasoning server is driven to classify the regenerated target image and output an updated probability distribution vector. Medical keywords are extracted based on the candidate disease category ranked first in the updated probability distribution vector, and a multimodal retrieval probe is generated based on the medical keywords. When the probability difference is not less than a preset threshold, medical keywords are extracted based on the candidate disease category ranked first in the probability distribution vector, and a multimodal retrieval probe is generated based on the medical keywords. The process of generating a multimodal retrieval probe includes: converting the medical keywords into text semantic vectors, and fusing the visual feature vectors and the text semantic vectors to generate a multimodal retrieval probe; A hybrid retrieval knowledge base is used to receive the multimodal retrieval probe, perform joint retrieval based on the multimodal retrieval probe, obtain a set of candidate knowledge fragments, and return the set of candidate knowledge fragments to the multi-agent interaction engine; The multi-agent interaction engine is also used to generate auxiliary diagnostic responses based on the candidate knowledge fragment set.

2. The ophthalmic image-assisted information generation system based on multi-agent collaboration according to claim 1, characterized in that: The preprocessing parameter adjustment instructions include color space switching instructions and Gaussian filter kernel adjustment instructions; The multimodal preprocessing engine executes in response to the color space switching command: The color space of the fundus image is switched from RGB color space to Lab color space, and contrast enhancement processing is performed on the a channel in the Lab color space; The multimodal preprocessing engine responds to the Gaussian filter kernel adjustment command and executes accordingly. A heatmap of lesion probability is generated based on the fundus image; The noise reduction process is performed on the fundus image using a Gaussian filter, and the kernel parameter of the Gaussian filter is adjusted according to the lesion probability heatmap; wherein, a first kernel parameter is used for fundus image regions in the lesion probability heatmap that have a probability greater than a preset probability, and a second kernel parameter is used for fundus image regions in the lesion probability heatmap that have a probability less than the preset probability, and the first kernel parameter is less than the second kernel parameter.

3. The ophthalmic image-assisted information generation system based on multi-agent cooperation according to claim 1, characterized in that: The diagnostic reasoning server includes a backbone network and a custom classification head connected in sequence; The backbone network is used to encode features of the target image and output the visual feature vector; The custom classification head sequentially includes a first discard layer, a first fully connected layer, a batch normalization layer, an activation layer, a second discard layer, and a second fully connected layer; the first fully connected layer is used to reduce the dimensionality of the visual feature vector, and the output of the second fully connected layer is used to generate the probability distribution vector.

4. The ophthalmic image-assisted information generation system based on multi-agent collaboration according to claim 1, characterized in that, When generating the auxiliary diagnostic response, the multi-agent interaction engine is also used for: According to the preset hierarchical mapping rules, the probability difference is mapped to a quantified diagnostic reliability score, and the diagnostic reliability score is attached to the auxiliary diagnostic response. The hierarchical mapping rule defines the mapping relationship between multiple probability difference intervals and diagnostic reliability score values. The higher the value corresponding to the probability difference interval, the higher the diagnostic reliability score obtained by mapping.

5. The ophthalmic image-assisted information generation system based on multi-agent collaboration according to claim 1, characterized in that, The hybrid retrieval knowledge base performs joint retrieval based on the multimodal retrieval probe to obtain a set of candidate knowledge fragments, including: The built-in vector database is used to perform vector matching recall based on the multimodal retrieval probe and keyword matching recall based on the medical keywords in parallel to obtain dual-path recall results. The first-level fusion sorting of the dual-path recall results is performed using a reciprocal sorting fusion algorithm. The results after the first-level fusion and sorting are input into a preset cross-coding model for re-sorting, and a specified number of document fragments at the top ranking are obtained as the candidate knowledge fragment set.

6. The ophthalmic image-assisted information generation system based on multi-agent cooperation according to claim 1, characterized in that: The system also includes a protocol extension gateway that is communicatively connected to the multi-agent interaction engine. The protocol extension gateway is configured to operate based on the model context protocol, and is used to receive normalized configuration information from external input, parse the normalized configuration information to obtain the calling interface of external third-party diagnostic tools, and map and store the calling interface of external third-party diagnostic tools in the database; The multi-agent interaction engine also has a built-in protocol translation layer; The multi-agent interaction engine is also used to: bridge the invocation operation for the external third-party diagnostic tool to the protocol extension gateway through the protocol translation layer, so as to dynamically discover and invoke the external third-party diagnostic tool.

7. The ophthalmic image-assisted information generation system based on multi-agent collaboration according to claim 1, characterized in that, The multimodal preprocessing engine, when performing the image enhancement and noise reduction processes, further includes: The target region in the fundus image is triangulated in three dimensions to reconstruct the vascular tree model, and the fractal dimension of the vascular tree model is calculated. If the fractal dimension value is greater than a preset normal threshold, an abnormal location warning for microvascular complexity is generated.

8. The ophthalmic image-assisted information generation system based on multi-agent collaboration according to claim 1, characterized in that, The multi-agent interaction engine is configured to generate the auxiliary diagnostic response based on hierarchical structured prompts; The hierarchical structured prompts include: The core instruction layer is used to define the role attributes of the multi-agent interaction engine. The tool invocation layer is used to guide the multi-agent interaction engine to perform the operation of generating the multimodal retrieval probe; An output constraint layer is used to constrain the generation of the content of the auxiliary diagnostic response, so that the auxiliary diagnostic response includes preset warning text information.

9. The ophthalmic image-assisted information generation system based on multi-agent collaboration according to claim 1, characterized in that: The system is also equipped with a front-end hash cache module; The pre-hash cache module is used to: calculate the hash value of the binary data of the fundus image before the multimodal preprocessing engine receives the fundus image, and query the database to see if there is a historical auxiliary diagnostic response that matches the hash value; If the query matches, the historical auxiliary diagnosis response will be returned directly. If the query fails to find a match, the fundus image is transmitted to the multimodal preprocessing engine.

10. The ophthalmic image-assisted information generation system based on multi-agent cooperation according to claim 1, characterized in that, The multi-agent interaction engine is configured as follows when generating the auxiliary diagnostic response: Establish a streaming communication channel for the server to send events; Acquire streaming output data during the interaction process, parse reply message fragments from the streaming output data, and transmit them incrementally through the streaming communication channel; The incrementally transmitted response message fragments are accumulated to generate the auxiliary diagnostic response, and the auxiliary diagnostic response is persisted to the database when the end of the streaming communication pipeline transmission is detected.

Images