Electronic clinical medical assessment method and system for ophthalmology diagnosis and treatment scheme

Abstract

The invention provides an electronic clinical medical assessment method and system for an ophthalmology diagnosis and treatment scheme, and relates to the technical field of medical information, and the method comprises the steps: 1, carrying out the multi-scale edge detection calculation of an input ophthalmology image-text mixed medical record image, and obtaining a medical record image; the method comprises the following steps: extracting a boundary contour of a text labeling area and a hand-drawn lesion schematic diagram through adaptive threshold segmentation and morphological closed operation, and establishing a coordinate mapping table containing text block circumscribed rectangular coordinates, geometric positions of key anatomical mark points of the schematic diagram and symbol spacing characteristics; 2, constructing a two-channel attention gating network based on the coordinate mapping table, fusing the semantic features of the text region and the morphological features of the schematic diagram through a dynamic weight distribution strategy, and generating a multi-modal fusion feature matrix with a spatial alignment relationship; according to the method, through multi-modal feature fusion, triangular verification region confidence regulation and control and three-dimensional topology modeling, analysis and structured output of ophthalmology image-text medical records are realized, and the spatial alignment of diagnosis and treatment information is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical information technology, and in particular to an electronic clinical medical evaluation method and system for ophthalmic diagnosis and treatment plans. Background Art

[0002] During ophthalmology clinical diagnosis and treatment, doctors' medical records are presented in a mixed format of text and images. Text annotations are used to describe the condition and diagnosis, while hand-drawn lesion diagrams visually demonstrate the location and morphology of eye lesions. Existing electronic clinical medical evaluation methods are unable to efficiently process such mixed text and image medical records.

[0003] For example, when diagnosing diabetic retinopathy, the text records in the medical record may contain information such as the degree of retinal vascular lesions and the number of microaneurysms, while the hand-drawn schematics mark the specific location of the lesion area in the fundus. However, when traditional methods digitize this information, they cannot accurately extract the key information in the text and schematics, nor can they establish an accurate association between the two. As a result, the generated electronic medical records have problems such as missing information and semantic understanding deviations, which affect subsequent disease assessment and treatment decisions. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an electronic clinical medical evaluation method and system for ophthalmic diagnosis and treatment plans, which can accurately extract the boundary contours of the text-annotated area and the hand-drawn lesion schematic diagram in the mixed ophthalmic medical record.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] In a first aspect, a method for electronic clinical medical evaluation of an ophthalmic diagnosis and treatment plan is provided, the method comprising:

[0007] Step 1: Perform multi-scale edge detection on the input ophthalmic medical record image, extract the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operation, and establish a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics;

[0008] Step 2: Based on the coordinate mapping table, a dual-channel attention gating network is constructed. The semantic features of the text area and the morphological features of the schematic diagram are fused through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship.

[0009] Step 3: Set three text detection reference points during the bidirectional LSTM decoding phase. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection between symbols and schematics, and the third reference point identifies cross-modal semantic conflict areas. A triangulation region is constructed based on the two-dimensional coordinates of the three reference points. The confidence adjustment parameters of the feature map are generated by calculating the area and orientation angle of the triangulation region.

[0010] Step 4: Adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the corrected spatial dependency weights;

[0011] Step 5: Construct a medical entity topology map based on the corrected spatial dependency weights, generate semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the schematic diagram, perform structural reorganization on the verified entity nodes, and output a three-dimensional electronic medical record report.

[0012] Furthermore, multi-scale edge detection calculations are performed on the input ophthalmic medical record image with mixed text and images. The boundary contours of the text annotation area and the hand-drawn lesion diagram are extracted through adaptive threshold segmentation and morphological closing operations. A coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics is established, including:

[0013] Multi-scale edge detection is performed on the input ophthalmological mixed medical record image. The image is convolutionally filtered layer by layer using a Gaussian kernel group with an exponentially increasing variance parameter. The filtering results of adjacent layers are differentially calculated to generate a multi-resolution edge intensity distribution matrix.

[0014] Based on the edge intensity distribution matrix, the pixel gradient amplitude at each resolution level is statistically analyzed by histogram, and the narrow peak distribution corresponding to the text area and the broad peak distribution corresponding to the schematic area in the histogram are identified to obtain the histogram statistical results.

[0015] Based on the histogram statistics, the narrow peak distribution corresponding to the text area and the broad peak distribution corresponding to the schematic area are identified, and the maximum inter-class variance algorithm is used to calculate the initial segmentation threshold;

[0016] The initial segmentation threshold is dynamically modified according to the distribution area ratio of the narrow peak and the broad peak to obtain the modified threshold;

[0017] Apply the modified threshold to the edge intensity distribution matrix for binary segmentation to generate a preliminary binary segmentation result;

[0018] Perform boundary verification on the preliminary binary segmentation result, calculate the gradient direction cosine similarity of adjacent pixels on the segmentation boundary, determine the segmentation error area, and trigger the threshold secondary correction to generate the optimized binary segmentation mask;

[0019] Based on the optimized binary segmentation mask, an elliptical structural element is dynamically generated according to the pixel area distribution of the connected domain of the text area, in which the long axis is aligned with the main direction of the text strokes and the short axis length is the width of the connected domain. At the same time, an elliptical structural element is dynamically generated according to the local curvature radius of the schematic boundary. The direction of the long axis is consistent with the direction of the boundary tangent and the length of the short axis is scaled according to the curvature radius. The schematic contour fragments are fused to generate a smoothly connected area partition result.

[0020] For smoothly connected areas, a rotation-sensitive minimum enclosing rectangle algorithm was used to extract the center coordinates, rotation angle, aspect ratio, and area ratio of the text blocks. The center coordinates of the retinal fovea and the extreme points of curvature of the optic disc margin were located based on the fused schematic outline. A topological network of pathological symbols was constructed using Delaunay triangulation, and the shortest Euclidean distance, orientation angle, and density distribution parameters between symbols were calculated.

[0021] Based on the spatial relationship between the center coordinates of the text block and the anatomical landmarks on the schematic diagram, the 6-degree-of-freedom parameters of the affine transformation matrix are dynamically calculated according to the polar coordinate offset between the text block and the anatomical landmarks to generate the spatial registration parameters for cross-modal alignment;

[0022] Based on the 6-degree-of-freedom parameters of the affine transformation matrix and the signed density gradient field, the spatial correlation strength coefficients between the text blocks and anatomical landmarks were calculated, and a multidimensional mapping table containing polar radius-azimuth parameters, topological connection strength, and spatial entropy values ​​was generated.

[0023] Furthermore, based on the spatial relationship between the center coordinates of the text block and the anatomical landmarks on the schematic diagram, and according to the polar coordinate offsets between the text block and the anatomical landmarks, the 6-DOF parameters of the affine transformation matrix are dynamically calculated to generate the spatial registration parameters for cross-modal alignment, including:

[0024] Based on the center coordinates of the text block and the polar radius-azimuth parameters of the anatomical landmarks on the schematic diagram, the polar radius deviation and azimuth deviation between the text block and the corresponding anatomical landmarks are calculated;

[0025] Based on the polar deviation, the translation parameters of the affine transformation matrix are dynamically calculated, where the horizontal translation is determined by the radial component of the polar deviation, and the vertical translation is adjusted by the sine component of the azimuth deviation. Based on the azimuth deviation, the rotation parameters are dynamically calculated. When the azimuth deviation exceeds 1°, the rotation angle is compensated by 0.5° for every 1° deviation. The scaling parameters are dynamically adjusted according to the symbol density distribution parameters.

[0026] The translation, rotation and scaling parameters are combined to generate a 6-DOF affine transformation matrix and output the spatial registration parameters.

[0027] Furthermore, a dual-channel attention gating network is constructed based on the coordinate mapping table. The semantic features of the text area and the morphological features of the schematic diagram are fused through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship, including:

[0028] Based on the multidimensional mapping table, a dual-channel parallel network is constructed. The text channel extracts text semantic feature vectors through a convolutional neural network, and the schematic channel models the topological relationship of anatomical landmarks through a graph neural network.

[0029] Based on the polar radius-azimuth parameters in the multidimensional mapping table, the polar radius deviation and azimuth deviation between the center coordinates of the text block and the corresponding anatomical landmark are calculated to generate a spatial alignment error matrix.

[0030] The spatial alignment error matrix is ​​fed into the gated attention mechanism to generate a dynamic weight mask through a two-layer fully connected network.

[0031] Fuse the dynamic weight mask with the text semantic feature vector channel by channel to generate a weighted text semantic feature vector;

[0032] Based on the 6-degree-of-freedom parameters of the affine transformation matrix, the weighted text semantic feature vector is projected onto the schematic coordinate system to obtain the projected text semantic feature;

[0033] The projected text semantic features and schematic morphological features are adaptively superimposed pixel by pixel to generate a multimodal fusion feature matrix.

[0034] Furthermore, three text detection reference points are set in the bidirectional LSTM decoding stage. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection relationship between symbols and schematics, and the third reference point identifies the cross-modal semantic conflict area. A triangulation verification area is constructed based on the two-dimensional coordinates of the three reference points. The feature map confidence adjustment parameters are generated by calculating the area and direction angle of the triangulation verification area, including:

[0035] Based on the multimodal fusion feature matrix, three text detection reference points with the greatest semantic relevance are selected during the bidirectional LSTM decoding process. The first reference point locates the starting boundary coordinates of high-frequency terms, the second reference point captures the coordinate intersection of the topological connection between the symbol and the schematic diagram, and the third reference point identifies the coordinate extreme point of the cross-modal semantic conflict area.

[0036] A triangular verification region is constructed based on the two-dimensional coordinates of the three reference points. The area parameters of the triangular region are calculated by vector cross product, and the main direction angle is solved by combining the inverse tangent function. The area and direction angle are input into a multi-layer perceptron to generate normalized confidence adjustment parameters.

[0037] The normalized confidence adjustment parameter is subjected to a Hadamard product operation with the multimodal fusion feature matrix to output a corrected context feature sequence.

[0038] Furthermore, the parameters are adjusted according to the confidence level, and the geometric characteristics of the triangulated verification area are used to constrain the calculation process of the self-attention score matrix to obtain the modified spatial dependency weights, including:

[0039] Based on the modified context feature sequence, the spatial dependency in the self-attention score matrix is ​​extracted, and the score matrix is ​​weighted and modified by the confidence adjustment parameter. The area parameter of the triangular verification region is used to adjust the local attention range, and the direction angle parameter is used to constrain the attention direction, so as to obtain the weighted and modified self-attention score matrix.

[0040] The weighted modified self-attention score matrix is ​​multiplied with the multimodal fusion feature matrix to generate the modified spatial dependency weights.

[0041] Furthermore, a medical entity topology map is constructed based on the corrected spatial dependency weights. By traversing the mapping relationship between anatomical landmarks and text annotations in the diagram, semantic nodes with orientation attributes are generated. After verification, structural reorganization is performed on the entity nodes to output a three-dimensional electronic medical record report, including:

[0042] Based on the modified spatial dependency weights, the polar radius-azimuth parameters of anatomical landmarks and text blocks in the multidimensional mapping table are traversed to generate semantic nodes with orientation attributes. The three-dimensional coordinates of the nodes are converted from the polar radius parameters to the spatial radius, and the azimuth parameters are converted to spherical coordinates. The node weight is determined by the weighted average of the spatial association strength coefficient and the topological connection strength.

[0043] The node distribution density is controlled based on the area parameter of the triangulation verification region, and the semantic association strength in the multimodal feature data is combined to construct the three-dimensional topological structure of the clinical entity.

[0044] The three-dimensional topological structure of clinical entities is optimized based on the spatial entropy value in the multidimensional mapping table, and a structured electronic medical record report containing the three-dimensional spherical coordinates of anatomical landmarks, spatial density distribution of pathological symbols, and semantic association strength is output.

[0045] Second, an electronic clinical medical evaluation system for ophthalmic diagnosis and treatment plans, including:

[0046] The feature extraction module is used to perform multi-scale edge detection calculations on the input ophthalmic medical record image with mixed text and images. It extracts the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operations, and establishes a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics;

[0047] The feature fusion module is used to build a dual-channel attention gating network based on the coordinate mapping table. It fuses the semantic features of the text area with the morphological features of the schematic diagram through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship.

[0048] The reference point construction module is used to set three text detection reference points in the bidirectional LSTM decoding stage. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection relationship between symbols and schematics, and the third reference point identifies the cross-modal semantic conflict area. The triangulation verification area is constructed based on the two-dimensional coordinates of the three reference points. The feature map confidence adjustment parameters are generated by calculating the area and direction angle of the triangulation verification area.

[0049] The weight correction module is used to adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the corrected spatial dependency weights;

[0050] The electronic medical record generation module is used to construct a medical entity topology map based on the corrected spatial dependency weights, generate semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the schematic diagram, perform structured reorganization on the verified entity nodes, and output a three-dimensional electronic medical record report.

[0051] According to a third aspect, a computing device includes:

[0052] one or more processors;

[0053] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0054] In a fourth aspect, a computer-readable storage medium stores a program, which implements the method when executed by a processor.

[0055] The above solution of the present invention includes at least the following beneficial effects:

[0056] Through multi-scale edge detection and adaptive threshold segmentation, the text annotations and hand-drawn schematics in ophthalmic medical records are effectively separated, and the coordinate mapping table is extracted to achieve accurate modeling of the spatial position of the image and text, solving the problems of blurred image and text boundaries and large positioning errors in traditional methods. The dual-channel attention gated network dynamically fuses the semantic features of the text with the morphological features of the schematic diagram. Using the spatial alignment error matrix and dynamic weight mask, it forces the image and text features to accurately correspond under the anatomical coordinates (such as the spatial matching of the text "macular edema" and the lesion area in the schematic diagram), improves the spatial consistency of feature expression, and avoids cross-modal information misalignment. By setting three text detection reference points to construct a triangulated verification area, the area and directional angle parameters are used to quantify the spatial range and directionality of the semantic association, constraining the self-attention mechanism to focus on high-confidence areas (such as the lesion core area) and suppressing low-confidence interference (such as the image and text conflict area), thereby improving the reliability and interpretability of feature decoding.

[0057] The graphic and text information is converted into three-dimensional semantic nodes with orientation attributes, and a topological network centered on anatomical landmarks is constructed to intuitively display the location of the lesion, the density of pathological symbols, and the strength of semantic associations (such as the "microaneurysm-macular edema" connection weight). The three-dimensional electronic medical record report supports clinicians in quickly understanding the spatial distribution and development context of complex diseases, and assists in remote consultations, surgical planning, and efficacy evaluation. The automated processing of the entire process reduces the cost of manual annotation, and the structured reorganization outputs three-dimensional reports that meet clinical standards, providing standardized data input for AI-assisted diagnosis. For example, by optimizing the topological structure through spatial entropy values, high-risk lesion-dense areas can be automatically identified, and the progression of lesions can be predicted in combination with the strength of semantic association, promoting the development of intelligent and precise ophthalmic diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of an electronic clinical medical evaluation method for an ophthalmic diagnosis and treatment plan provided by an embodiment of the present invention.

[0059] Figure 2 Schematic diagram of an electronic clinical medical evaluation system for ophthalmic diagnosis and treatment plans provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0061] like Figure 1 As shown, an embodiment of the present invention provides an electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans, the method comprising the following steps:

[0062] Step 1: Perform multi-scale edge detection on the input ophthalmic medical record image, extract the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operation, and establish a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics;

[0063] Step 2: Based on the coordinate mapping table, a dual-channel attention gating network is constructed. The semantic features of the text area and the morphological features of the schematic diagram are fused through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship.

[0064] Step 3: Set three text detection reference points during the bidirectional LSTM decoding phase. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection between symbols and schematics, and the third reference point identifies cross-modal semantic conflict areas. A triangulation region is constructed based on the two-dimensional coordinates of the three reference points. The confidence adjustment parameters of the feature map are generated by calculating the area and orientation angle of the triangulation region.

[0065] Step 4: Adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the corrected spatial dependency weights;

[0066] Step 5: Construct a medical entity topology map based on the corrected spatial dependency weights, generate semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the schematic diagram, perform structural reorganization on the verified entity nodes, and output a three-dimensional electronic medical record report.

[0067] In an embodiment of the present invention, multi-scale edge detection and adaptive threshold segmentation are used to extract the boundary contours of text and schematics. The resulting coordinate mapping table provides a spatial alignment foundation for subsequent cross-modal analysis, improving the accuracy of information extraction from mixed medical records containing text and images. A dual-channel attention-gated network dynamically fuses the semantics of text and the morphological features of schematics. The resulting multimodal fusion feature matrix effectively preserves the spatial correlation between text and images, providing a more comprehensive feature representation for semantic understanding of complex ophthalmic medical records. The triangulated verification region innovatively introduces geometric property constraints, locating key semantic regions through reference points and generating confidence parameters, effectively resolving cross-modal semantic conflicts and improving the reliability of feature representation. The geometric properties of the triangulated verification region are used to modify self-attention weights, enhancing the model's perception of spatial dependencies and enabling the generated electronic medical records to more accurately reflect the spatial relationships of lesion areas. The constructed three-dimensional medical entity topology map transforms abstract semantics into an intuitive spatial structure. Through orientation-attributed semantic nodes and structured reorganization, the output three-dimensional electronic medical record report is more consistent with clinical thinking, improving the efficiency and accuracy of diagnosis and treatment decisions.

[0068] In a preferred embodiment of the present invention, the above step 1, performing multi-scale edge detection calculations on the input ophthalmic text-and-graphic mixed medical record image, extracting the boundary contours of the text-annotated area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operations, and establishing a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of key anatomical landmarks on the diagram, and the symbol spacing characteristics, may include:

[0069] Step 100: Perform multi-scale edge detection on the input ophthalmological mixed medical record image, use a Gaussian kernel group with exponentially increasing variance parameters to perform convolution filtering on the image layer by layer, and perform differential operation on the filtering results of adjacent layers to generate a multi-resolution edge intensity distribution matrix;

[0070] Step 101: Based on the edge intensity distribution matrix, perform histogram statistics on the pixel gradient amplitude at each resolution level, identify the narrow peak distribution corresponding to the text area and the broad peak distribution corresponding to the schematic area in the histogram, and obtain a histogram statistical result;

[0071] Step 102: Based on the histogram statistics, identify the narrow peak distribution corresponding to the text area and the broad peak distribution corresponding to the schematic area, and calculate the initial segmentation threshold using the maximum inter-class variance algorithm;

[0072] Step 103, dynamically correcting the initial segmentation threshold according to the distribution area ratio of the narrow peak to the broad peak to obtain a corrected threshold;

[0073] Step 104, applying the corrected threshold to the edge intensity distribution matrix to perform binary segmentation and generate a preliminary binary segmentation result;

[0074] Step 105 , performing boundary verification on the preliminary binary segmentation result, calculating the gradient direction cosine similarity of adjacent pixels on the segmentation boundary, determining the segmentation error area, and triggering the threshold secondary correction to generate an optimized binary segmentation mask;

[0075] Step 106: Based on the optimized binary segmentation mask, dynamically generate an elliptical structure element according to the pixel area distribution of the connected domain of the text area, wherein the long axis direction is aligned with the main direction of the text stroke and the short axis length is the width of the connected domain. At the same time, an elliptical structural element is dynamically generated according to the local curvature radius of the schematic boundary. The direction of the long axis is consistent with the direction of the boundary tangent and the length of the short axis is scaled according to the curvature radius. The schematic contour fragments are fused to generate a smoothly connected area partition result.

[0076] Step 107: For the smoothly connected areas, a rotation-sensitive minimum enclosing rectangle algorithm is used to extract the center coordinates, rotation angle, aspect ratio, and area ratio of the text block. Simultaneously, the center coordinates of the retinal fovea and the extreme points of curvature of the optic disc edge are located based on the fused schematic outline. A topological network of pathological symbols is constructed using Delaunay triangulation, and the shortest Euclidean distance, orientation angle, and density distribution parameters between symbols are calculated.

[0077] Step 108 , based on the spatial relationship between the center coordinates of the text block and the anatomical landmarks on the schematic diagram, and according to the polar coordinate offsets between the text block and the anatomical landmarks, dynamically calculate the 6-DOF parameters of the affine transformation matrix to generate spatial registration parameters for cross-modal alignment, specifically including:

[0078] Step 1080 , calculating the polar deviation and azimuth deviation between the text block and the corresponding anatomical landmark based on the center coordinates of the text block and the polar radius-azimuth parameters of the anatomical landmark in the schematic diagram;

[0079] Step 1081: Dynamically calculate the translation parameters of the affine transformation matrix based on the polar deviation, where the horizontal translation is determined by the radial component of the polar deviation, and the vertical translation is adjusted by the sine component of the azimuth deviation. Dynamically calculate the rotation parameters based on the azimuth deviation. When the azimuth deviation exceeds 1°, the rotation angle is compensated by 0.5° for every 1° deviation. Dynamically adjust the scaling parameters based on the symbol density distribution parameter.

[0080] Step 1082: Combine the translation, rotation, and scaling parameters to generate a 6-DOF affine transformation matrix and output the spatial registration parameters.

[0081] Step 109, based on the 6-DOF parameters of the affine transformation matrix and the symbol density gradient field, calculate the spatial correlation strength coefficient between the text block and the anatomical landmarks, and generate a multidimensional mapping table including polar radius-azimuth parameters, topological connection strength and spatial entropy value.

[0082] In an embodiment of the present invention, when processing an ophthalmic image with mixed text and images of 300×300 pixels in size, multi-scale edge detection is performed. A Gaussian kernel group containing three exponentially increasing variance parameters is used, which are set to σ1=0.5, σ2=1.0, and σ3=2.0 respectively. Taking a pixel with coordinates (100,100) in the image as an example, the original grayscale value of the pixel is 128 (the grayscale value range is 0-255). When a Gaussian kernel with a variance of σ1=0.5 is used for convolution filtering, since the Gaussian kernel is small in size and focuses on details, it will focus on the pixel and several pixels immediately surrounding it. Assume that after the convolution calculation, the filtering result value of the pixel at this layer becomes 85. This value reflects the edge-related information of the position at a small scale, and captures the subtle grayscale changes of the local strokes of the text. Next, the image is convolved again using a Gaussian kernel with a variance of σ2 = 1.0. At this time, the coverage of the Gaussian kernel is expanded, and the grayscale relationship between the pixel and its surrounding pixels in a slightly larger range is comprehensively considered. After calculation, the filtering result value of this pixel at this layer is 92, which reflects the edge features at a slightly larger scale than the previous layer, and may reflect the overall grayscale trend of the small area of ​​the text stroke. Then, convolution is performed using a Gaussian kernel with a variance of σ3 = 2.0, and its coverage is further increased. After calculation, the filtering result value of this pixel at this layer is 100. This value represents edge information at a larger scale, such as being able to capture the approximate contour grayscale features of the local area of ​​the text.

[0083] The results of adjacent layer filters are then subtracted. For example, the difference between the second layer (σ2 = 1.0) and the first layer (σ1 = 0.5) filters yields a value of 92-85 = 7 for the pixel at (100, 100). The difference between the third layer (σ3 = 2.0) and the second layer (σ2 = 1.0) filters yields a value of 100-92 = 8 for the pixel at (100, 100). Larger differences indicate more pronounced changes in edge information at adjacent scales, suggesting a higher likelihood of a true edge. By performing these multiple convolutional filtering and subtraction operations on every pixel in the image, a 300×300 multiresolution edge intensity distribution matrix is ​​generated. The value of the element at position (100, 100) in the matrix represents the final edge intensity value obtained through the above calculations. For example, after comprehensive calculations, the value of the element at this position is 15. Larger values ​​indicate a more pronounced edge at that location.

[0084] Step 101: Continuing with the 300×300 multi-resolution edge intensity distribution matrix generated in the previous step, we analyze one of the resolution levels (e.g., the result after filtering with a Gaussian kernel with variance (σ² = 1.0)). At this level, we traverse all 300×300 = 90,000 pixels in the image and calculate the gradient magnitude for each pixel. Assume that the edge magnitude of pixel (120, 151) is 60, and the edge magnitude of its neighboring pixel (120, 151) is 65. Based on the gradient calculation method (e.g., the difference between adjacent pixels), the gradient magnitude of this pixel is |65 - 60| = 5. After counting the gradient magnitudes of all pixels, we construct a histogram with the gradient magnitude (assuming a range of 0-50) on the horizontal axis and the number of pixels on the vertical axis. Statistics show that within the gradient amplitude range of 5-10, the number of pixels reaches 20,000, forming a relatively concentrated narrow peak. These pixels primarily come from the text area in the medical record. In contrast, within the gradient amplitude range of 15-30, the number of pixels reaches 35,000, with a wider distribution, corresponding to the schematic area. This histogram clearly demonstrates the difference in pixel gradient amplitude distribution between the text area and the schematic area.

[0085] In step 102, based on the histogram statistics obtained in step 101, the maximum between-class variance algorithm (OTSU) is used to calculate the initial segmentation threshold. Assuming the gradient amplitude in the histogram ranges from 0 to 50, the algorithm will try different thresholds t starting from 0. When t = 12, the image pixels are divided into two parts: foreground (gradient amplitude greater than or equal to 12) and background (gradient amplitude less than 12). The average gradient amplitude of the foreground pixels at this time is calculated to be 22, and the average gradient amplitude of the background pixels is 8. Using the between-class variance calculation formula, the between-class variance is 0.35. Continuing to try t = 13, the average gradient amplitudes of the foreground and background are recalculated to be 23 and 7, respectively, and the between-class variance becomes 0.38. After traversing all possible thresholds (0-50), it is found that when t = 14, the between-class variance reaches its maximum value of 0.42. Therefore, the initial segmentation threshold is determined to be 14 to preliminarily distinguish the foreground (text or schematic portion) from the background in the image.

[0086] Step 103: Observe the histogram obtained in step 102. The ratio of the number of pixels in the narrow peak distribution interval (5-10) corresponding to the text area to the total number of pixels is The percentage of pixels in the broad peak distribution range (15-30) corresponding to the schematic area is Since the pixel ratio of the text area is relatively small, in order to more accurately extract the text area, the initial segmentation threshold of 14 is adjusted downward according to the preset adjustment rule. Assume that the adjustment rule is: when the pixel ratio of the text area is less than 0.3, the threshold is subtracted from the ratio difference and multiplied by a coefficient (here the coefficient is 10), that is, 14-(0.3-0.22)×10=13.2. After rounding, the corrected threshold is 13, which makes the segmentation threshold more suitable for the actual situation of the current medical record image.

[0087] In step 104, the corrected threshold value of 13 is applied to the edge intensity distribution matrix for binary segmentation. For the pixel with coordinates (110, 130) in the matrix, its edge intensity value is 15. Since 15>13, the pixel is marked as foreground and assigned a value of 1; and the pixel with coordinates (180, 200) has an edge intensity value of 12. Since 12<13, it is marked as background and assigned a value of 0. After processing all 90,000 pixels in the image, a preliminary binary segmentation result is generated. At this point, the text annotation area and the hand-drawn lesion diagram are preliminarily separated in the image. For example, the text part is mostly assigned a value of 1, and the background area is mostly 0. However, there may be some text edge parts that are misjudged as background, or the diagram and background may not be accurately distinguished.

[0088] Step 105, perform boundary verification on the preliminary binary segmentation result. Take a segmentation boundary centered at coordinates (150, 160) as an example. The gradient directions of this point and the adjacent pixel point (150, 161) are 30° and 120°, respectively. The similarity between the two is calculated according to the gradient direction cosine similarity formula as cos(120°-30°). The similarity is low, indicating that there may be a segmentation error at this location. Further inspection found that there are multiple similar low-similarity situations near this boundary, triggering a secondary correction of the threshold. Based on the distribution and degree of the error area, the threshold is adjusted from 13 to 12.5, and binary segmentation is performed again to generate an optimized binary segmentation mask. After this adjustment, the boundaries of the text and schematic diagram are more accurate. For example, the edge of the text that was originally misjudged is correctly divided into the foreground area.

[0089] Step 106: Based on the optimized binary segmentation mask, a connected domain is found in the text area with a width of 15 pixels. After analysis, it is determined that the main direction of the text strokes is horizontal. According to the rules, an elliptical structure element is dynamically generated, with the major axis aligned with the horizontal direction, the length is set to 15 pixels (the same as the width of the connected domain), and the minor axis length is Pixels, this structural element can closely fit the shape of the text. In the schematic area, the local curvature radius of a certain boundary is 8 pixels. According to the rules, an elliptical structural element is generated, with the major axis direction consistent with the boundary tangent direction, and the minor axis length is scaled according to the curvature radius, set to 8×0.4=3.2 pixels (assuming the scaling factor is 0.4). Through these dynamically generated structural elements, the schematic contour segments are fused, and the originally discontinuous schematic boundaries become smoothly connected. For example, the contours of some tiny pathological symbols are completely connected.

[0090] Step 107: Feature extraction and topology network construction:

[0091] For the smoothly connected area, a rotation-sensitive minimum bounding rectangle algorithm is used to process a text block. The text block contains 10×15 pixels. The algorithm calculates its center coordinates to be (200,210), the rotation angle is 15°, and the aspect ratio is The area ratio is (10×15)÷(300×300)≈0.017. In the schematic diagram, the coordinates of the center of the circle located at the fovea of ​​the retina are (100,100), and the coordinates of the extreme point of curvature of the optic disc edge are (120,110). A topological network of pathological symbols was constructed using Delaunay triangulation. There are three pathological symbols in the network. The shortest Euclidean distance between symbol A and symbol B is 10 pixels, and the azimuth angle is 45°. Based on the symbol distribution, the density distribution parameter of this area is calculated to be 0.8, which obtains the spatial relationship information between the key anatomical landmarks and pathological symbols in the schematic diagram.

[0092] Step 108, assuming that the coordinates of the center of the text block are (180, 180), the coordinates of the center of the corresponding anatomical landmark (fovea) are (100, 100). A polar coordinate system is established with the upper left corner of the image as the origin. The polar diameter of the center of the text block is The azimuth is The polar diameter of the anatomical landmark is The azimuth angle is 45°. The calculated polar deviation is 254.6-141.4=113.2, and the azimuth deviation is 45°-45°=0°, which quantifies the difference in spatial position between the text block and the anatomical landmarks.

[0093] Step 1081 calculates the translation parameters of the affine transformation matrix based on the polar deviation 113.2. The horizontal translation is determined by the radial component of the polar deviation. Assuming the radial correlation coefficient between the horizontal direction and the polar deviation is 0.6, the horizontal translation is 113.2 × 0.6 = 67.92. Since the azimuth deviation is 0°, the vertical translation adjustment value is 0. Because the azimuth deviation is 0°, which is less than 1°, the rotation parameters do not need to be corrected. Based on the symbol density distribution parameter (assuming it is 0.7), the scaling parameter is dynamically adjusted. Set the scaling ratio to 0.7 and determine the adjustment parameters of the text block from the three dimensions of translation, rotation, and scaling.

[0094] In step 1082, the calculated translation parameters (horizontally 67.92, vertically 0), rotation parameter (0°), and scale parameter (0.7) are combined to generate a 6-DOF affine transformation matrix. The text block is transformed according to the parameters in the matrix. After translation and scaling, the text block, which was originally positioned and sized improperly, is now accurately spatially aligned with the anatomical landmarks in the schematic diagram. For example, the lesion location described in the text corresponds exactly to the lesion area in the schematic diagram.

[0095] Step 109 calculates the spatial correlation strength between the text block and the anatomical landmark (e.g., the fovea) based on the 6-DOF parameters (translation, rotation, and scaling parameters) of the affine transformation matrix and the symbol density gradient field. The core of the calculation is to evaluate the spatial "closeness" between the text description and the anatomical structure:

[0096] Distance factor: The Euclidean distance between the center coordinates of the text block (180,180) and the center coordinates of the fovea (100,100) is 254.6 pixels in diameter. The closer the distance, the higher the correlation strength (assuming that the coefficient increases by 0.2 for every 100 pixels decrease in distance).

[0097] Angle factor: An azimuth angle of 45° indicates the direction of the text block relative to the fovea (such as "superior temporal"). The smaller the angle deviation (such as consistent with the anatomical standard orientation), the higher the association strength (assuming that the coefficient decreases by 0.1 for every ±10° angle deviation).

[0098] Density factor: The symbol density gradient field reflects the density of text and schematic diagrams in the medical record (e.g., when the lesion area is densely labeled). The higher the density, the stronger the semantic association between the text and the anatomical structure may be (assuming that the coefficient increases by 0.05 for every 0.1 increase in density).

[0099] Comprehensive calculation example: By taking the weighted sum of the above factors (distance accounts for 40%, angle accounts for 30%, and density accounts for 30%), the spatial correlation strength coefficient is 0.85, indicating that the text block is highly spatially correlated with the fovea (for example, the text description "macular edema" directly corresponds to the macula position in the schematic diagram).

[0100] Integrate the following information into a multi-dimensional mapping table to build a cross-modal association "archive" of text and diagrams:

[0101] Polar diameter-azimuth parameters: A polar diameter of 254.6 pixels quantifies the spatial distance between the text block and the anatomical landmark (which can be converted to actual physical distance, such as 2.546 mm when 1 mm = 100 pixels); an azimuth of 45° (toward the upper right corner) specifies the orientation of the text block relative to the anatomical landmark (e.g., "superior temporal quadrant of the macula").

[0102] Topological connection strength (0.7): When constructing the pathological symbol network, the pathological symbols (such as microaneurysms, hard exudates) in the retinal schematic diagram are connected to anatomical landmarks (such as the fovea) through Delaunay triangulation to form a "map"-like structure. If the text block describes "microaneurysms are located on the temporal side of the macula", the system will check whether the pathological symbol (microaneurysms) mentioned in the text is directly or indirectly connected to the fovea in the triangular network. If there is a close path association between the two, and the weight of the triangle edge passing through is high (representing the credibility of the association), a higher connection strength value is assigned. For example, in this medical record, the lesion symbol described in the text is connected to the fovea through two key paths, and the symbol distribution on the path is reasonable and the association logic is clear. After comprehensive evaluation, the topological connection strength is 0.7, indicating that the text and the anatomical landmarks have established a close association through the pathological symbols.

[0103] Spatial entropy (0.6): This is used to measure the complexity of the semantic information in the area surrounding the text block. The system will demarcate an area with a radius of 100 pixels centered on the text block and count the number, type, and distribution of annotation elements within it. The more types of annotation elements, the more complex the semantics, and the higher the entropy value; the denser the annotations, the more information per unit space, and the higher the entropy value; if multiple annotations point to the same anatomical region, it means that the region is important and there is semantic overlap, which will also increase the entropy value. In this example, there are three pathological symbols around the text block. The annotation distribution is neither sparse nor dense, and some annotations target the same area. Taking all these factors into consideration, the final spatial entropy value is 0.6, indicating that the semantic information complexity of this area is moderate and it is an area that requires special attention when analyzing the condition.

[0104] The multidimensional mapping table records the detailed relationship between the text block and the anatomical landmark points. For example, a row in the table records: the center coordinates of the text block are (180,180), the corresponding anatomical landmark coordinates are (100,100), the polar diameter is 254.6, the azimuth angle is 45°, the spatial correlation strength coefficient is 0.85, the topological connection strength is 0.7, and the spatial entropy value is 0.6.

[0105] Through a series of processes such as multi-scale edge detection, adaptive threshold adjustment, dynamic structural element generation, and affine transformation, the boundary contours of text-annotated areas and hand-drawn lesion diagrams in mixed ophthalmic medical records are accurately extracted, and detailed coordinate mapping tables and multi-dimensional mapping tables are established. On the one hand, this effectively overcomes the problem that traditional methods have difficulty in accurately distinguishing and extracting text-image information, and can cope with the complex distribution of text and diagrams in different medical records; on the other hand, the established mapping table provides a precise spatial and semantic foundation for subsequent multimodal feature fusion and electronic medical record structured generation, improving the accuracy and efficiency of electronic clinical medical evaluation of ophthalmic diagnosis and treatment plans, helping doctors to obtain key information in medical records more quickly and accurately, and assisting in diagnosis and treatment decisions.

[0106] In a preferred embodiment of the present invention, the above step 2, constructing a dual-channel attention gating network based on the coordinate mapping table, fusing the semantic features of the text area with the morphological features of the schematic diagram through a dynamic weight allocation strategy, and generating a multimodal fusion feature matrix with a spatial alignment relationship, may include:

[0107] Step 200: construct a dual-channel parallel network based on the multidimensional mapping table, wherein the text channel extracts text semantic feature vectors through a convolutional neural network, and the schematic channel models the topological relationship of anatomical landmarks through a graph neural network;

[0108] Step 201 , based on the polar radius-azimuth parameters in the multidimensional mapping table, the polar radius deviation and azimuth deviation between the center coordinates of the text block and the corresponding anatomical landmark are calculated to generate a spatial alignment error matrix;

[0109] Step 202: Input the spatial alignment error matrix into the gated attention mechanism to generate a dynamic weight mask through a two-layer fully connected network;

[0110] Step 203: Fusing the dynamic weight mask with the text semantic feature vector channel by channel to generate a weighted text semantic feature vector;

[0111] Step 204: projecting the weighted text semantic feature vector to the schematic coordinate system based on the 6-degree-of-freedom parameters of the affine transformation matrix to obtain the projected text semantic feature;

[0112] Step 205 : Adaptively superimpose the projected text semantic features and schematic diagram morphological features pixel by pixel to generate a multimodal fusion feature matrix.

[0113] In the embodiment of the present invention, the specific analysis process of constructing a dual-channel parallel network is as follows:

[0114] Text channel: Convolutional neural network (CNN) extracts semantic features:

[0115] The pixel values ​​of the bounding rectangle of the text block in the coordinate mapping table (e.g., the text area cropped from the medical record image, with a size of W × H pixels) are used. A lightweight CNN (e.g., MobileNet) is used, consisting of three convolutional layers (each with a kernel size of 3 × 3, a stride of 1, and a padding of 1) and two fully connected layers. The first convolution layer extracts local features of the text strokes (e.g., horizontal and vertical lines, corners), and outputs a feature map of size W × H × 32. The second convolution layer fuses local features to generate semantic fragments (e.g., radicals and letter combinations), and outputs a feature map of size W / 2 × H / 2 × 64. The third convolution layer further abstracts semantics (e.g., complete Chinese characters, medical terms), and outputs a feature map of size W / 4 × H / 4 × 128. After global average pooling and a fully connected layer, the output is a text semantic feature vector of dimension 1 × 1 × 256 (e.g., "macular edema" corresponds to a specific vector pattern).

[0116] Schematic channel: Graph Neural Network (GNN) modeling topological relationship:

[0117] The coordinate mapping table contains the two-dimensional coordinates of anatomical landmarks (such as the fovea and optic disc edge) and a topological network of pathological symbols (node ​​connectivity constructed through Delaunay triangulation). A graph convolutional network (GCN) was used, consisting of two convolutional layers and one fully connected layer. Anatomical landmarks were considered as graph nodes (each node contains attributes such as coordinates x, y, and curvature extremes), and pathological symbol connectivity was considered as graph edges (edge ​​weights are the inverse of the Euclidean distance between symbols). The first layer of graph convolution aggregated features of adjacent nodes (e.g., a node aggregated the coordinates and curvature information of its triangulated neighbors) and updated the node embedding vector to a dimension of 128. The second layer of graph convolution further captured global topological features (such as the shape of the lesion area and the density of symbol distribution), outputting a schematic morphological feature with a dimension of N×128 (N is the total number of landmarks).

[0118] Step 201, polar diameter deviation = |r 文字块 -r 解剖标志点 |(For example, if the polar diameter of the text block is 254.6 and the polar diameter of the marker point is 141.4, then the polar diameter deviation = 113.2). Azimuth deviation = |θ 文字块 -θ 解剖标志点 ∣(For example, if the azimuth angle of the text block is 45° and the azimuth angle of the marker point is 0°, then the azimuth angle deviation = 45°).

[0119] Normalize the polar deviation and azimuth deviation of each text block and the corresponding marker point to the interval [0, 1] (for example, the polar deviation is normalized to The azimuth deviation is According to the spatial position correspondence between the text blocks and the landmarks, a spatial alignment error matrix with a dimension of M×N is generated (M is the number of text blocks, N is the number of landmarks), and the matrix elements are the normalized error values ​​of the corresponding positions.

[0120] In step 202, the spatial alignment error matrix is ​​flattened into a one-dimensional vector and input into a two-layer fully connected network (512 neurons in the first layer and 256 neurons in the second layer). The first fully connected layer uses the ReLU activation function to extract error features (e.g., high error areas correspond to spatial misalignment between text and landmarks); the second fully connected layer uses the Sigmoid activation function to output a dynamic weight mask (with dimensions consistent with the text semantic feature vector, and element values ​​in the range [0, 1]). For example: If the polar deviation between a certain text block and the landmark is large (e.g., polar deviation = 0.3), the corresponding weight mask element value is low (e.g., 0.4), indicating that the semantic features of the text need to be weighted down.

[0121] In step 203, the dynamic weight mask is multiplied by the text semantic feature vector channel by channel (e.g., if a channel value in the feature vector is 0.8 and the corresponding weight is 0.4, the weighted value is 0.8 × 0.4 = 0.32). This generates a weighted text semantic feature vector that emphasizes well-aligned text features (e.g., high-weighted channels corresponding to terms like "macula" that match the location of the landmarks) and suppresses interference from misplaced text (e.g., low-weighted channels corresponding to irrelevant background text).

[0122] Step 204 : performing coordinate transformation on the weighted text semantic feature vector based on the 6-degree-of-freedom parameters (translation, rotation, and scaling) of the affine transformation matrix.

[0123] Specific operations:

[0124] Translation: Move the center coordinates (x w ,y w ) is adjusted to the coordinates of the marker point (x m ,y m )(e.g. horizontal translation 67.92, vertical translation 0);

[0125] Rotation: Compensate the rotation angle according to the azimuth deviation (e.g., if the deviation is 45°, compensate 22.5° according to the rule);

[0126] Scaling: Adjust the scale of the eigenvector according to the symbol density (e.g., if the density parameter is 0.7, the eigenvalue is multiplied by 0.7).

[0127] Results: The semantic features of the text were "mapped" from the text coordinate system to the schematic coordinate system, corresponding one-to-one with the spatial positions of anatomical landmarks (e.g., the feature vector of "macular edema" was located in the macular area of ​​the schematic).

[0128] Step 205: Preprocessing of schematic morphological features: The landmark features (dimension N×128) output by the GNN are expanded into a two-dimensional grid of the same size as the text features through interpolation (e.g., generating a feature map of W / 4×H / 4×128 from N discrete points). For each pixel position, the weighted sum of the text semantic features and the schematic morphological features is calculated: fusion feature = α×text feature + (1-α)×schematic feature, where α ranges from [0, 1], as described below:

[0129] When the spatial alignment error is 0 (the text is completely aligned with the anatomical landmarks), α takes the maximum value of 1. At this time, the fusion feature is completely dominated by the semantic features of the text, and the weight of the schematic morphological features is 0.

[0130] When the spatial alignment error is at its maximum value (the text is completely misaligned with the anatomical landmarks), α takes the minimum value of 0. At this time, the fusion feature is completely dominated by the schematic morphological features, and the weight of the text semantic features is 0.

[0131] In practice, α is dynamically adjusted based on the size of the error: smaller errors lead to α closer to 1, while larger errors lead to α closer to 0. For example, when the error is small (e.g., an error value corresponding to α = 0.7), text features account for 70% and schematic features account for 30%, strengthening the association between semantics and structure. When the error is moderate (e.g., α = 0.5), both are weighted equally, balancing semantic and morphological information. When the error is large (e.g., α = 0.1), text features account for only 10%, preventing diagnostic interference from misplaced text. For example, at the pixel corresponding to the fovea, if the alignment error between text and landmarks is small (α = 0.7), the fused feature is 0.7 × text features + 0.3 × schematic features, strengthening the association between the semantics related to the "macula" and the anatomical structure.

[0132] The text channel captures semantic information (such as the diagnostic significance of "microaneurysm"), and the schematic channel captures spatial morphology (such as lesion location and contour). The fusion of the two avoids the loss of single modal information. Through polar radius / azimuth deviation calculation and dynamic weight allocation, the text semantics and anatomical structure are forced to correspond precisely in space, solving the problem of text and image misalignment in traditional methods (such as the mismatch between text description and schematic area). The gated attention mechanism suppresses spatially misaligned text features, highlights valid semantics (such as only retaining terms such as "edema" and "exudation" related to the current anatomical area), and improves the accuracy of feature expression. The generated multimodal fusion feature matrix can be directly used for the construction of three-dimensional electronic medical records, so that the text description and the lesion schematic are spatially one-to-one corresponding, helping doctors quickly understand complex conditions (such as multi-region damage in diabetic retinopathy) and improve the efficiency of diagnosis and treatment.

[0133] In a preferred embodiment of the present invention, step 3 above sets three text detection reference points in the bidirectional LSTM decoding stage, wherein the first reference point locates the starting boundary of the high-frequency term, the second reference point captures the topological connection relationship between the symbol and the schematic diagram, and the third reference point identifies the cross-modal semantic conflict area. A triangulated verification area is constructed based on the two-dimensional coordinates of the three reference points, and a feature map confidence adjustment parameter is generated by calculating the area and direction angle of the triangulated verification area. This may include:

[0134] Step 300: Based on the multimodal fusion feature matrix, three text detection reference points with the greatest semantic relevance are selected during the bidirectional LSTM decoding process. The first reference point locates the starting boundary coordinates of high-frequency terms, the second reference point captures the coordinate intersection of the topological connection relationship between the symbol and the schematic diagram, and the third reference point identifies the coordinate extreme point of the cross-modal semantic conflict area.

[0135] Step 301: construct a triangular verification region based on the two-dimensional coordinates of the three reference points, calculate the area parameters of the triangular region by using the vector cross product, and solve the main direction angle by combining the inverse tangent function. The area and direction angle are input into a multi-layer perceptron to generate a normalized confidence adjustment parameter.

[0136] Step 302 : Perform a Hadamard product operation on the normalized confidence adjustment parameter and the multimodal fusion feature matrix to output a corrected context feature sequence.

[0137] In this embodiment of the present invention, during the bidirectional LSTM decoding process, the semantic relevance of each character candidate point is determined by the similarity between the eigenvector in the multimodal fusion feature matrix and the current decoding state (e.g., cosine similarity). Assume that the eigenvector of the high-frequency term "macular edema" is v1, and the current decoding state vector is h t , then the semantic relevance is

[0138] Benchmark type and location:

[0139] First reference point: Locate the starting boundary coordinates of high-frequency terms (such as "microvascular tumor" and "exudation") in the feature map. By traversing the feature map, find the starting point of a continuous region with a semantic relevance greater than a threshold (such as 0.8), for example, the coordinates (x1, y1).

[0140] Secondary reference points: Capture the intersection of pathological symbols (e.g., arrows, circles) and the schematic topology network. Using the topological relationships output by the graph neural network, find the midpoint between the symbol and the anatomical landmark (e.g., the optic disc), for example, the coordinates (x2, y2).

[0141] The third reference point identifies the coordinate extremes of areas of cross-modal semantic conflict (e.g., text describing "no edema" but the diagram showing edema). By comparing the semantic differences between the text and the morphological features of the diagram, the coordinates of the point with the greatest discrepancy are located, for example, at (x3, y3).

[0142] Step 301: Constructing triangulation verification area and parameter calculation:

[0143] The two-dimensional coordinates of the three reference points (x1, y1), (x2, y2), and (x3, y3) are used as the vertices of a triangle to form a triangular calibration area. The area of ​​the triangle is calculated using the vector cross product formula: If the coordinates of the three points are (10,20), (30,40), and (50,10), then the area is

[0144] Calculate the main direction angle of the triangle, that is, the angle between the vector from the first reference point to the triangle centroid and the horizontal axis: The center of mass is (30,23.3), then the direction angle is

[0145] The area and orientation angle are input into a multi-layer perceptron (MLP), which uses a nonlinear transformation (such as the ReLU activation function) to output a normalized confidence adjustment parameter β∈[0,1]. A larger area indicates a wider semantic association region and a higher confidence level. A orientation angle closer to 0° or 180° indicates a more ordered semantic distribution and a higher confidence level.

[0146] Step 302: multiply the normalized confidence parameter β by the multimodal fusion feature matrix element by element (Hadamard product), and the formula is: i,j = Fusion features i,j ×β, if the fusion feature value of a pixel is 0.6 and β = 0.8, then the corrected feature value is 0.6 × 0.8 = 0.48. The corrected context feature sequence highlights high-confidence areas (such as semantically consistent macular lesion areas) and suppresses low-confidence areas (such as mislabeled areas with cross-modal conflicts).

[0147] Accurately locate key semantic positions through the starting points of high-frequency terms (first reference points) to avoid generalized misdetection (such as misjudging "vitreous" as "retina"). Use the topological intersection of symbols and schematics (second reference points) to strengthen the spatial association between images and text, for example, to ensure that the text "microaneurysm" corresponds one-to-one with the lesion symbol in the schematic. Identify conflicting areas between images and text (such as the text describes "normal" but the schematic shows abnormality) through cross-modal conflict extreme points (third reference points) to reduce the risk of misdiagnosis. The area and directional angle parameters of the triangulation verification area quantify semantic reliability from a spatial geometric perspective, so that the LSTM decoding process pays more attention to high-confidence features, improving the accuracy and clinical credibility of electronic medical record generation.

[0148] In a preferred embodiment of the present invention, step 4, adjusting the parameters according to the confidence level and utilizing the geometric characteristics of the triangulated verification region to constrain the calculation process of the self-attention score matrix to obtain the modified spatial dependency weights, may include:

[0149] Step 401: Based on the modified context feature sequence, the spatial dependency in the self-attention score matrix is ​​extracted, and the score matrix is ​​weighted and modified using the confidence adjustment parameter. The area parameter of the triangular verification region is used to adjust the local attention range, and the direction angle parameter is used to constrain the attention direction, so as to obtain a weighted and modified self-attention score matrix.

[0150] In step 402, a matrix multiplication operation is performed on the weighted and corrected self-attention score matrix and the multimodal fusion feature matrix to generate a corrected spatial dependency weight.

[0151] In this embodiment of the present invention, the self-attention score matrix is ​​used to measure the semantic association strength between different pixel positions in the feature map. Each element S in the initial score matrix I,J Represents the attention weight of position I on position J, and its value range is between [0, 1]. The closer this value is to 1, the higher the semantic dependence of position I on position J is; the closer it is to 0, the weaker the association between the two.

[0152] In the specific calculation, the initial score S is determined by comparing the similarity of the feature vectors at position I and position J. I,J For example, using cosine similarity calculation, the feature vector at position I is recorded as V I , the eigenvector of position J is recorded as V J , then the initial score After normalization, the values ​​are ensured to fall within the range [0, 1]. For example, in the area describing "retinal blood vessels", the initial scores between pixel positions related to blood vessel features will be relatively high.

[0153] The area parameter adjusts the local attention range:

[0154] The area parameter A of the triangulation area reflects the spatial range of the semantic association. The larger the area, the wider the semantic association area, and the more extensive the local attention coverage needs to be. First, the area parameter A is normalized to the interval [0, 1]. Assuming the original area is A0, the normalized area is Among them A min and A max are the minimum and maximum areas of all triangulated check regions respectively.

[0155] Then, according to the normalized area A norm Determine the neighborhood radius r for attention enhancement a =k×A norm (k is the set proportional coefficient, for example, k = 100). For the scoring matrix with the centroid of the triangular area as the center and the radius r a Elements S in the neighborhood of I,J , multiply it by (1+A norm ) for weighted enhancement. For example, if the normalized area A norm =0.6, then the score value S in the neighborhood I,J Change to S I,J ×(1+0.6)=1.6×S I,H , which increases the attention weight between pixels in the area.

[0156] The direction angle parameter constrains the attention direction:

[0157] The direction angle parameter θ represents the main direction of the semantic association. Calculate the direction angle θ of the line connecting position I and position J I,J and compare it with the direction angle θ of the triangle verification area. I,J -θ|>45°, indicating that the association direction between position I and position J deviates greatly from the main semantic direction. In this case, the score S I,J Multiply by the attenuation factor 0.5. For example, when |θ i,j When -θ|=60°, S I,J Change to S I,H × 0.5 = 0.5 × S I,H , thereby suppressing invalid attention perpendicular to the main semantic direction.

[0158] After adjusting the area and direction angle parameters, the weighted corrected self-attention score matrix is ​​obtained Each element in the matrix S I,J is the initial score S I,J Based on the above, it is multiplied by the area weighting factor and the direction weighting factor, that is, S I,J =S I,J× area weighting factor × direction weighting factor. For example, a position J is located in the neighborhood of the triangular area and the direction angle deviation is 30°, and the initial score S I,J = 0.7, the area weighting factor is 1.4, and the direction weighting factor is 0.8, then the corrected score S I,J =0.7×1.4×0.8=0.784.

[0159] Step 402: The weighted modified self-attention score matrix (dimension is H×W, H and W are the height and width of the feature map respectively) and performs matrix multiplication with the multimodal fusion feature matrix F (dimension is H×W×C, C is the number of feature channels).

[0160] For each channel c of the multimodal fusion feature matrix, calculate During the specific operation, for each position (x, y) on the feature map, the dependency weight c (x,y) is equal to The element in row x and column y in F c The sum of the products of the eigenvalues ​​at the (x, y) positions in the matrix. After the matrix multiplication operation, the dimension of the modified spatial dependency weight matrix generated is also H×W×C. The value of each element in the matrix represents the semantic dependency strength of the corresponding position after considering the geometric constraints of the triangulation area. In areas with close semantic associations and conforming to geometric constraints, such as the text accurately describing "macular edema" and its corresponding schematic area, the spatial dependency weights of the related positions will be larger; while in areas with weak semantic associations or those that do not conform to geometric constraints, the weights will be smaller, thereby highlighting the characteristics of key semantic areas and suppressing the interference of irrelevant or erroneous information.

[0161] The area parameter makes the attention coverage match the spatial scale of the semantic association (such as a large-scale lesion area expands attention, and a small-scale symbol focuses on the local area), avoiding the computational redundancy of global attention. The direction angle parameter forces attention to propagate along the main direction of the semantic association (such as the direction of the text line, the direction of the lesion spread), improving the spatial consistency of contextual features, for example, ensuring that the text "temporal vessels" only focuses on the vascular morphological features in the horizontal direction. The geometric characteristics (area, direction) of the triangulated verification area are used to filter invalid attention connections (such as low-confidence associations in cross-modal conflict areas), reducing the risk of misjudgment caused by image and text misalignment or semantic contradictions. The corrected spatial dependency weight has clear geometric semantics (such as the area of ​​the triangular area represents the range of the lesion, and the direction angle represents the direction of the anatomical structure), making the model decision-making process more consistent with the visual cognitive logic of clinicians and improving interpretability.

[0162] In a preferred embodiment of the present invention, step 5, constructing a medical entity topology map based on the corrected spatial dependency weights, generating semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the diagram, performing structural reorganization on the verified entity nodes, and outputting a three-dimensional electronic medical record report, may include:

[0163] Step 500: Based on the modified spatial dependency weights, the polar radius-azimuth parameters of the anatomical landmarks and text blocks in the multidimensional mapping table are traversed to generate semantic nodes with orientation attributes, wherein the three-dimensional coordinates of the nodes are converted from the polar radius parameters to the spatial radius, and the azimuth parameters are converted to spherical coordinates. The node weight is determined by the weighted average of the spatial correlation strength coefficient and the topological connection strength.

[0164] Step 501: Control the node distribution density based on the area parameter of the triangulation verification region and construct a three-dimensional topological structure of the clinical entity by combining the semantic association strength in the multimodal feature data;

[0165] Step 502 , optimize the three-dimensional topological structure of the clinical entity based on the spatial entropy value in the multidimensional mapping table, and output a structured electronic medical record report containing the three-dimensional spherical coordinates of anatomical landmarks, spatial density distribution of pathological symbols, and semantic association strength.

[0166] In the embodiment of the present invention, the polar diameter parameter r x (Unit: pixel) is converted proportionally to the radius R in three-dimensional space x (Unit: mm), assuming that the mapping ratio of pixels to millimeters is 1mm = 100 pixels, then For example, the polar radius r x = 254.6 pixels converted to R x =2.546mm.

[0167] Azimuth angle to spherical coordinates: Take the anatomical landmark (such as the fovea) as the origin and establish a spherical coordinate system: the azimuth angle θ (horizontal angle) is directly used as the azimuth angle of the spherical coordinates; the polar angle Set it to 0° (assuming all nodes are on the same horizontal plane) and generate three-dimensional coordinates

[0168] Node weight is calculated by taking the weighted average of the spatial correlation strength coefficient and the topological connection strength. Specifically, it is calculated as follows: node weight = 0.6 × spatial correlation strength + 0.4 × topological connection strength. The spatial correlation strength coefficient measures the spatial closeness between a text block and an anatomical landmark (such as the fovea of ​​the retina). Its value range is [0, 1], with values ​​closer to 1 indicating a stronger association. This coefficient is calculated by comprehensively considering factors such as distance, angle, and symbol density:

[0169] Distance factor: The Euclidean distance between the center coordinates of the text block and the anatomical landmark is a key consideration. For example, if the center coordinates of the text block are (180, 180), the fovea coordinates are (100, 100), and the epipolar diameter is 254.6 pixels, the closer the distance, the higher the spatial correlation strength coefficient. Assume that for every 100-pixel decrease in distance, the coefficient increases by 0.2.

[0170] Angle factor: The azimuth angle represents the orientation of the text block relative to the anatomical landmark. When the orientation of the text block matches the actual orientation of the anatomical landmark, the angular deviation is small and the association strength is high. For example, if the azimuth angle is 45°, the coefficient decreases by 0.1 for every ±10° of angular deviation.

[0171] Density factor: The symbol density gradient field reflects the density of text and symbol distribution in the medical record. Text and symbols are denser near the lesion area. For every 0.1 increase in symbol density in that area, the coefficient increases by 0.05.

[0172] By taking a weighted sum of these factors (distance accounts for 40%, angle accounts for 30%, and density accounts for 30%), we ultimately arrive at the spatial correlation strength coefficient. For example, the spatial correlation strength coefficient between a certain text block and the fovea is calculated to be 0.85, indicating a strong spatial and semantic association between the two.

[0173] The topological connection strength is used to indicate the closeness of the connection between the text block and the anatomical landmark through the pathological symbol. The value is also between [0, 1]. The closer the value is to 1, the closer the connection is. Its calculation is based on the pathological symbol network constructed by Delaunay triangulation:

[0174] Network connection path: The system checks whether the pathological symbols mentioned in the text (such as microaneurysms) are directly or indirectly connected to anatomical landmarks (such as the fovea) in the triangular network. If there is a close path association and the weight of the triangle edge passing through is high (representing the credibility of the association), a higher connection strength value is assigned. Assuming that in a medical record, the lesion symbol described in the text is connected to the fovea through two key paths, and the symbols on the path are reasonably distributed and the association logic is clear, the topological connection strength is 0.7 after comprehensive evaluation.

[0175] After determining the spatial association strength coefficient to be 0.85 and the topological connection strength to be 0.7, the node weight was calculated using the weighted average formula: Node weight = 0.6 × 0.85 + 0.4 × 0.7 = 0.51 + 0.28 = 0.79. This calculated result, 0.79, represents the importance of the node in the medical entity topology map. A higher weight indicates a closer association between the text information represented by the node and the anatomical structure and pathology symbols.

[0176] Each semantic node contains the following attributes:

[0177] Text description: such as "macular edema";

[0178] Three-dimensional coordinates: converted from polar radius and azimuth (R x ,θ,□);

[0179] Direction attribute: such as "45° above the temporal axis";

[0180] Node weight: A weighted value that combines the strength of association and topological connection.

[0181] Step 501: The area parameter A (unit: pixel2) of the triangulation verification region reflects the spatial range of the semantic association. The larger the area, the lower the distribution density of the corresponding nodes in the topology graph (avoiding dense overlap).

[0182] Density adjustment formula: The density is That is, nodes are sparsely distributed in three-dimensional space at this density. Based on the semantic association strength (e.g., the degree of match between text and diagram) in the multimodal fusion features, directed edges are established between nodes. When the association strength exceeds a threshold (e.g., 0.6), a connecting edge is generated, with the edge weight being the association strength value. For example, if the association strength between the "macular edema" node and the "hard exudate" node is 0.75, a connection with an edge weight of 0.75 is established.

[0183] Map all semantic nodes and their connecting edges into three-dimensional space to form a radial topological graph centered on the anatomical landmark. For example:

[0184] The fovea serves as the origin;

[0185] The “macular edema” node is located 45° superior to the temporal axis, with a radius of 2.5 mm, and is connected to the “microaneurysm” node (edge ​​weight 0.8).

[0186] Step 502: The spatial entropy value (value range [0, 1], such as 0.6) reflects the semantic complexity of the region:

[0187] The higher the entropy value, the denser the annotations (e.g., multiple lesion areas), and the connection weights between nodes need to be increased;

[0188] The lower the entropy value, the simpler the semantics (such as normal organization), which can simplify the topological structure.

[0189] Optimization operation: For areas with entropy values ​​higher than 0.5, the node connection weight is increased by 20%; for areas with entropy values ​​lower than 0.3, weak connections (weight < 0.4) are deleted.

[0190] The report contains the following:

[0191] Coordinates of anatomical landmarks: such as the three-dimensional coordinates of the fovea (0, 0, 0);

[0192] Pathological symbol density: the number of symbols counted by area (e.g., 3 microaneurysms per square millimeter);

[0193] Semantic association strength: a list of connection weights between nodes (e.g., “macular edema” - “microaneurysm”: 0.8);

[0194] 3D visualization parameters: node color (coded by weight, such as red for high weight), size (scaled by density).

[0195] The graphic and text information in the two-dimensional medical records is converted into three-dimensional nodes with orientation attributes, which intuitively presents the spatial relationship between the lesion location and the anatomical structure (such as "exudation is located 2.5mm temporal to the macula"), meeting the clinical needs of ophthalmology for spatial positioning. Node weights and connecting edges are directly associated with pathological features (such as spatial correlation strength reflects the credibility of text descriptions, and topological connection strength reflects the pathological connection between lesions), helping doctors quickly understand the progression of the disease (such as the association path between edema and microaneurysms). The topological structure is optimized through spatial entropy value, redundant information is automatically filtered, and a report focusing on key lesions is generated, reducing manual sorting time. For example, a diabetic retinopathy report can directly display the three-dimensional model and correlation strength of high-entropy areas (areas with dense multiple lesions). The three-dimensional topological structure can be directly input into the AI ​​diagnostic model, combined with the pathological feature density and semantic association, to assist in predicting lesion progression (such as a combination of lesions with high weight connections indicates a more serious course of disease), thereby improving the efficiency and accuracy of diagnosis and treatment.

[0196] like Figure 2 As shown, an embodiment of the present invention further provides an electronic clinical medical evaluation system for ophthalmic diagnosis and treatment plans, comprising:

[0197] The feature extraction module is used to perform multi-scale edge detection calculations on the input ophthalmic medical record image with mixed text and images. It extracts the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operations, and establishes a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics;

[0198] The feature fusion module is used to build a dual-channel attention gating network based on the coordinate mapping table. It fuses the semantic features of the text area with the morphological features of the schematic diagram through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship.

[0199] The reference point construction module is used to set three text detection reference points in the bidirectional LSTM decoding stage. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection relationship between symbols and schematics, and the third reference point identifies the cross-modal semantic conflict area. The triangulation verification area is constructed based on the two-dimensional coordinates of the three reference points. The feature map confidence adjustment parameters are generated by calculating the area and direction angle of the triangulation verification area.

[0200] The weight correction module is used to adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the corrected spatial dependency weights;

[0201] The electronic medical record generation module is used to construct a medical entity topology map based on the corrected spatial dependency weights, generate semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the schematic diagram, perform structured reorganization on the verified entity nodes, and output a three-dimensional electronic medical record report.

[0202] It should be noted that this system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0203] An embodiment of the present invention further provides a computing device comprising: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the above-described method. All implementations in the above-described method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0204] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, causes the computer to execute the above-described method. All implementations in the above-described method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0205] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans, characterized in that: The method comprises: Step 1: Perform multi-scale edge detection on the input ophthalmic medical record image, extract the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operation, and establish a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics; Step 2: Based on the coordinate mapping table, a dual-channel attention gating network is constructed. The semantic features of the text area and the morphological features of the schematic diagram are fused through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship. Step 3: Set three text detection reference points during the bidirectional LSTM decoding phase. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection between symbols and schematics, and the third reference point identifies cross-modal semantic conflict areas. A triangulation region is constructed based on the two-dimensional coordinates of the three reference points. The confidence adjustment parameters of the feature map are generated by calculating the area and orientation angle of the triangulation region. Step 4: Adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the corrected spatial dependency weights; Step 5: Construct a medical entity topology map based on the corrected spatial dependency weights, generate semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the schematic diagram, perform structural reorganization on the verified entity nodes, and output a three-dimensional electronic medical record report.

2. The electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans according to claim 1, characterized in that: Perform multi-scale edge detection on the input ophthalmic medical record image, extract the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operation, and establish a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics, including: Multi-scale edge detection is performed on the input ophthalmological mixed medical record image. The image is convolutionally filtered layer by layer using a Gaussian kernel group with an exponentially increasing variance parameter. The filtering results of adjacent layers are differentially calculated to generate a multi-resolution edge intensity distribution matrix. Based on the edge intensity distribution matrix, the pixel gradient amplitude at each resolution level is statistically analyzed by histogram, and the narrow peak distribution corresponding to the text area and the broad peak distribution corresponding to the schematic area in the histogram are identified to obtain the histogram statistical results. Based on the histogram statistics, the narrow peak distribution corresponding to the text area and the broad peak distribution corresponding to the schematic area are identified, and the maximum inter-class variance algorithm is used to calculate the initial segmentation threshold; The initial segmentation threshold is dynamically modified according to the distribution area ratio of the narrow peak and the broad peak to obtain the modified threshold; Apply the modified threshold to the edge intensity distribution matrix for binary segmentation to generate a preliminary binary segmentation result; Perform boundary verification on the preliminary binary segmentation result, calculate the gradient direction cosine similarity of adjacent pixels on the segmentation boundary, determine the segmentation error area, and trigger the threshold secondary correction to generate the optimized binary segmentation mask; Based on the optimized binary segmentation mask, an elliptical structural element is dynamically generated according to the pixel area distribution of the connected domain of the text area, in which the long axis is aligned with the main direction of the text strokes and the short axis length is the width of the connected domain. At the same time, an elliptical structural element is dynamically generated according to the local curvature radius of the schematic boundary. The direction of the long axis is consistent with the direction of the boundary tangent and the length of the short axis is scaled according to the curvature radius. The schematic contour fragments are fused to generate a smoothly connected area partition result. For smoothly connected areas, a rotation-sensitive minimum enclosing rectangle algorithm was used to extract the center coordinates, rotation angle, aspect ratio, and area ratio of the text blocks. The center coordinates of the retinal fovea and the extreme points of curvature of the optic disc margin were located based on the fused schematic outline. A topological network of pathological symbols was constructed using Delaunay triangulation, and the shortest Euclidean distance, orientation angle, and density distribution parameters between symbols were calculated. Based on the spatial relationship between the center coordinates of the text block and the anatomical landmarks on the schematic diagram, the 6-degree-of-freedom parameters of the affine transformation matrix are dynamically calculated according to the polar coordinate offset between the text block and the anatomical landmarks to generate the spatial registration parameters for cross-modal alignment; Based on the 6-degree-of-freedom parameters of the affine transformation matrix and the signed density gradient field, the spatial correlation strength coefficients between the text blocks and anatomical landmarks were calculated, and a multidimensional mapping table containing polar radius-azimuth parameters, topological connection strength, and spatial entropy values ​​was generated.

3. The electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans according to claim 2, characterized in that: Based on the spatial relationship between the center coordinates of the text block and the anatomical landmarks on the schematic diagram, and according to the polar coordinate offset between the text block and the anatomical landmarks, the 6-degree-of-freedom parameters of the affine transformation matrix are dynamically calculated to generate the spatial registration parameters for cross-modal alignment, including: Based on the center coordinates of the text block and the polar radius-azimuth parameters of the anatomical landmarks on the schematic diagram, the polar radius deviation and azimuth deviation between the text block and the corresponding anatomical landmarks are calculated; Based on the polar deviation, the translation parameters of the affine transformation matrix are dynamically calculated, where the horizontal translation is determined by the radial component of the polar deviation, and the vertical translation is adjusted by the sine component of the azimuth deviation. Based on the azimuth deviation, the rotation parameters are dynamically calculated. When the azimuth deviation exceeds 1°, the rotation angle is compensated by 0.5° for every 1° deviation. The scaling parameters are dynamically adjusted according to the symbol density distribution parameters. The translation, rotation and scaling parameters are combined to generate a 6-DOF affine transformation matrix and output the spatial registration parameters.

4. The electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans according to claim 3, characterized in that: A dual-channel attention gating network is constructed based on the coordinate mapping table. The semantic features of the text area and the morphological features of the schematic diagram are fused through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with spatial alignment, including: Based on the multidimensional mapping table, a dual-channel parallel network is constructed. The text channel extracts text semantic feature vectors through a convolutional neural network, and the schematic channel models the topological relationship of anatomical landmarks through a graph neural network. Based on the polar radius-azimuth parameters in the multidimensional mapping table, the polar radius deviation and azimuth deviation between the center coordinates of the text block and the corresponding anatomical landmark are calculated to generate a spatial alignment error matrix. The spatial alignment error matrix is ​​fed into the gated attention mechanism to generate a dynamic weight mask through a two-layer fully connected network. Fuse the dynamic weight mask with the text semantic feature vector channel by channel to generate a weighted text semantic feature vector; Based on the 6-degree-of-freedom parameters of the affine transformation matrix, the weighted text semantic feature vector is projected onto the schematic coordinate system to obtain the projected text semantic feature; The projected text semantic features and schematic morphological features are adaptively superimposed pixel by pixel to generate a multimodal fusion feature matrix.

5. The electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans according to claim 4, characterized in that: Three text detection reference points are set in the bidirectional LSTM decoding stage. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection between symbols and schematics, and the third reference point identifies the cross-modal semantic conflict area. A triangulated verification area is constructed based on the two-dimensional coordinates of the three reference points. The feature map confidence adjustment parameters are generated by calculating the area and orientation angle of the triangulated verification area, including: Based on the multimodal fusion feature matrix, three text detection reference points with the greatest semantic relevance are selected during the bidirectional LSTM decoding process. The first reference point locates the starting boundary coordinates of high-frequency terms, the second reference point captures the coordinate intersection of the topological connection between the symbol and the schematic diagram, and the third reference point identifies the coordinate extreme point of the cross-modal semantic conflict area. A triangular verification region is constructed based on the two-dimensional coordinates of the three reference points. The area parameters of the triangular region are calculated by vector cross product, and the main direction angle is solved by combining the inverse tangent function. The area and direction angle are input into a multi-layer perceptron to generate normalized confidence adjustment parameters. The normalized confidence adjustment parameter is subjected to a Hadamard product operation with the multimodal fusion feature matrix to output a corrected context feature sequence.

6. The electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans according to claim 5, characterized in that: Adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the modified spatial dependency weights, including: Based on the modified context feature sequence, the spatial dependency in the self-attention score matrix is ​​extracted, and the score matrix is ​​weighted and modified by the confidence adjustment parameter. The area parameter of the triangular verification region is used to adjust the local attention range, and the direction angle parameter is used to constrain the attention direction, so as to obtain the weighted and modified self-attention score matrix. The weighted modified self-attention score matrix is ​​multiplied with the multimodal fusion feature matrix to generate the modified spatial dependency weights.

7. The electronic clinical medical evaluation method for ophthalmic diagnosis and treatment plans according to claim 6, characterized in that: A medical entity topology map is constructed based on the corrected spatial dependency weights. Semantic nodes with orientation attributes are generated by traversing the mapping relationship between anatomical landmarks and text annotations in the diagram. After verification, structural reorganization is performed on the entity nodes to output a 3D electronic medical record report, including: Based on the modified spatial dependency weights, the polar radius-azimuth parameters of anatomical landmarks and text blocks in the multidimensional mapping table are traversed to generate semantic nodes with orientation attributes. The three-dimensional coordinates of the nodes are converted from the polar radius parameters to the spatial radius, and the azimuth parameters are converted to spherical coordinates. The node weight is determined by the weighted average of the spatial association strength coefficient and the topological connection strength. The node distribution density is controlled based on the area parameter of the triangulation verification region, and the semantic association strength in the multimodal feature data is combined to construct the three-dimensional topological structure of the clinical entity. The three-dimensional topological structure of clinical entities is optimized based on the spatial entropy value in the multidimensional mapping table, and a structured electronic medical record report containing the three-dimensional spherical coordinates of anatomical landmarks, spatial density distribution of pathological symbols, and semantic association strength is output.

8. An electronic clinical medical evaluation system for ophthalmic diagnosis and treatment plans, the system implementing the method according to any one of claims 1 to 7, characterized in that: include: The feature extraction module is used to perform multi-scale edge detection calculations on the input ophthalmic medical record image with mixed text and images. It extracts the boundary contours of the text annotation area and the hand-drawn lesion diagram through adaptive threshold segmentation and morphological closing operations, and establishes a coordinate mapping table containing the coordinates of the circumscribed rectangle of the text block, the geometric positions of the key anatomical landmarks on the diagram, and the symbol spacing characteristics; The feature fusion module is used to build a dual-channel attention gating network based on the coordinate mapping table. It fuses the semantic features of the text area with the morphological features of the schematic diagram through a dynamic weight allocation strategy to generate a multimodal fusion feature matrix with a spatial alignment relationship. The reference point construction module is used to set three text detection reference points in the bidirectional LSTM decoding stage. The first reference point locates the starting boundary of high-frequency terms, the second reference point captures the topological connection relationship between symbols and schematics, and the third reference point identifies the cross-modal semantic conflict area. The triangulation verification area is constructed based on the two-dimensional coordinates of the three reference points. The feature map confidence adjustment parameters are generated by calculating the area and direction angle of the triangulation verification area. The weight correction module is used to adjust the parameters according to the confidence level and use the geometric characteristics of the triangulated verification area to constrain the calculation process of the self-attention score matrix to obtain the corrected spatial dependency weights; The electronic medical record generation module is used to construct a medical entity topology map based on the corrected spatial dependency weights, generate semantic nodes with orientation attributes by traversing the mapping relationship between anatomical landmarks and text annotations in the schematic diagram, perform structured reorganization on the verified entity nodes, and output a three-dimensional electronic medical record report.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images