Medical form data identification method and system based on OCR and MLLM

Abstract

The invention discloses a medical form data identification method and system based on OCR and MLLM, and the method comprises the steps: receiving original image data, and generating an optimized image sequence through adaptive contrast enhancement, regional importance weighting and multi-resolution optimization; feature extraction is carried out based on the optimized image sequence, a text-space attention map is constructed, and text recognition is realized; performing visual coding and text coding on the optimized image sequence and the text recognition result, and obtaining uniform feature representation through feature alignment; constructing an information level diagram based on unified feature representation, performing relation reasoning, and combining professional knowledge verification to obtain structured feature information; and performing data verification on the structured feature information to obtain normalized data. According to the invention, efficient identification and accurate structuring of the medical form are realized, and the digitization efficiency of the medical data is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data recognition, and in particular to a medical form data recognition method and system based on OCR and MLLM. Background Art

[0002] Intelligent recognition and structuring of medical form data is an important foundation for the construction of medical informationization. A large number of medical forms such as test reports, prescriptions, and medical records contain key health information of patients. Converting these unstructured form data into standardized electronic information is of great significance for improving medical efficiency, supporting clinical decision-making, and promoting medical big data analysis. Automated form data recognition technology can reduce the workload of manual entry, reduce data error rate, accelerate the digitization process of medical data, and provide data support for the development of smart medicine.

[0003] At present, medical form data recognition mainly adopts a method that combines traditional OCR technology with rule templates. The typical processing flow includes steps such as image preprocessing, text detection, character recognition, and template matching. Some studies have improved the accuracy of text recognition by introducing deep learning technology, such as using CNN for feature extraction, using LSTM for sequence modeling, or using Transformer to enhance the processing ability of long texts. In terms of form structure analysis, it mainly relies on predefined layout analysis rules and fixed table parsing templates, and realizes the recognition of table lines and the division of cells through computer vision technologies such as edge detection and connected domain analysis.

[0004] However, existing technologies still face many challenges when processing medical forms: First, the multi-scale visual feature extraction in medical forms is insufficient. For areas that contain visual elements of different sizes such as table lines, text, and numbers, it is difficult to balance the feature expression of global structure and local details; second, the spatial position information is ignored in the text recognition process, resulting in low efficiency of serial text recognition in table cells and prone to string errors; third, due to the lack of professional medical knowledge constraints, it is impossible to accurately understand the semantic association between test items and corresponding values, units, and reference values; fourth, the alignment of cross-modal features is not accurate enough, and the mapping relationship between visual features and text features is fuzzy, which affects the accuracy of structured information; fifth, data verification and correction lack global consistency considerations, and local data corrections may lead to the destruction of the overall logical relationship. These technical problems restrict the efficiency and reliability of medical form data recognition. Summary of the invention

[0005] The purpose of the invention is to provide a medical form data recognition method and system based on OCR and MLLM to solve the above-mentioned problems existing in the prior art.

[0006] The technical solution is a medical form data recognition method based on OCR and MLLM, which includes the following steps:

[0007] S1. Receive original image data of a professional field form, perform contrast enhancement, and obtain an enhanced image; calculate region weights based on the enhanced image to obtain a region importance weight matrix; adjust the resolution based on the region importance weight matrix to obtain a multi-resolution image sequence; perform edge optimization processing on the multi-resolution image sequence to obtain a multi-resolution optimized image sequence;

[0008] S2. Based on the multi-resolution optimized image sequence, feature vectors are extracted to obtain text feature vectors and spatial position feature vectors; based on the text feature vectors and spatial position feature vectors, attention associations are constructed to obtain a text-spatial attention map; based on the text-spatial attention map, text recognition is performed to obtain an initial text recognition result and a confidence matrix; based on the initial text recognition result and the confidence matrix, low confidence areas are extracted and optimized to obtain an updated text recognition result;

[0009] S3, visually encode the multi-resolution optimized image sequence to obtain a visual feature tensor; perform text encoding on the updated text recognition result to obtain a text feature tensor; calculate feature alignment based on the visual feature tensor and the text feature tensor to obtain a cross-modal alignment matrix; based on the cross-modal alignment matrix, fuse the visual feature tensor and the text feature tensor to obtain a unified feature representation;

[0010] S4. Based on the unified feature representation, a hierarchical graph structure is constructed to obtain an information hierarchy graph; relational reasoning is performed on the information hierarchy graph to obtain a structural relationship matrix; based on the structural relationship matrix and the pre-stored professional knowledge base, the relationship confidence matrix is ​​verified to obtain the relationship confidence matrix; the structural relationship matrix and the relationship confidence matrix are integrated to obtain structured domain feature information;

[0011] S5. Perform data verification on the structured domain feature information to obtain a verification result matrix; and generate normalized data based on the verification result matrix.

[0012] Medical form data recognition system based on OCR and MLLM, including:

[0013] at least one processor; and,

[0014] a memory communicatively connected to at least one of the processors; wherein,

[0015] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the medical form data recognition method based on OCR and MLLM.

[0016] Beneficial effects: The present invention solves the problem of diverse imaging quality in medical forms, providing a high-quality image foundation for subsequent recognition; accurately captures the correlation characteristics of text content and position in the form, breaking through the limitations of traditional OCR in processing complex forms; realizes the deep integration of visual layout information and text semantic information, providing strong support for accurate understanding of form content; combines professional knowledge for deep reasoning, ensures the professionalism and logic of data association, and realizes automated guarantee of data quality; improves the efficiency and accuracy of medical data digitization, and provides reliable technical support for the construction of medical informatization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of the method of the present invention.

[0018] Figure 2 This is a flow chart of step S1 of the present invention.

[0019] Figure 3 This is a flow chart of step S2 of the present invention.

[0020] Figure 4 This is a flow chart of step S3 of the present invention.

[0021] Figure 5 This is a flow chart of step S4 of the present invention.

[0022] Figure 6 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, this application proposes a medical form data recognition method based on OCR and MLLM, comprising the following steps:

[0024] S1. Receive original image data of a professional field form, perform contrast enhancement, and obtain an enhanced image; calculate region weights based on the enhanced image to obtain a region importance weight matrix; adjust the resolution based on the region importance weight matrix to obtain a multi-resolution image sequence; perform edge optimization processing on the multi-resolution image sequence to obtain a multi-resolution optimized image sequence;

[0025] S2. Based on the multi-resolution optimized image sequence, feature vectors are extracted to obtain text feature vectors and spatial position feature vectors; based on the text feature vectors and spatial position feature vectors, attention associations are constructed to obtain a text-spatial attention map; based on the text-spatial attention map, text recognition is performed to obtain an initial text recognition result and a confidence matrix; based on the initial text recognition result and the confidence matrix, low confidence areas are extracted and optimized to obtain an updated text recognition result;

[0026] S3, visually encode the multi-resolution optimized image sequence to obtain a visual feature tensor; perform text encoding on the updated text recognition result to obtain a text feature tensor; calculate feature alignment based on the visual feature tensor and the text feature tensor to obtain a cross-modal alignment matrix; based on the cross-modal alignment matrix, fuse the visual feature tensor and the text feature tensor to obtain a unified feature representation;

[0027] S4. Based on the unified feature representation, a hierarchical graph structure is constructed to obtain an information hierarchy graph; relational reasoning is performed on the information hierarchy graph to obtain a structural relationship matrix; based on the structural relationship matrix and the pre-stored professional knowledge base, the relationship confidence matrix is ​​verified to obtain the relationship confidence matrix; the structural relationship matrix and the relationship confidence matrix are integrated to obtain structured domain feature information;

[0028] S5. Perform data verification on the structured domain feature information to obtain a verification result matrix; and generate normalized data based on the verification result matrix.

[0029] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0030] S11, receiving the original image data of the professional field form, calculating the grayscale histogram distribution characteristics of each area of ​​the image; based on the grayscale histogram distribution characteristics, dynamically adjusting the local contrast parameters, performing adaptive histogram equalization processing, and generating an enhanced image;

[0031] S12. Based on the enhanced image, extract an image feature map through a convolutional neural network; calculate the attention score of each region in the image feature map; based on the attention score, construct a regional importance distribution, and generate a regional importance weight matrix representing the importance of each region;

[0032] S13, based on the weight values ​​in the region importance weight matrix, resampling different regions in the enhanced image using different resolution sampling rates to generate a multi-resolution image sequence including a predetermined number of different resolution versions;

[0033] S14. Based on the multi-resolution image sequence, an anisotropic diffusion filtering algorithm with adaptive threshold is used to suppress image noise, and an edge enhancement algorithm based on local gradient is used to enhance the edge features of the image to generate a multi-resolution optimized image sequence.

[0034] In one embodiment of the present application, the process of performing adaptive histogram equalization processing is specifically as follows: enhancing the image Ie(x, y) = φ(x, y) · [H(x, y) - μ(x, y)] / σ(x, y) + μ(x, y); wherein H(x, y) = ∑[w(i, j) · I(i, j)] / (∑w(i, j)), H(x, y) is the local weighted histogram mean, w(i, j) = exp(-||(i, j) - (x, y)|| 2 / λ 2 ) is the spatial weight function, I(i, j) is the original image; φ(x, y) = 1 +η·log(1 + var(x, y)) is the adaptive enhancement coefficient, var(x, y) is the local variance; μ(x, y) is the local mean, σ(x, y) is the local standard deviation; λ is the spatial weight decay parameter; η is the enhancement strength parameter; x, y are pixel coordinates; the local statistics are calculated in an r×r window, and r is the local window radius.

[0035] The specific process of different resolution sampling rates is as follows: sampling rate R(x, y) = Rmin + (Rmax - Rmin)·[1 - exp(-θ·W(x, y))]; where W(x, y) is the regional importance weight; Rmin is the minimum sampling rate, and Rmax is the maximum sampling rate; θ is the sampling rate adjustment parameter; the sampled image coordinates (x', y') = (x·R(x, y), y·R(x, y)); x, y are the original image coordinates; the sampling rate R(x, y)∈[Rmin, Rmax], and increases monotonically with regional importance; θ controls the steepness of the sampling rate change.

[0036] The specific process of generating the regional importance weight matrix is ​​as follows: regional importance weight W(x, y) = α·A(x, y) + β·S(x, y) + γ·C(x, y); where A(x, y) = softmax(V(x, y)·K(x, y) T / sqrt(d))·V(x, y), A(x, y) is the spatial attention score, V(x, y) is the regional visual feature, K(x, y) is the regional query feature, and d is the feature dimension; S(x, y) = exp(-||P(x, y) - P(x', y')|| 2 / σ 2), S(x, y) is the spatial smoothness, P(x, y) is the position coordinate, σ is the smoothing factor; C(x, y) = ∑(w_i·f_i(x, y)) / (∑w_i), C(x, y) is the channel importance, f_i(x, y) is the feature response of the i-th channel, w_i is the channel weight; α, β, γ are weight coefficients, and α+β+γ=1; x, y are image coordinate positions; the weight coefficients are obtained by gradient descent optimization.

[0037] Anisotropic diffusion algorithm: Image optimization function I'(x, y, t) = div[c(x, y, t) · ▽I(x, y, t)]; where c(x, y, t) = exp(-||▽I(x, y, t)|| 2 / k 2 ) is the diffusion coefficient, k is the gradient threshold parameter; ▽I is the image gradient, div is the divergence operator; the edge enhancement term E(x, y) = I(x, y) + λ·sign(▽ 2 I)·|▽I|,▽ 2 I is the Laplace operator; t is the iteration time step; x, y are pixel coordinates; λ is the edge enhancement strength parameter.

[0038] This embodiment achieves robust image preprocessing for the common diverse imaging quality problems in medical forms (such as uneven illumination, low contrast, noise interference, etc.) by constructing an adaptive image enhancement and optimization process. First, adaptive histogram equalization is performed through dynamically adjusted local contrast parameters, which effectively improves the visibility of details in different regions; second, the regional importance weights calculated by the attention mechanism are used to accurately locate and enhance key information areas (such as test values, reference values, etc.), reducing the interference of irrelevant backgrounds; third, through the importance-based multi-resolution sampling strategy, while maintaining high resolution in key areas, the secondary areas are downsampled, reducing the consumption of computing resources; finally, the anisotropic diffusion filter with adaptive threshold is combined with the edge enhancement based on local gradient, while suppressing noise, the clarity of the text edge is maintained, providing a high-quality image foundation for subsequent optical character (OCR) recognition. This embodiment is particularly suitable for the characteristics of medical forms, and can effectively handle the enhancement needs of various visual elements such as numbers, text, and table lines in the form.

[0039] According to one aspect of the present application, step S12 is further:

[0040] S121. Based on the enhanced image, hierarchical visual features are extracted through a multi-scale convolutional network; based on the hierarchical visual features, a feature pyramid is constructed to generate a multi-scale feature tensor containing information of different resolutions;

[0041] S122, based on the multi-scale feature tensor, using an adaptive feature aggregation algorithm to calculate the attention weight of each scale layer; performing weighted fusion on the multi-scale feature tensor and the corresponding attention weight to generate a feature aggregation tensor;

[0042] S123. Based on the feature aggregation tensor, a spatial attention network is constructed, and an attention feature map is generated through dual modeling of channel attention and position attention.

[0043] S124, based on the attention feature map, using a preconfigured probability graph model to model the spatial dependency relationship to obtain a spatial dependency relationship model; based on the spatial dependency relationship model, using a variational inference method to calculate the conditional probability distribution between regions to generate a spatial dependency matrix;

[0044] S125. Based on the spatial dependency matrix and attention feature map, a regional importance evaluation network is constructed, and a regional scoring matrix is ​​generated through comprehensive calculation of multi-dimensional indicators;

[0045] S126. Based on the regional score matrix, the scores are stratified using an adaptive threshold segmentation algorithm, and smoothed using density estimation to generate a final regional importance weight matrix.

[0046] This embodiment achieves accurate attention allocation by constructing a multi-level regional importance evaluation mechanism for the information value differences of different regions in the medical form. First, through the multi-scale convolutional network and feature pyramid structure, the multi-resolution expression of image features is realized, and the hierarchical visual features from the overall layout to the detailed text are accurately captured; secondly, the adaptive feature aggregation algorithm is used to dynamically adjust the weights of different scale layers, effectively balancing the importance of global structure and local details; thirdly, the dual attention modeling mechanism simultaneously considers the information distribution of channel dimension and spatial dimension, and accurately locates the key areas in the form; then, the probabilistic graph model is used to deeply model the spatial dependency relationship, accurately grasping the correlation strength between different regions; finally, through the smoothing processing of adaptive threshold segmentation and density estimation, a stable and continuous importance weight distribution is generated. This embodiment is particularly suitable for the differentiated processing needs of different types of information such as test items, numerical values, reference values, etc., and provides important regional guidance information for subsequent identification and analysis.

[0047] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0048] S21, based on the multi-resolution optimized image sequence, extracting the visual features of the text area through the first convolutional neural network to obtain a text feature vector; and at the same time extracting the spatial layout information of the text through the second convolutional neural network to obtain a spatial position feature vector;

[0049] S22, calculating the similarity matrix between the text feature vector and the spatial position feature vector; based on the similarity matrix, performing normalization processing through a softmax function, generating a text-space attention map representing the correlation strength between the text region and the spatial region;

[0050] S23, weightedly fusing the text-space attention map with the text feature vector and the spatial position feature vector to obtain a fused feature vector; inputting the fused feature vector into a preconfigured text recognition model for decoding to generate an initial text recognition result; based on the initial text recognition result, calculating the confidence score of each recognition result and constructing a confidence matrix;

[0051] S24. Based on the initial text recognition result, extract the image blocks corresponding to the areas below the preset threshold in the confidence matrix to obtain the low-confidence area image blocks; based on the low-confidence area image blocks, use the preconfigured high-precision text recognition model to re-recognize to obtain the re-recognition result; replace the low-confidence area image blocks with the re-recognition result to obtain the updated text recognition result.

[0052] In one embodiment of the present application, the process of generating a text-space attention map is as follows: attention score At(i, j) = ρ·exp(Qt(i)·Ks(j) T / τ)·M(i, j); where Qt(i) is the text feature query vector, Ks(j) is the spatial feature key vector; M(i, j) = sigmoid(Wm·[Qt(i);Ks(j)] + bm) is the multimodal fusion gating, Wm is the fusion weight matrix, and bm is the bias term; τ is the temperature parameter, which controls the smoothness of the attention distribution; ρ is the attention normalization factor, which ensures that ∑jAt(i, j) = 1; i is the text sequence position index, and j is the spatial feature position index; [;] represents the feature concatenation operation.

[0053] The preconfigured high-precision text recognition model is specifically as follows: confidence score C(x)=χ*·Pc(x) + (1-χ*)·Ps(x); where Pc(x) = max(softmax(Wc·fx + bc)) is the character-level confidence, fx is the feature vector; Ps(x)= exp(-D(x, T) / ν) is the structural similarity, D(x, T) is the minimum distance to the template library T; χ* is the balancing factor; Wc is the classification weight matrix, bc is the bias vector; ν is the similarity scale parameter; x is the text fragment to be recognized.

[0054] This embodiment achieves high-precision text positioning and recognition for the complex layout and strict data association requirements in medical forms by constructing a text recognition framework that integrates spatial position information. First, the text visual features and spatial layout features are extracted respectively through a two-stream convolutional neural network, and the association mapping between text content and spatial position is established through the attention mechanism, which effectively solves the serial recognition problem of traditional OCR when processing table structures; secondly, the weighted feature fusion mechanism based on the text-spatial attention map improves the recognition accuracy of specific areas in the form (such as the name of the test item and the numerical area); thirdly, through hierarchical confidence evaluation and adaptive re-identification strategy, the low-confidence area is refined and the recognition accuracy of difficult-to-recognize text (such as medical terms and special symbols) is improved. This embodiment is particularly suitable for complex layout characteristics, can accurately understand the spatial relationship between data items, and provide a reliable text basis for subsequent semantic analysis.

[0055] According to one aspect of the present application, step S23 is further:

[0056] S231, using an adaptive grid partitioning algorithm to divide the text-space attention map into blocks, determining the optimal segmentation threshold according to the distribution characteristics of the attention value, and generating an attention area grid;

[0057] S232, based on the attention distribution of the attention area grid, calculating the weight coefficients of the text feature vector and the spatial position feature vector; based on the weight coefficients, using a weighted average method to perform feature fusion and generate a fusion feature matrix;

[0058] S233, based on the fusion feature matrix, calculating the local statistical features of the fusion features in each grid; based on the local statistical features and the pre-stored character feature template library, performing template matching to generate a character similarity matrix;

[0059] S234, constructing a spatial constraint relationship graph between characters based on the character similarity matrix; calculating an optimal character combination scheme through a graph matching algorithm based on the spatial constraint relationship graph, and generating a text combination scheme matrix;

[0060] S235, based on the text combination scheme matrix, using a preconfigured conditional random field model to perform probability modeling on the character sequence, calculating the conditional probability distribution of different text combinations, and generating a sequence probability matrix;

[0061] S236. Based on the sequence probability matrix, a dynamic programming algorithm is used to find the optimal text sequence path to obtain an initial text recognition result; based on the initial text recognition result, a confidence score of each recognition result is calculated to generate a confidence matrix.

[0062] In one embodiment of the present application, the process of dividing the text-space attention map into blocks is specifically as follows: determining the grid weight Gr(i, j) = Ψ(i, j)·exp(-Dt(i, j) / ε)·Ns(i, j); wherein Ψ(i, j) = ∑[At(x, y)·Δ(x, y∈Bij)] / (|Bij|) is the average attention value in the grid unit Bij, At(x, y) is the attention map value, and Δ(·) is the indicator function; Dt(i, j) = ||Ct(i) - Ct(j)|| 2 is the grid center distance, Ct(i) is the centroid coordinate of grid i; Ns(i, j) = exp(-|Wi - Wj| / κ) is the grid similarity, Wi is the eigenvector of grid i; ε is the distance decay parameter; κ is the similarity adjustment parameter; i, j are grid indexes; |Bij| represents the number of pixels in the grid unit.

[0063] The specific process of probabilistic modeling is as follows: character sequence probability P(Y|X) = ∏[ψt(yt)·φt(yt,yt-1)·ωt(yt,X)]; where ψt(yt) = softmax(Wp·ht + bp) is the character output probability, ht is the hidden state vector; φt(yt,yt-1) = exp(Wt·[e(yt);e(yt-1)] / υ) is the transition probability, e(·) is the character embedding vector; ωt(yt,X) = sigmoid(Wa·[ht;ct] + ba) is the attention weight, ct is the context vector; Wp, Wt, Wa are weight matrices, bp, ba are bias vectors; υ is the transfer temperature parameter; X is the input feature sequence, Y is the output character sequence; t is the sequence position index, and ∏ is the product operator.

[0064] This embodiment achieves high-precision text recognition and organization by constructing a topology-aware text sequence decoding mechanism for strict text layout and association requirements in medical forms. First, an adaptive grid division algorithm is used to accurately divide the attention map into blocks, and the topological structure of the text area is maintained through attention-weighted centroid calculation, effectively avoiding the text breakage problem caused by traditional fixed grid division; secondly, based on the feature fusion strategy of attention distribution, the adaptive integration of text features and spatial features is realized, and the correspondence between text and position in the form is accurately captured; thirdly, through the deep matching of local statistical features and pre-stored character templates, combined with the global optimization of the spatial constraint relationship graph, the recognition accuracy of professional terms and special symbols is improved; finally, the conditional random field model is used to probabilistically model the character sequence, and the optimal text path is found through the dynamic programming algorithm, realizing the accurate recognition and reliability evaluation of continuous text in medical forms. This embodiment is particularly suitable for standardized layout requirements, can accurately restore the spatial organization structure of the text, and provide a reliable text basis for subsequent semantic understanding.

[0065] According to one aspect of the present application, step S231 is further:

[0066] S2311. Receive the text-space attention map At, calculate the global statistical features of the attention map, including the attention mean μ and standard deviation σ, determine the threshold in an adaptive manner based on μ+0.5σ, and generate an adaptive threshold matrix Th.

[0067] S2312, receiving the adaptive threshold matrix Th and the text-space attention map At, dividing the attention map into 8×8 grid units, calculating the attention weighted average for each grid unit, and generating a grid attention matrix Ga.

[0068] S2313, receiving the grid attention matrix Ga and the text-space attention map At, calculating the attention-weighted centroid for each grid unit, obtaining the centroid coordinates by performing attention-weighted averaging on the pixel coordinates in the grid, and generating the grid centroid matrix Gc.

[0069] S2314, receiving the grid centroid matrix Gc and the grid attention matrix Ga, and calculating the topological relationship between the grid units. For any two grid units, the topological association strength is calculated based on their centroid distance and attention value similarity, where the distance adopts the Euclidean distance and is attenuated by an exponential function, and the attention similarity is calculated by the normalized difference to generate a topological relationship matrix Tp.

[0070] S2315, receiving the topological relationship matrix Tp and the text-space attention map At, extracting features for each grid unit. First, calculate the local statistical features, including the average attention, the standard deviation of attention, the maximum attention value, and the total attention value; then extract the spatial distribution features, including the first-order moment and the second-order central moment of the horizontal and vertical directions of the attention weight; finally, extract the topological relationship features, including the mean, standard deviation, and significant connection ratio of the topological weight, and generate the grid feature matrix Gf.

[0071] S2316, receiving the grid feature matrix Gf and the topological relationship matrix Tp, performing weighted combination of features according to the topological relationship through the topologically aware feature aggregation operation, maintaining the local structural information, and generating the attention region grid Gr.

[0072] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0073] S31. Based on the multi-resolution optimized image sequence, the visual transformer network is used to extract multi-level visual features, and feature aggregation is performed to generate a visual feature tensor representing the image content.

[0074] S32, based on the updated text recognition results, encoding the text sequence through the text transformer network to generate a text feature tensor representing the text semantics;

[0075] S33, calculating the cosine similarity between the visual feature tensor and the text feature tensor, and generating a cross-modal alignment matrix representing the correspondence between the visual feature and the text feature through normalization processing;

[0076] S34. Based on the correspondence in the cross-modal alignment matrix, the visual feature tensor and the text feature tensor are weightedly fused to generate a unified feature representation.

[0077] In one embodiment of the present application, the process of generating a cross-modal alignment matrix is ​​specifically as follows: alignment matrix M(i, j) = π·exp(-D(i, j) / γ)·Z(i, j); where D(i, j) = ||Sv(i) - St(j)|| 2 / ξ+α·[1 - cos(Sv(i), St(j))] is the feature distance metric, Sv(i) is the visual feature vector, and St(j) is the text feature vector; Z(i, j) = sigmoid(Wz·[Sv(i); St(j)] + bz) is the modality fusion gating; cos(·, ·) is the cosine similarity; π is the double random matrix after iterative optimization of the Sinkhorn algorithm; γ is the temperature parameter; ξ is the distance scale factor; α is the similarity weight; i is the visual feature index, j is the text feature index; Wz is the fusion weight matrix, bz is the bias vector, and sigmoid( ) is the activation function.

[0078] This embodiment achieves efficient multimodal information integration by designing a cross-modal feature alignment and fusion mechanism for the deep correlation features between text content and visual layout in medical forms. First, a visual transformer network is used to hierarchically encode multi-resolution image sequences, and the visual feature dependencies at different scales are captured through a multi-head self-attention mechanism, effectively extracting the hierarchical visual representation of the form; secondly, a text transformer network is used to semantically encode the recognition results, and the context association of the text sequence is established through a position-aware attention mechanism, accurately grasping the semantic relationship between medical terms and numerical values; thirdly, by designing a cross-modal alignment mechanism based on cosine similarity and combining it with a graph attention network based on structural similarity, an accurate mapping relationship between visual features and text features is established, solving the alignment bias problem of traditional methods when processing heterogeneous information; finally, through an adaptive weighted feature fusion strategy, complementary enhancement of information is achieved while maintaining modal specificity, providing a unified feature representation for subsequent structured analysis. This embodiment is particularly suitable for multimodal characteristics and can make full use of the complementary information of visual layout and text semantics.

[0079] According to one aspect of the present application, step S31 is further:

[0080] S311, receiving a multi-resolution optimized image sequence Io, dividing the image into a fixed-size block sequence through an image block encoder, and embedding position coding information to generate an image block feature sequence Ps.

[0081] S312, receiving the image block feature sequence Ps, calculating the attention weights between blocks through the multi-head self-attention module, combining the feedforward neural network to process the attention output, and generating the self-attention feature matrix Sa.

[0082] S313, receiving the self-attention feature matrix Sa, using a cross-scale transformer encoder to build the association between feature blocks of different resolutions, and generating a cross-scale feature matrix Cs through multi-level information transmission.

[0083] S314, receiving the cross-scale feature matrix Cs, extracting global semantic information through the hierarchical transformer decoder, and generating a hierarchical decoding feature matrix Hd in combination with the multi-layer attention mechanism.

[0084] S315, receiving the hierarchical decoding feature matrix Hd and the cross-scale feature matrix Cs, using a bidirectional transformer network to fuse the encoding and decoding features, and generating a fused feature matrix Mf through residual connection.

[0085] S316, receiving the fused feature matrix Mf, performing feature alignment and aggregation through the feature normalization transformer network, and using adaptive pooling processing to generate the final visual feature tensor Fv.

[0086] In one embodiment of the present application, the process of generating a cross-scale feature matrix is ​​specifically as follows: feature update F'l(p) = Fl(p) + ∑[G(s)·T(Fl+s(N(p)))]; wherein Fl(p) is the feature of position p at level l; G(s) is the scale offset weight function, s∈{-1, 0, 1} represents the adjacent level; T(·) = MLP(LayerNorm(·)) is the feature transformation function, MLP is a multilayer perceptron; N(p) is the corresponding area of ​​position p at the adjacent level; T(·) contains a parameter matrix WT and a bias vector bT; l is the feature level index, p is the feature position coordinate; the feature dimension remains consistent at all levels.

[0087] The process of fusing encoding and decoding features is as follows: fusion feature M(p) = ∑[αi·Fi(p) + βi·Bi(p)]; where Fi(p) is the forward feature and Bi(p) is the backward feature; αi, βi are adaptive weight coefficients, calculated by αi = softmax(Wa·Fi(p)), βi = softmax(Wb·Bi(p)); Wa, Wb are weight matrices; p is the feature position; i is the feature level index; the feature dimensions are unified through linear transformation.

[0088] This embodiment achieves efficient multi-scale information encoding for the complex visual structure features in medical forms by constructing a hierarchical visual feature extraction framework. First, an image block encoder is used to convert the image into a fixed-size block sequence, and the spatial position information is maintained through position encoding, which effectively solves the memory bottleneck problem of traditional methods when processing large-size images; secondly, a multi-head self-attention mechanism is used to deeply model the image block features, accurately capture the long-range dependencies between different regions, and enhance the ability to understand the overall structure of the form; thirdly, the association between features of different resolutions is established through a cross-scale transformer encoder, and a unified expression from global layout to local details is achieved by combining a multi-level information transmission mechanism; then, a bidirectional transformer network is used to fuse encoding and decoding features, and the original visual information is effectively maintained through residual connections; finally, feature normalization and adaptive pooling are used to generate a unified visual feature representation. This embodiment is particularly suitable for multi-level visual elements and can effectively process different types of visual information such as table lines, text, and graphics.

[0089] According to one aspect of the present application, step S32 is further:

[0090] S321. Receive the text recognition result Rt, tokenize the text through a word segmentation encoder, combine position encoding and semantic type encoding, and generate a text encoding sequence Te.

[0091] S322, receive the text encoding sequence Te, use the multi-head self-attention transformer layer to calculate the attention distribution between tokens, process the attention output through the feedforward network, and generate the attention feature matrix At.

[0092] S323. Receive the attention feature matrix At, build the association between different semantic levels through the hierarchical transformer encoder, and use the multi-layer cross attention mechanism to generate the hierarchical semantic matrix Ht.

[0093] S324, receiving the hierarchical semantic matrix Ht, using a deep transformer decoder to integrate long-range semantic dependencies, and generating a semantic dependency matrix Dt through a bidirectional feature propagation network.

[0094] S325, receiving the semantic dependency matrix Dt and the hierarchical semantic matrix Ht, fusing the local and global semantic features through a cascaded transformer network, and using residual connections to generate a fused semantic matrix Mt.

[0095] S326, receiving the fused semantic matrix Mt, performing dimension alignment and feature aggregation through the feature aggregation transformer network, and combining the adaptive pooling operation to generate the final text feature tensor Ft.

[0096] In one embodiment of the present application, the process of generating a semantic dependency matrix is ​​specifically as follows: dependency strength D(i, j) = ρ·tanh(Wd·[si; sj; si-sj; si⊕sj] + bd); wherein si, sj are node semantic vectors; tanh() is the hyperbolic tangent function; ⊕ is the vector outer product operation; Wd is the dependency weight matrix, bd is the bias vector; ρ is the normalization factor; [;] represents vector concatenation; i, j are sequence position indexes; and the dependency tree is constructed by the maximum spanning tree algorithm.

[0097] This embodiment achieves high-quality text feature representation by designing a deep text semantic encoding framework for the particularity of professional terms and numerical expressions in medical forms. First, through the word segmentation encoder combined with position encoding and semantic type encoding, accurate marking of different types of text such as medical terms, numerical values, units, etc. is achieved, effectively solving the limitations of traditional word segmentation methods in processing professional vocabulary; secondly, a multi-head self-attention transformer layer is used to deeply model the token sequence, and the long-distance semantic dependencies are accurately captured through the attention mechanism, which improves the ability to understand complex medical expressions; thirdly, the hierarchical transformer encoder is used to establish the association between different semantic levels, and combined with the multi-layer cross-attention mechanism, a hierarchical representation from the word level to the semantic level is achieved; then, through the deep transformer decoder and the bidirectional feature propagation network, the perception of the context is enhanced, and the relationship between the test items and their corresponding values ​​is accurately understood; finally, a cascade transformer network and feature aggregation operation are used to generate a unified text feature representation. This embodiment is particularly suitable for strict professional expression requirements and can accurately understand and express complex medical information.

[0098] According to one aspect of the present application, step S33 is further:

[0099] S331, receiving the visual feature tensor Fv and the text feature tensor Ft, using projection transformation to map the two feature tensors to a common semantic space, and generating a visual semantic vector Sv and a text semantic vector St.

[0100] S332, receiving the visual semantic vector Sv and the text semantic vector St, calculating the cosine similarity between the two vectors, and adjusting the similarity distribution by the temperature scaling factor to generate an initial similarity matrix Si.

[0101] S333, receiving the initial similarity matrix Si and the visual semantic vector Sv, constructing a local neighborhood graph, calculating the structural similarity between nodes through the graph attention network, and generating a structural similarity matrix Ss.

[0102] S334, receiving the initial similarity matrix Si and the structural similarity matrix Ss, calculating the comprehensive similarity through an adaptive weighted fusion algorithm, and performing non-maximum suppression to generate a fused similarity matrix Sf.

[0103] S335, receiving the fused similarity matrix Sf, using the optimal transmission algorithm to calculate the optimal matching relationship between cross-modal features, and generating a matching relationship matrix Rm.

[0104] S336, receiving the matching relationship matrix Rm and the fusion similarity matrix Sf, screening reliable matching pairs through a bidirectional consistency check, and performing normalization processing to generate the final cross-modal alignment matrix M.

[0105] In step S334, when calculating the comprehensive similarity, not only the information with strong correlation (high similarity) is focused on, but also some information with weak correlation (low similarity) is retained; the weak correlation information is retained through a soft threshold mechanism.

[0106] This embodiment achieves reliable feature mapping by constructing an accurate cross-modal alignment mechanism to meet the needs of deep integration of visual information and text information in medical forms. First, a projection transformation is used to map visual features and text features to a unified semantic space, and the similarity distribution is adjusted by a temperature scaling factor, which effectively solves the problem of inconsistent scales of features in different modalities; secondly, a local neighborhood graph is constructed using a graph attention network to accurately capture the structural similarity between nodes and enhance the ability to recognize related elements in the form; thirdly, the comprehensive similarity is calculated using an adaptive weighted fusion algorithm, combined with non-maximum suppression technology, to effectively filter noise matches and maintain the clarity of key information; then, the optimal transmission algorithm is used to calculate the optimal matching relationship between cross-modal features to ensure the accurate correspondence between visual elements and text content; finally, reliable matching pairs are screened through a two-way consistency check to generate a stable cross-modal alignment result. This embodiment is particularly suitable for strict information correspondence requirements and can accurately establish a mapping relationship between visual layout and text content.

[0107] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0108] S41. Based on the unified feature representation, a hierarchical clustering algorithm is used to perform grouping to obtain a grouping result; based on the grouping result, a hierarchical graph structure including test items, numerical values, units, and reference value nodes is constructed to generate an information hierarchy graph representing the hierarchical relationship of information;

[0109] S42, based on the information hierarchy graph, message passing and information aggregation are performed on the nodes in the graph through the graph neural network to obtain an updated node representation; based on the updated node representation, the association relationship and dependency degree between the nodes are inferred to generate a structural relationship matrix representing the structural relationship between the nodes;

[0110] S43, calling a pre-stored professional knowledge base to verify whether the structural relationship in the structural relationship matrix conforms to the professional knowledge, and obtaining a verification result; based on the verification result, calculating the credibility score of each relationship, and generating a relationship confidence matrix;

[0111] S44. Based on the confidence in the relationship confidence matrix, the structural relationships in the structural relationship matrix are screened and optimized to obtain structural relationships that meet the requirements; the structural relationships that meet the requirements are integrated into a standardized data structure to generate structured domain feature information.

[0112] In one embodiment of the present application, the process of constructing a hierarchical graph structure is specifically as follows: node association strength E(i, j) = β·H(i, j) + (1-β)·L(i, j); where H(i, j) = tanh(Wh·[hi;hj] + bh) is the high-level semantic association, hi is the semantic feature of node i; L(i, j) = σ(di, j)·exp(-||pi-pj|| 2 / μ) is the low-level structural association, di,j is the node type compatibility, pi is the node position vector; σ(·) is the structural compatibility function; β is the hierarchical balance factor; Wh is the semantic transformation matrix, bh is the bias vector; μ is the spatial scale parameter; i,j are the graph node indexes.

[0113] This embodiment achieves highly reliable semantic understanding and structured representation for the complex data associations and professional domain knowledge constraints in medical forms by establishing a hierarchical information structure parsing framework. First, the initial hierarchical structure is constructed using adaptive feature decomposition and spectral clustering methods, and the hierarchical dependencies between test items, values, units and other elements are accurately captured through semantic association analysis; secondly, the feature clusters are mapped to predefined medical concept nodes using a concept ontology mapping network, and a structured representation that conforms to the professional knowledge system is established through dynamic graph construction and edge weight learning; thirdly, based on a multi-hop reasoning network and hierarchical attention pooling, deep information transmission between nodes is achieved, and the implicit associations between test results are accurately inferred; finally, the necessity and sufficiency of the association are verified through a causal reasoning network, ensuring the logical consistency of structured information. This embodiment is particularly suitable for strict logical requirements and can accurately understand and maintain complex medical data associations.

[0114] According to one aspect of the present application, step S41 is further:

[0115] S411, receiving the unified feature representation Fu, dividing the feature space into semantic subspaces through an adaptive feature decomposition network, using a spectral clustering algorithm to calculate the hierarchical structure of feature clusters, and generating an initial hierarchical matrix Hi.

[0116] S412, receiving the initial hierarchical matrix Hi, identifying the hierarchical dependency relationship between feature clusters through the semantic association analysis module, and adopting a bottom-up iterative aggregation strategy to generate a hierarchical dependency matrix Hd.

[0117] S413, receiving the hierarchical dependency matrix Hd, constructing a concept ontology mapping network, mapping the feature clusters to predefined professional field concept nodes (such as test items, numerical values, units, reference values), and generating a concept mapping matrix Cm.

[0118] S414, receiving the concept mapping matrix Cm and the hierarchical dependency matrix Hd, creating the initial connection relationship between the nodes through the dynamic graph builder, and using the edge weight adaptive learning algorithm to generate the graph connection matrix Gc.

[0119] S415. Receive the graph connection matrix Gc, use the graph structure optimization network to prune and enhance the connection relationship, calculate the importance score of the edge through the graph attention mechanism, and generate the optimized graph structure matrix Go.

[0120] S416, receiving the optimized graph structure matrix Go and the concept mapping matrix Cm, integrating the node semantic information and the topological structure information through the hierarchical graph fusion network, and using the graph embedding technology to generate the final information hierarchical graph Gh.

[0121] In one embodiment of the present application, the graph structure optimization network is specifically as follows: edge weight update W'(i, j) = W(i, j) + Δ(i, j)·R(i, j); wherein Δ(i, j) = sigmoid(WΔ·[hi; hj] + bΔ) is the edge update strength, hi, hj are node features; R(i, j) = max(0, θ-min(k≠i, j)W(i, k)) is the sparsification regularization term; WΔ is the update weight matrix, bΔ is the bias vector; θ is the sparsification threshold; i, j, k are graph node indexes.

[0122] This embodiment achieves highly reliable semantic hierarchical representation for the complex data association structure in medical forms by constructing a hierarchical information organization framework driven by professional knowledge. First, an adaptive feature decomposition network is used to divide the unified feature space into different semantic subspaces, and the hierarchical structure of feature clusters is accurately calculated through the spectral clustering algorithm, which effectively solves the confusion problem of traditional methods when processing heterogeneous medical information; secondly, the semantic association analysis module is used to identify the hierarchical dependency between feature clusters from the bottom up, and the logical association between test items, values, units and other elements is accurately grasped; thirdly, the abstract features are mapped to specific medical concept nodes through the concept ontology mapping network, and the data organization structure that meets professional specifications is established by combining dynamic graph construction technology; then, the graph structure optimization network is used to dynamically adjust the connection relationship, and the important semantic links are highlighted through the graph attention mechanism, and redundant associations are filtered; finally, the hierarchical graph fusion network is used to integrate the node semantics and topological structure, and a hierarchical representation that meets both medical professional knowledge and data integrity is generated. This embodiment is particularly suitable for strict data specification requirements and can accurately reflect the professional association relationship between various types of medical data.

[0123] According to one aspect of the present application, step S42 is further:

[0124] S421. Based on the information hierarchy graph, calculate the context representation of each node in the graph through the self-attention mechanism; based on the information hierarchy graph, extract the local structural features of the node; based on the context representation and the local structural features, generate a node context feature matrix;

[0125] S422, constructing a heterogeneous relationship graph network based on the node context feature matrix and the information hierarchy graph; based on the heterogeneous relationship graph network, using a relationship-aware message passing algorithm to calculate the direct association strength between nodes and generate a direct association matrix;

[0126] S423, based on the direct association matrix and the node context feature matrix, simulating a multi-step information propagation process through a multi-hop reasoning network, calculating the transitive dependency relationship between nodes, and generating a transitive dependency matrix;

[0127] S424, based on the transitive dependency matrix, a hierarchical attention pooling algorithm is used to perform weighted aggregation on dependencies at different levels, and sparse constraints are used to remove noise associations to generate a hierarchical dependency matrix;

[0128] S425. Based on the hierarchical dependency matrix and the direct association matrix, a causal reasoning network is constructed, and the necessity and sufficiency of the association are verified by the counterfactual reasoning method to generate a causal relationship matrix;

[0129] S426. Based on the causal relationship matrix and the hierarchical dependency matrix, the comprehensive relationship score is calculated through an adaptive fusion network, and the relationship strength is calibrated to generate the final structural relationship matrix.

[0130] In one embodiment of the present application, the multi-step information propagation process is simulated by a multi-hop reasoning network as follows: k-hop relationship strength Rk(i, j) = ∑[λt·∏(m=1 to k)A(path(t, m))]; wherein A(p) = softmax(Wa·[hp;hq] + ba) is a single-step attention score, p is a path edge, hp, hq are node features connected by the edge; path(t, m) represents the m-step edge of the t-th path of length k; λt is the path importance weight, satisfying ∑λt = 1; Wa is an attention weight matrix, ba is a bias vector; i, j are the start and end node indexes; k is a hop count parameter.

[0131] This embodiment achieves high-precision structured analysis for complex logical relationships and implicit dependencies in medical forms by constructing a deep relational reasoning framework. First, the self-attention mechanism is used to calculate the contextual representation of each node, and the local structural features of the node are combined to effectively capture the contextual association between the test items and related data; secondly, the direct association strength between different types of nodes is accurately calculated through the heterogeneous relational graph network and the relation-aware message passing algorithm, which solves the deviation problem of traditional methods in dealing with heterogeneous relationships; thirdly, a multi-hop reasoning network is used to simulate multi-step information propagation, and through hierarchical attention pooling, remote dependencies are accurately identified, which improves the ability to understand complex medical indicator relationships; then, the causal reasoning network is used to verify the necessity and sufficiency of the association, and the counterfactual reasoning method is used to ensure the reliability and rationality of the reasoning results; finally, an adaptive fusion network is used to integrate various types of relationship information to generate a structured representation that accurately reflects the internal connection of medical data. This embodiment is particularly suitable for strict logical reasoning requirements and can accurately discover and verify potential associations between data.

[0132] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0133] S51, performing multi-dimensional verification of the format, value range and logical relationship of the data in the structured domain feature information according to preset professional data rules and value ranges, and generating a verification result matrix representing the verification results;

[0134] S52, organizing and packaging the data in the verification result matrix according to a standardized format to generate normalized data that can be used for subsequent processing.

[0135] This embodiment achieves highly reliable data quality assurance by implementing an intelligent data verification mechanism to meet the data accuracy and consistency requirements in medical forms. Multi-dimensional verification of data based on preset professional rules can effectively reduce the medical risks caused by data errors.

[0136] In another embodiment of the present application, step S31 may also be:

[0137] S31a, receiving a multi-resolution optimized image sequence Io, calculating local response features through a multi-scale feature decomposition network, extracting multi-scale context information in combination with spatial pyramid pooling, and generating a hierarchical visual feature matrix Hv.

[0138] S31b, receiving the hierarchical visual feature matrix Hv, using the adaptive attention module to calculate the importance distribution of the feature area, performing feature enhancement through a dynamic weight allocation mechanism, and generating an attention-enhanced feature matrix Av.

[0139] S31c, receive the attention enhancement feature matrix Av, build a cross-scale feature interaction network, realize the information exchange of features of different scales through bidirectional feature propagation, and generate a cross-scale interaction feature matrix Cv.

[0140] S31d, receive the cross-scale interaction feature matrix Cv, use the non-local attention mechanism to capture the long-range dependencies, calculate the global context representation through adaptive feature aggregation, and generate the global context feature matrix Gv.

[0141] S31e, receiving the global context feature matrix Gv and the cross-scale interaction feature matrix Cv, merging local details and global semantic information through the residual feature fusion network to generate a multi-scale fusion feature matrix Mv.

[0142] S31f, receive the multi-scale fusion feature matrix Mv, use the feature calibration network to align the feature distribution, and generate the final visual feature tensor Fv through adaptive normalization processing.

[0143] In another embodiment of the present application, step S32 may also be:

[0144] S32a, receiving the text recognition result Rt, extracting multi-granularity language units by hierarchical tokenization, and generating an initial text representation matrix It by position-aware embedding coding.

[0145] S32b, receiving the initial text representation matrix It, constructing a hierarchical semantic parsing network, extracting semantic features at different levels through progressive semantic understanding, and generating a multi-level semantic feature matrix Ht.

[0146] S32c, receiving a multi-level semantic feature matrix Ht, using a semantic dependency analysis module to establish dependencies between text units, and generating a semantic dependency feature matrix Dt through a structured attention mechanism.

[0147] S32d, receiving the semantic dependency feature matrix Dt, integrating the long-distance semantic associations through the context enhancement network, and using the bidirectional feature propagation mechanism to generate the context enhancement feature matrix Ct.

[0148] S32e, receiving the context-enhanced feature matrix Ct and the multi-level semantic feature matrix Ht, using an adaptive feature selection network to filter key semantic information, and generating a core semantic feature matrix Kt through importance-weighted aggregation.

[0149] S32f, receive the core semantic feature matrix Kt, perform distribution alignment and dimension unification through the feature normalization network, and use adaptive pooling operation to generate the final text feature tensor Ft.

[0150] According to one aspect of the present application, a medical form data recognition method based on OCR and MLLM includes the following steps:

[0151] Step 1: Collect the image to be processed, construct a prompt, and input it into MLLM for image classification. Construct a prompt based on the image category and text set, and use a large language model to obtain medical form data recognition information; use MLLM to perform more fine-grained classification of medical forms to more accurately determine the information to be extracted.

[0152] Step 2: Get the information to be extracted according to the image category and perform preprocessing such as cropping on the image to obtain a sub-image. According to different types of medical forms, dynamically adjust the list of extracted information to ensure that the extracted content is more accurate and relevant; according to the information to be extracted, further crop and preprocess the original image to generate multiple sub-images, each of which corresponds to a specific extraction task.

[0153] Step 3: Use OCR to recognize the subgraph to obtain a text set. Construct a prompt based on the coordinates of the center point of the text line in the third set and the OCR content recognition results, including setting a task description, providing contextual information or display related to the task, setting the input format, setting the placeholder, setting the output format requirements, inputting the constructed prompt into a large language model for context understanding and position relationship analysis, extracting the expense detail recognition information, and outputting the results.

[0154] In one embodiment of the present application, a user captures an image of a medical checkup form by taking a photo or using a scanner. These images can be photos or scans of paper documents. Based on the content of the captured image, an appropriate prompt is constructed to describe the basic information of the image and the operations that need to be performed. The constructed prompt and the image are input into a pre-trained multimodal large language model (MLLM). MLLM is able to process text and image data simultaneously and classify images. MLLM outputs the category of the image, such as "Doppler ultrasound checkup form", "epidermal cell count checkup form", etc.

[0155] According to the category of the image, the system determines the key information that needs to be extracted from the examination form of this type from the predefined information extraction rule library. For example, for a Doppler ultrasound examination form, the information that may need to be extracted includes "ultrasound prompts" and so on. According to the information to be extracted, the original image is cropped and only the part containing the key information is retained. The cropped sub-image is preprocessed by contrast enhancement and other preprocessing operations to improve the accuracy of OCR recognition. After preprocessing, a clear sub-image containing key information is obtained.

[0156] Use optical character recognition (OCR) technology to perform text recognition on the preprocessed sub-images and generate a text set. Based on the image category, the information to be extracted, and the OCR results, a new prompt statement (Prompt) is constructed. This Prompt describes in detail the specific information that needs to be extracted from the OCR results. The new Prompt and the OCR results are input into the MLLM model together. The MLLM model performs semantic understanding and information extraction on the input text and outputs structured key information. The system parses the reasoning results of the MLLM model, extracts the required key information, and outputs it in a structured format. This information can be further used for storage, analysis, or other subsequent processing.

[0157] In this embodiment, through image classification, the system can determine the specific type of the checklist, thereby selecting appropriate preprocessing steps and information extraction rules, which helps to improve the pertinence and accuracy of subsequent processing. Preprocessing operations such as cropping and contrast enhancement of images according to picture categories can remove irrelevant information, highlight key content, and improve the accuracy of OCR recognition. Use OCR technology to perform text recognition on preprocessed sub-images to generate high-quality text sets, providing a basis for subsequent semantic understanding and information extraction. Build Prompt and input the OCR results into the MLLM large model for semantic understanding and information extraction. Through the powerful semantic understanding ability of MLLM, the system can accurately extract the required key information from the OCR results. Parse the reasoning results of MLLM, output structured key information, and generate structured data output for subsequent processing and storage.

[0158] According to one aspect of the present application, a medical form image recognition system based on OCR and MLLM includes: an image acquisition module: used to acquire the medical form image to be processed; an MLLM image classification module: classifies the collected image to determine its specific type (such as a blood routine test form, a urine routine test form, etc.); a correction processing module: performs operations such as cropping and contrast enhancement on the image according to the classification result to generate a sub-image; an OCR recognition module: uses OCR technology to recognize text in the sub-image and generates a text set; an MLLM text extraction module: uses MLLM to perform semantic understanding and information extraction on the text set; an output module: generates structured data output for subsequent processing or storage.

[0159] This embodiment achieves efficient and accurate parsing of medical examination forms by combining OCR technology and multimodal large language model (MLLM). Through image classification and preprocessing, OCR recognition is ensured to be performed under optimal conditions, thereby improving the accuracy of text recognition; using the powerful semantic understanding and information extraction capabilities of MLLM, complex medical terms and structured data can be parsed more accurately; the automated processing flow reduces the workload of manual entry and improves the speed and efficiency of data processing; the system can handle examination forms in various formats provided by different medical institutions, with high flexibility and versatility.

[0160] According to one aspect of the present application, a medical form data recognition system based on OCR and MLLM includes:

[0161] at least one processor; and,

[0162] a memory communicatively connected to at least one of the processors; wherein,

[0163] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the medical form data recognition method based on OCR and MLLM described in any of the above embodiments.

[0164] The present invention realizes the full process automation from image input to standardized data output by constructing an end-to-end medical form intelligent recognition system. First, through the multi-level image enhancement and optimization mechanism, the imaging quality problem of the diversity of medical forms is solved, providing a high-quality image foundation for subsequent recognition; secondly, the text recognition framework integrating spatial position information accurately captures the correlation characteristics of text content and position in the form, breaking through the limitations of traditional OCR in processing complex forms; thirdly, based on the transformer-based cross-modal feature alignment and fusion mechanism, the deep integration of visual layout information and text semantic information is realized, providing strong support for accurate understanding of the form content; fourthly, through the hierarchical information structure analysis framework, combined with professional medical knowledge, deep reasoning is performed to ensure the professionalism and logic of data association; finally, the intelligent data verification and correction mechanism is adopted to realize the automatic guarantee of data quality. The present invention is particularly adapted to the characteristics of medical forms, and can efficiently process different types of medical documents such as test reports and prescriptions, improve the efficiency and accuracy of medical data digitization, and provide reliable technical support for the construction of medical informatization.

[0165] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A medical form data recognition method based on OCR and MLLM, characterized in that: The steps include: S1, receiving the original image data of the professional field form, performing contrast enhancement, and obtaining an enhanced image; Based on the enhanced image, the regional weights are calculated to obtain a regional importance weight matrix; based on the regional importance weight matrix, the resolution is adjusted to obtain a multi-resolution image sequence; Performing edge optimization processing on a multi-resolution image sequence to obtain a multi-resolution optimized image sequence; S2. Based on the multi-resolution optimized image sequence, feature vectors are extracted to obtain text feature vectors and spatial position feature vectors; based on the text feature vectors and spatial position feature vectors, attention associations are constructed to obtain a text-spatial attention map; based on the text-spatial attention map, text recognition is performed to obtain an initial text recognition result and a confidence matrix; based on the initial text recognition result and the confidence matrix, low confidence areas are extracted and optimized to obtain an updated text recognition result; S3, visually encode the multi-resolution optimized image sequence to obtain a visual feature tensor; Perform text encoding on the updated text recognition results to obtain a text feature tensor; calculate feature alignment based on the visual feature tensor and the text feature tensor to obtain a cross-modal alignment matrix; Based on the cross-modal alignment matrix, the visual feature tensor and the text feature tensor are fused to obtain a unified feature representation; S4. Based on the unified feature representation, a hierarchical graph structure is constructed to obtain an information hierarchy graph; relationship reasoning is performed on the information hierarchy graph to obtain a structural relationship matrix; Based on the structural relationship matrix and the pre-stored professional knowledge base, the relationship confidence matrix is ​​verified; the structural relationship matrix and the relationship confidence matrix are integrated to obtain structured domain feature information; S5. Perform data verification on the structured domain feature information to obtain a verification result matrix; Generate normalized data based on the verification result matrix.

2. The medical form data recognition method based on OCR and MLLM according to claim 1, characterized in that: Step S1 is further as follows: S11, receiving the original image data of the professional field form, calculating the grayscale histogram distribution characteristics of each area of ​​the image; based on the grayscale histogram distribution characteristics, dynamically adjusting the local contrast parameters, performing adaptive histogram equalization processing, and generating an enhanced image; S12, based on the enhanced image, extracting the image feature map through the convolutional neural network; Calculate the attention score of each region in the image feature map; Based on the attention scores, the regional importance distribution is constructed to generate a regional importance weight matrix that represents the importance of each region; S13, based on the weight values ​​in the region importance weight matrix, resampling different regions in the enhanced image using different resolution sampling rates to generate a multi-resolution image sequence including a predetermined number of different resolution versions; S14. Based on the multi-resolution image sequence, an anisotropic diffusion filtering algorithm with adaptive threshold is used to suppress image noise, and an edge enhancement algorithm based on local gradient is used to enhance the edge features of the image to generate a multi-resolution optimized image sequence.

3. The medical form data recognition method based on OCR and MLLM according to claim 2 is characterized in that: Step S2 is further as follows: S21, based on the multi-resolution optimized image sequence, extracting the visual features of the text area through the first convolutional neural network to obtain a text feature vector; and at the same time extracting the spatial layout information of the text through the second convolutional neural network to obtain a spatial position feature vector; S22, calculating the similarity matrix between the text feature vector and the spatial position feature vector; based on the similarity matrix, performing normalization processing through a softmax function, generating a text-space attention map representing the correlation strength between the text region and the spatial region; S23, weighted fusion of the text-space attention map with the text feature vector and the spatial position feature vector to obtain a fused feature vector; The fused feature vector is input into the preconfigured text recognition model for decoding to generate the initial text recognition result; based on the initial text recognition result, the confidence score of each recognition result is calculated to construct a confidence matrix; S24. Based on the initial text recognition result, extract the image blocks corresponding to the areas below the preset threshold in the confidence matrix to obtain the low-confidence area image blocks; based on the low-confidence area image blocks, use the preconfigured high-precision text recognition model to re-recognize to obtain the re-recognition result; replace the low-confidence area image blocks with the re-recognition result to obtain the updated text recognition result.

4. The medical form data recognition method based on OCR and MLLM according to claim 3 is characterized in that: Step S3 is further as follows: S31. Based on the multi-resolution optimized image sequence, the visual transformer network is used to extract multi-level visual features, and feature aggregation is performed to generate a visual feature tensor representing the image content. S32, based on the updated text recognition results, encoding the text sequence through the text transformer network to generate a text feature tensor representing the text semantics; S33, calculating the cosine similarity between the visual feature tensor and the text feature tensor, and generating a cross-modal alignment matrix representing the correspondence between the visual feature and the text feature through normalization processing; S34. Based on the correspondence in the cross-modal alignment matrix, the visual feature tensor and the text feature tensor are weightedly fused to generate a unified feature representation.

5. The medical form data recognition method based on OCR and MLLM according to claim 4 is characterized in that: Step S4 is further as follows: S41. Based on the unified feature representation, a hierarchical clustering algorithm is used to perform grouping to obtain a grouping result; based on the grouping result, a hierarchical graph structure including test items, numerical values, units, and reference value nodes is constructed to generate an information hierarchy graph representing the hierarchical relationship of information; S42, based on the information hierarchy graph, message passing and information aggregation are performed on the nodes in the graph through the graph neural network to obtain an updated node representation; based on the updated node representation, the association relationship and dependency degree between the nodes are inferred to generate a structural relationship matrix representing the structural relationship between the nodes; S43, calling a pre-stored professional knowledge base to verify whether the structural relationship in the structural relationship matrix conforms to the professional knowledge, and obtaining a verification result; Based on the verification results, the credibility score of each relationship is calculated to generate a relationship confidence matrix; S44, based on the confidence in the relationship confidence matrix, screening and optimizing the structural relationships in the structural relationship matrix to obtain structural relationships that meet the requirements; Integrate the structural relationships that meet the requirements into a standardized data structure to generate structured domain feature information.

6. The medical form data recognition method based on OCR and MLLM according to claim 5, characterized in that: Step S5 is further as follows: S51, performing multi-dimensional verification of the format, value range and logical relationship of the data in the structured domain feature information according to preset professional data rules and value ranges, and generating a verification result matrix representing the verification results; S52, organizing and packaging the data in the verification result matrix according to a standardized format to generate normalized data that can be used for subsequent processing.

7. The medical form data recognition method based on OCR and MLLM according to claim 6, characterized in that: Step S12 is further as follows: S121. Based on the enhanced image, hierarchical visual features are extracted through a multi-scale convolutional network; based on the hierarchical visual features, a feature pyramid is constructed to generate a multi-scale feature tensor containing information of different resolutions; S122, based on the multi-scale feature tensor, using an adaptive feature aggregation algorithm to calculate the attention weight of each scale layer; performing weighted fusion on the multi-scale feature tensor and the corresponding attention weight to generate a feature aggregation tensor; S123. Based on the feature aggregation tensor, a spatial attention network is constructed, and an attention feature map is generated through dual modeling of channel attention and position attention. S124, based on the attention feature map, using a preconfigured probability graph model to model the spatial dependency relationship to obtain a spatial dependency relationship model; Based on the spatial dependency model, the variational inference method is used to calculate the conditional probability distribution between regions and generate the spatial dependency matrix. S125. Based on the spatial dependency matrix and attention feature map, a regional importance evaluation network is constructed, and a regional scoring matrix is ​​generated through comprehensive calculation of multi-dimensional indicators; S126. Based on the regional score matrix, the scores are stratified using an adaptive threshold segmentation algorithm, and smoothed using density estimation to generate a final regional importance weight matrix.

8. The medical form data recognition method based on OCR and MLLM according to claim 6, characterized in that: Step S23 is further as follows: S231, using an adaptive grid partitioning algorithm to divide the text-space attention map into blocks to generate an attention area grid; S232, calculating weight coefficients of text feature vectors and spatial position feature vectors based on the attention distribution of the attention area grid; Based on the weight coefficient, the weighted average method is used to fuse features and generate a fusion feature matrix; S233, based on the fusion feature matrix, calculating the local statistical features of the fusion features in each grid; based on the local statistical features and the pre-stored character feature template library, performing template matching to generate a character similarity matrix; S234, constructing a spatial constraint relationship diagram between characters based on the character similarity matrix; Based on the spatial constraint relationship graph, the optimal character combination scheme is calculated through the graph matching algorithm to generate a text combination scheme matrix; S235, based on the text combination scheme matrix, using a preconfigured conditional random field model to perform probability modeling on the character sequence, calculating the conditional probability distribution of different text combinations, and generating a sequence probability matrix; S236. Based on the sequence probability matrix, a dynamic programming algorithm is used to find the optimal text sequence path to obtain an initial text recognition result; based on the initial text recognition result, a confidence score of each recognition result is calculated to generate a confidence matrix.

9. The medical form data recognition method based on OCR and MLLM according to claim 6, characterized in that: Step S42 is further as follows: S421. Based on the information hierarchy graph, calculate the context representation of each node in the graph through the self-attention mechanism; based on the information hierarchy graph, extract the local structural features of the node; based on the context representation and the local structural features, generate a node context feature matrix; S422, constructing a heterogeneous relationship graph network based on the node context feature matrix and the information hierarchy graph; based on the heterogeneous relationship graph network, using a relationship-aware message passing algorithm to calculate the direct association strength between nodes and generate a direct association matrix; S423, based on the direct association matrix and the node context feature matrix, simulating a multi-step information propagation process through a multi-hop reasoning network, calculating the transitive dependency relationship between nodes, and generating a transitive dependency matrix; S424, based on the transitive dependency matrix, a hierarchical attention pooling algorithm is used to perform weighted aggregation on dependencies at different levels, and sparse constraints are used to remove noise associations to generate a hierarchical dependency matrix; S425. Based on the hierarchical dependency matrix and the direct association matrix, a causal reasoning network is constructed, and the necessity and sufficiency of the association are verified by the counterfactual reasoning method to generate a causal relationship matrix; S426. Based on the causal relationship matrix and the hierarchical dependency matrix, the comprehensive relationship score is calculated through an adaptive fusion network, and the relationship strength is calibrated to generate the final structural relationship matrix.

10. A medical form data recognition system based on OCR and MLLM, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the medical form data recognition method based on OCR and MLLM as described in any one of claims 1 to 9.

Images