Medical document semantic entity recognition method and system based on spatial semantic enhancement

Abstract

The invention provides a medical document semantic entity recognition method and system based on spatial semantic enhancement, and the method comprises the steps: detecting a text region in a medical document through an OCR technology, and extracting text content and bounding box coordinate information thereof; based on the spatial geometrical relationship between the textboxes, calculating the distance and angle between the textboxes, fusing the spatial semantic features with the text hiding state, and constructing a spatial-temporal feature matrix; through a multi-head attention mechanism, deep interaction is carried out on the fused geometric and text features, and the understanding of the model on the semantic relationship between the texts is further strengthened; and inputting the enhanced features into a classification head for accurately identifying and classifying semantic entities. According to the scheme, the spatial semantic information is introduced, so that the model can better understand the spatial layout of the text in the document, and the accuracy and robustness of multi-category semantic entity recognition in the complex medical document are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing, computer vision and medical information processing, and in particular to a semantic entity recognition method and system for understanding and analyzing multimodal medical documents, specifically including using spatial semantic information to enhance the recognition capability of semantic entities in medical documents, and is suitable for structured processing and semantic analysis of a variety of complex medical documents including test orders, outpatient records, etc. Background Art

[0002] With the rapid development of medical informatization, more and more medical documents are stored in digital form, including test reports, outpatient records, electronic medical records, etc., which contain a large amount of important patient information, diagnosis results, treatment plans and other key medical data. However, these medical documents have various formats, complex layouts, and exist in various forms such as handwritten text and scanned documents, which brings huge challenges to information extraction and automatic processing.

[0003] Traditional optical character recognition (OCR) technology can be used to extract text information from documents, but it is often difficult to effectively understand the semantic structure and spatial relationship of the document by relying solely on text extracted by OCR. Especially in medical documents, the spatial layout between different text areas is crucial to understanding the content of the document. For example, the various test results in a test report are often associated with specific spatial positions, and different categories of semantic entities, such as patient information, test items, and results, need to be accurately distinguished and associated. Traditional text content-based entity recognition methods often ignore the spatial relationship between text boxes when processing these multimodal information, resulting in reduced recognition accuracy.

[0004] Most existing research focuses on semantic analysis and entity recognition in the field of natural language processing, or on OCR technology in the field of computer vision. However, how to effectively combine the semantic information of the text with its spatial location information in the document remains an urgent problem to be solved. Especially in medical documents, due to their highly structured and standardized characteristics, the joint processing of spatial information and text content can greatly improve the accuracy and robustness of semantic entity recognition.

[0005] Therefore, there is an urgent need for a semantic entity recognition method that can comprehensively consider text content and spatial layout information to process these complex medical documents and realize the automatic extraction and analysis of key medical information. This is of great significance for improving the efficiency and accuracy of medical data processing and assisting the construction of intelligent medical systems. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a multimodal medical document understanding solution that combines spatial semantic information and semantic features, aiming to solve the problem of insufficient accuracy of semantic entity recognition in the prior art due to ignoring the geometric relationship between text boxes (such as distance and angle). By introducing a geometric enhancement module, the spatial information between text boxes is fused with the pre-trained semantic features of the existing model, thereby improving the recognition performance on complex layouts and small sample data sets, especially improving the model's ability to classify complex entities in medical documents, and overcoming the limitation of traditional models that underutilize spatial information.

[0007] Specifically, the present invention provides the following technical solutions:

[0008] In one aspect, the present invention provides a method for semantic entity recognition of medical documents based on spatial semantic enhancement, the method comprising the following steps:

[0009] S1. Dataset creation: annotate medical document images from multiple sources, including medical entity data such as patient information, examination items, and diagnosis results, to ensure that the location and semantics are accurate; expand the dataset through data enhancement, improve data diversity, and obtain medical document images;

[0010] S2, extracting text content in the medical document image through optical character recognition, and obtaining the position coordinates of the text box;

[0011] S3, sending the medical document image, the position coordinates of the text box output by S2, and the text content into a document understanding pre-training model (such as LayoutXLM) to obtain text features;

[0012] S3, spatial semantic information extraction: based on the position coordinates of the text boxes output by S2, the spatial relationship between the text boxes is calculated, the spatial relationship includes the relative distance and angle between the text boxes, and the spatial relationship constitutes spatial semantic information, which is used to reflect the layout and mutual relationship of the text boxes in the document;

[0013] S4, fusion of text features and spatial semantic information: combining the text features generated by OCR in S2 with the spatial semantic information to form a spatiotemporal feature matrix; the fusion method is: projecting the spatial semantic information through a multi-layer perceptron (MLP) so that it is expressed in the same feature space as the text features, thereby achieving effective fusion of spatial layout information and text content;

[0014] S5. Input the spatiotemporal feature matrix after the fusion of text features and spatial semantic information into the multi-head attention module, use the multi-head attention module to calculate the dependency between text boxes, and highlight the key semantic entities and ignore irrelevant information by allocating attention weights;

[0015] S6. Semantic entity recognition: The text features processed by the multi-head attention module are classified through the classifier, the semantic category of each text box is identified, and its corresponding confidence is output.

[0016] Optionally, in step S1, the text and position extraction includes:

[0017] Use OCR technology to preprocess documents and preprocess medical document images to improve the effect of OCR detection, including image denoising, contrast enhancement and size scaling;

[0018] Use the DBNet model to detect text areas in the document and generate the boundary coordinates (upper left and lower right corner coordinates) of each text box. This step ensures that all text blocks can be accurately identified;

[0019] The CRNN model is used to perform character recognition on the detected text box content and convert the text in the image into readable text. The model combines convolutional neural network (CNN) for feature extraction and recurrent neural network (RNN) for sequence prediction;

[0020] The final output is the text content and its corresponding text box position coordinates, which will be used in the subsequent semantic and spatial semantic feature extraction steps.

[0021] Optionally, in step S2, the semantic and spatial semantic feature extraction includes:

[0022] Use the LayoutXLM pre-trained model to semantically encode the content of each text block and generate the corresponding text semantic vector. This model combines text and layout information to better understand the spatial structure of the document.

[0023] Optionally, in step S3, feature fusion includes:

[0024] According to the boundary coordinates of the text boxes, the geometric relationship between the text boxes is calculated. Specifically, it includes: calculating the coordinates of the center points of the text boxes, calculating the Euclidean distance between the center points of each text box, generating a distance matrix, and calculating the relative angles between the text boxes to generate an angle matrix;

[0025] Normalize the distance and angle matrices to ensure consistency of spatial semantic features across different document sizes or formats;

[0026] The extracted semantic features and the normalized spatial semantic features are output as input for feature fusion in subsequent steps.

[0027] Optionally, in step S4, feature fusion includes:

[0028] Construct a multi-head attention mechanism, through which the semantic features of each text block are weighted with its spatial semantic features to generate fused features. The multi-head attention mechanism allows the model to capture the relationship between different features at different attention heads;

[0029] Using the self-attention mechanism in the Transformer architecture, the attention weights between each text box and other text boxes are calculated. These weights are based on the relationship between semantic features and spatial semantic features;

[0030] According to the calculated attention weights, the text features and spatial semantic features are weighted to generate a fused feature representation. This fusion can capture the spatial and semantic interaction between text blocks;

[0031] The fused features weighted by the attention mechanism are output as the input of the final classification model.

[0032] Optionally, in step S5, entity identification and classification includes:

[0033] The fused features are input into a multi-layer perceptron (MLP) and further processed through several fully connected layers. Each layer uses a nonlinear activation function (such as ReLU) to enhance the representation ability of the model.

[0034] During the model training process, the cross entropy loss function is used to optimize the model parameters. This loss function is suitable for classification tasks and can effectively handle the classification problem of multi-category entities;

[0035] The model is trained to classify different semantic entities in medical documents. The classification results include patient information, diagnosis results, test reports, etc.

[0036] The final classification result is output, which indicates the semantic category corresponding to each text block in the document, and realizes the accurate recognition of the semantic entities of the document.

[0037] On the other hand, the present invention also provides a medical document semantic entity recognition system based on spatial semantic enhancement, comprising:

[0038] The OCR module is used to detect and recognize the text area in the medical document image, output the content of the text box and its corresponding spatial coordinate information to obtain the text features;

[0039] A spatial semantic information extraction module is used to calculate the relative distance and angle between text boxes based on the spatial coordinate information of the text boxes, and generate spatial semantic information;

[0040] The feature fusion module is used to fuse the text features obtained by the OCR module with the spatial semantic information to generate spatiotemporal features;

[0041] The attention mechanism module processes the fused spatiotemporal features through a multi-head attention mechanism, highlights important text box information, and generates the final semantic features;

[0042] The classification module is used to classify the processed semantic features and output the category and corresponding confidence of each text box.

[0043] In another aspect, the present invention further provides an electronic device, comprising:

[0044] processor;

[0045] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the above-mentioned method for extracting and generating semantic entity data of medical documents are implemented.

[0046] On the other hand, a computer-readable storage medium is provided, in which program code is stored. The program code can be called by a processor to execute the steps of the above-mentioned method for extracting and generating semantic entity data of medical documents.

[0047] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0048] In view of the problems of insufficient utilization of spatial information, insufficient extraction of semantic features, and lack of layout complexity processing capabilities in existing medical document understanding models, the present invention proposes a multimodal medical document understanding method that combines geometric relationships. This solution effectively solves the problem of low semantic entity recognition accuracy caused by ignoring the geometric relationships of text boxes (such as angles and distances), especially showing stronger robustness in complex document layouts and small sample data sets. By mining the spatial features between text boxes, the spatial relationships between different entities can be accurately captured, which improves the recognition and classification capabilities of complex semantic entities in medical documents and provides high-quality feature input for medical intelligent document processing systems.

[0049] This medical solution not only shows higher accuracy in semantic entity recognition, but also significantly expands the adaptability and generalization ability of the model by combining geometric relationships with semantic information. This technical solution can effectively improve the processing efficiency and accuracy of medical documents, provide a high-quality data foundation for intelligent applications such as automated processing of medical information, medical record classification, and patient information extraction, and provide important support for the development of intelligent diagnosis and treatment systems and automatic information extraction algorithms in the field of medical document processing.

[0050] This solution can be applied to multiple medical-related fields such as automated data processing of electronic medical record systems, medical document classification and information extraction. It can provide effective data support for medical intelligent systems and related algorithms, help the intelligent and precise development of medical document processing, and promote the advancement of medical information processing and analysis technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a flow chart of a method provided by an embodiment of the present invention;

[0053] Figure 2 is a flow chart of obtaining pre-trained semantic features provided by an embodiment of the present invention;

[0054] Figure 3 is a flow chart of multimodal feature fusion of medical documents provided by an embodiment of the present invention;

[0055] Figure 4 It is a block diagram of a multimodal medical document semantic entity recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0057] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0058] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0059] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0061] In order to solve the technical problem that multimodal emotion recognition in the prior art is not accurate enough, the embodiment of the present invention provides a multimodal emotion recognition method and device. The implementation scheme of the method is as follows:

[0062] On the one hand, a multimodal emotion recognition method is provided, such as Figure 1 As shown, the method includes:

[0063] S1. Use OCR technology to extract text content and its location information from medical documents. This step detects text boxes in the document, generates the boundary coordinates of each text box, and obtains the corresponding text content and spatial position;

[0064] S2, using the pre-trained model to encode the semantic features of the extracted text, and combining it with the image information to generate pre-trained text features;

[0065] S3, calculating the geometric relationship between the text boxes, including relative angles and distances, to form spatial semantic features for describing the document layout;

[0066] S4, fuses text features and spatial semantic features through a multi-head attention mechanism to generate a comprehensive feature representation that captures the complex interaction of semantic and spatial relationships in the document;

[0067] S5. Input the learned feature vector into the fully connected layer for classification, and finally achieve accurate recognition and classification of semantic entities in the document, which is especially suitable for medical documents with complex layout.

[0068] In the embodiment of the present invention, the working process of S1 is as follows:

[0069] S11, load the OCR model:

[0070] Use the DBNet model for text detection and the CRNN model for text recognition. First, load the DBNet model for text region detection in the document through open source implementation or trained model, and load the CRNN model for character recognition in the detected text region.

[0071] S12, document data processing:

[0072] Input the medical document image into the DBNet model loaded by S11. The DBNet model will detect the text area in the document and generate the boundary coordinates of each text box (such as the upper left corner and lower right corner coordinates). Then, these text boxes are input into the CRNN model for character recognition and the corresponding text content is extracted.

[0073] S13, feature extraction:

[0074] The output of the DBNet model is the location information of each text box, while the CRNN model outputs the characters in each text box. The text box location is represented as a four-dimensional coordinate (x1, y1, x2, y2), and the text content is represented as a string. The combination of the two forms a structured text and location information pair.

[0075] S14, processing text and position:

[0076] The extracted text and text box coordinate information are combined into a matrix, where each row of the matrix represents a text box, including the text content and its boundary coordinates. This matrix not only provides semantic information of the text, but also provides spatial location information of the text in the document.

[0077] S15, return result:

[0078] Finally, the text content and its location information are returned. These text contents and their location information constitute a matrix, which serves as the input for subsequent multimodal medical document understanding tasks, including the basis for spatial semantic feature calculation and text feature extraction.

[0079] In order to address the problems that text features are difficult to process in complex layouts in multimodal medical document understanding and that their contribution is insufficient compared to geometric and visual information, an embodiment of the present invention provides a text feature extraction method combined with a geometric enhancement module. This method can improve the contribution of text modality in multimodal medical document understanding by combining text semantics with spatial semantic information, namely step S2.

[0080] In this embodiment, combined with Figure 2 As shown, the extraction process of pre-trained semantic features is mainly included in S2 and S3.

[0081] The more preferred workflow of S2 is:

[0082] S21, load the pre-trained model for text semantic encoding:

[0083] The extracted text is encoded with semantic features using a pre-trained Transformer model. The specific steps include: segmenting the medical document text obtained by OCR to generate the corresponding text sequence. Then, the text sequence is input into the LayoutXLM pre-trained model to generate text features. This process involves the pre-trained language model processing the text information and extracting deep semantic features.

[0084] S22, fusion combined with image information:

[0085] While generating text features, the visual encoder is used to extract features from the document image to generate visual features. The visual features can be processed by a convolutional neural network (such as ResNet or ResNeXt-FPN) to process the image information of the document and extract feature representations of local image blocks.

[0086] S23, text and image feature alignment:

[0087] In order to better combine text and image information, the text-image alignment task is used to align the bounding box of the text with the corresponding image area. In this way, the semantic information of the text and image is interactively modeled in the same space. The text-image alignment task specifically includes three subtasks:

[0088] S231, using masks to randomly cover part of the text content, requiring the pre-trained model to restore these contents so that the model knows the position of each mask on the page;

[0089] S232, randomly selecting some text lines, covering the corresponding image area on the medical document image, building a classification layer on the encoder output, predicting whether the text tag is covered, and calculating the binary cross entropy loss;

[0090] S233, predict whether the image and the text are from the same medical document page.

[0091] S24, multimodal information fusion:

[0092] Finally, text features, image features, and spatial semantic information are fused through a multimodal Transformer model. In this process, the Transformer model captures the interactive relationship between different modalities through a multi-head self-attention mechanism and generates a comprehensive feature representation. This fused multimodal feature can help the model better understand the content and layout relationship in the document.

[0093] In view of the high complexity of text feature fusion processing with image information in multimodal medical document understanding and the insufficient contribution of a single modality, this embodiment provides a text semantic encoding method combined with visual features. This method can significantly improve the contribution of text modality in multimodal medical document understanding by combining multimodal feature fusion of text semantics and image information, and enhance the model's ability to understand document content and layout.

[0094] In this embodiment, the workflow of S3 is as follows:

[0095] S31, extracting the location information of the text box:

[0096] The boundary coordinates (x1, y1, x2, y2) of each text box are obtained from the OCR result in step S1. These coordinates represent the position of the text box in the document and are used for subsequent spatial semantic feature calculation.

[0097] S32, normalized text box coordinates:

[0098] First, the bounding coordinates of the text box are normalized to ensure uniform scaling under different document sizes and layout structures. The specific normalization method is as follows:

[0099]

[0100] Among them, bbox is the boundary coordinate of the text box, the shape is [batch_size, seq_len, 4], ε is a small constant used to prevent division by zero errors, bbox norm The normalized text box coordinates are processed uniformly in different documents. batch_size represents the number of images in each training batch, and seq_len represents the longest text sequence length.

[0101] Through normalization, the deviation introduced by differences in document size and layout is eliminated, ensuring that the model handles all text boxes consistently during training and inference.

[0102] S33, the text box coordinates after projection normalization:

[0103] The normalized text box coordinates are projected into a high-dimensional space that is aligned with the hidden state dimension of the text features. The specific projection method is as follows:

[0104] bbox proj =W bbox ×bbox norm +b bbox

[0105] Among them, W bbox is the projection matrix, which can be obtained through training and learning, b bboxis the bias term, bbox proj It is a high-dimensional representation after projection, which contains the spatial features of the text box. The shape is the same as the text features obtained by the LayoutXLM pre-trained model.

[0106] This projection combines spatial information with textual features to capture both the spatial layout and semantic content of the document.

[0107] S34, calculate the center point of the text box:

[0108] According to the coordinates of the upper left corner and lower right corner of the text box, the center point of the text box is calculated as the reference point of the spatial semantic feature. The specific calculation method is as follows:

[0109]

[0110] Among them, bbox[:,:,:2] is the coordinate of the upper left corner of the text box, bbox[:,:,2:] is the coordinate of the lower right corner of the text box, centers is the center point of the text box, indicating the position of the text box, and the shape is [batch_size,seq_len,2].

[0111] S35, calculate the distance matrix:

[0112] The Euclidean distance between each pair of text boxes is calculated through the center point of the text box and normalized. The specific calculation method is as follows:

[0113]

[0114] Among them, D ij is the normalized Euclidean distance between text boxes i and j, is the maximum distance between all text boxes, which is used for normalization to ensure that the scale of the distance matrix is ​​not affected by the document size. (x i ,y i ) represents the horizontal and vertical coordinates of the center position of text box i, (x j ,y j ) represents the horizontal and vertical coordinates of the center position of text box j.

[0115] S36, calculate the angle matrix:

[0116] By calculating the relative direction of the center point of the text box, the angle matrix is ​​obtained. The formula is as follows:

[0117]

[0118] Among them, A ij is the relative angle between text boxes i and j, which is between 0 and 1 after normalization.

[0119] S37, fusion of spatial semantic features:

[0120] The distance matrix and the angle matrix are concatenated to form a complete spatial feature matrix. The formula is as follows:

[0121] spatial_features=MLP(concat(D,A))

[0122] Among them, D is the normalized distance matrix, A is the normalized angle matrix, and spatial_features is the spatial features processed by the multi-layer perceptron (MLP).

[0123] S38, return result:

[0124] Finally, the fused spatial features are combined with the text features to obtain the spatiotemporal feature matrix, which serves as the input for subsequent document semantic entity recognition.

[0125] In this embodiment, in order to accurately extract the text box position information and calculate the spatial semantic features, S3 provides a processing method for extracting spatial semantic features that combines the text box position, distance and angle features. By normalizing the text box coordinates, the deviation introduced by the document size difference is eliminated to ensure the consistency of all text boxes. At the same time, the spatial semantic features are quantified by calculating the distance and relative angle between the text boxes, and these features are fused through a multi-layer perceptron to form a complete spatial feature representation. After the spatial features are combined with the text features, the model's ability to understand the document layout is effectively improved, and the accuracy of semantic entity recognition is enhanced.

[0126] In the embodiment of the present invention, combined with Figure 3 As shown, the specific process of S4 is:

[0127] S41, input text features and spatial semantic features:

[0128] The text features obtained from S2 and the spatial semantic features (including distance and angle matrices) calculated from S3 are taken as input. These features are used to capture the complex interaction between the spatial relationship and semantic content between text blocks.

[0129] S42, application of multi-head attention mechanism:

[0130] Multi-Head Attention is one of the key components in the Transformer model. It captures contextual information at different positions in the input sequence through a parallel multi-head self-attention mechanism. The multi-head attention mechanism can not only enrich feature expression, but also effectively learn different types of relationships.

[0131] In the multi-head attention mechanism, there are three key matrices: Q (Query), K (Key), and V (Value). Q is the query vector, which represents the information to be found, similar to finding specific information in the entire input. K is the key vector, which represents the identification information in the entire input and helps the query vector find the corresponding information. V is the value vector, which represents the actual information obtained after finding a match, usually corresponding one-to-one with the key.

[0132] Q, K, and V are obtained by linearly transforming the input data. Assuming the input is a sequence X, after linear projection (linear layer), we get:

[0133] Q=XW Q

[0134] K=XW K

[0135] V=XW V

[0136] Where W Q , W K , W V is the parameter matrix that needs to be learned.

[0137] The text features and spatial semantic features are fused through a multi-head attention mechanism. The attention mechanism calculates the attention weight based on the semantic information of each text block (as the query Q) and its geometric relationship (as the key K and value V). The formula is as follows:

[0138]

[0139] Among them, Q is generated by text features and represents the semantic information of text content, K and V are generated by spatial semantic features and reflect the spatial relationship of text blocks (including distance and angle matrices), d k Indicates the dimension of the key.

[0140] The attention mechanism calculates the spatial interactions between text blocks and assigns different weights to different text blocks, enabling the model to focus on text blocks with higher importance in the document layout.

[0141] S43, generation of fusion features:

[0142] The output of the multi-head attention mechanism generates a feature representation that combines text features and spatial semantic features. This fusion feature not only considers the semantic information of the text, but also captures the spatial relationship between text blocks.

[0143] S44, concatenate original text features and fusion features:

[0144] In order to preserve the original semantic information of the text, the fusion features generated by multi-head attention are concatenated with the original text features in the dimension of the longest sequence length to form a comprehensive feature representation. This step combines content and spatial layout information through concatenation.

[0145] S45, generate the final representation of the concatenated features:

[0146] The final feature representation combines the semantic information of text content and the spatial information of geometric layout, which can better capture the complex interactions in the document and provide richer input for subsequent entity classification and recognition tasks.

[0147] In an embodiment of the present invention, S4 fuses text features with spatial semantic features through a multi-head attention mechanism to generate a comprehensive feature representation containing spatial and semantic information. Through the attention mechanism, the model can dynamically adjust the degree of attention to different text blocks according to the spatial layout and relative importance between text blocks, thereby capturing the complex semantics and spatial relationships in the document. During the fusion process, the model not only considers the text content, but also retains the spatial position and relative relationship of the text in the document. By splicing the attention-enhanced features with the original text features, a more comprehensive feature representation is formed. This combination improves the model's entity recognition ability in complex layout documents, and is particularly suitable for highly structured fields such as medical documents.

[0148] In the embodiment of the present invention, the working process of S5 is as follows:

[0149] S51, input fused feature representation:

[0150] The comprehensive representation of text features and spatial semantic features fused by the multi-head attention mechanism in step S4 is used as input. These features contain the semantic information and spatial position relationship of each text block in the document, which can reflect the complex layout and structure of the document.

[0151] S52, feature vector input fully connected layer:

[0152] The fused feature vector is input to the fully connected layer (FC layer), which is used to further process and transform the features and map the high-dimensional feature vector to the dimensions required for the classification task. The function of the fully connected layer is to integrate the features and perform linear transformation to provide an effective feature representation for the classification task.

[0153] S53, activation function processing:

[0154] The output of the fully connected layer is processed through a nonlinear activation function (such as ReLU) to enhance the model's expressiveness and enable it to learn more complex features. The activation function can capture the nonlinear relationship in the input features and help improve the accuracy of the classification task.

[0155] S54, Category:

[0156] The feature vector processed by the fully connected layer and the activation function is used for the classification task. The classifier divides the text block into different semantic entity categories, such as patient information, diagnosis results, test data, etc., based on the input features. This step calculates the probability of each category through the Softmax function and finally determines the category of the text block.

[0157] S55, output classification results:

[0158] Output the classification results of each text block in the document. Each text block is assigned a specific semantic entity label to form an accurate understanding and classification of the entire document. These classification results can be used for subsequent document processing tasks, such as information extraction, automated report generation, etc. At the same time, the confidence of each classification result is output.

[0159] In an embodiment of the present invention, step S5 further refines and processes the fused multimodal features by inputting them into a fully connected layer, and classifies them through a classifier, thereby ultimately achieving accurate recognition and classification of semantic entities in the document. The function of the fully connected layer is to map high-dimensional features into representations suitable for classification, and to enhance the nonlinear learning capability of the model through activation functions. The classification results can accurately identify various entities in the document, and are particularly suitable for medical documents with complex layouts, so that the model can effectively process highly structured and multimodal documents. The design of this step greatly improves the performance of the model in document semantic entity recognition tasks, especially in the medical field, where document structures are complex and information-intensive, and accurate entity classification is essential.

[0160] In addition, combined Figure 4 As shown, the solution of the present invention can also be implemented by means of a device or system, which may include a first acquisition module 401 for acquiring training data sets and test data sets; a second acquisition module 402 for acquiring text box position and text content data; a pre-training module 403 for extracting text features; a fusion module 404 for fusing text features and spatial semantic information; and a semantic entity recognition module 405 for identifying semantic entities of medical documents.

[0161] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0162] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A medical document semantic entity recognition method based on spatial semantic enhancement, characterized in that: The following steps are involved: S1. Dataset creation: annotate medical document images from multiple sources, cover medical entity data, and ensure that the location and semantics are accurate; expand the dataset through data enhancement, improve data diversity, and obtain medical document images; S2, extracting text content in the medical document image through optical character recognition, and obtaining the position coordinates of the text box; S3, sending the medical document image, the position coordinates of the text box output by S2, and the text content into the document understanding pre-training model to obtain text features; S3, spatial semantic information extraction: based on the position coordinates of the text boxes output by S2, the spatial relationship between the text boxes is calculated, the spatial relationship includes the relative distance and angle between the text boxes, and the spatial relationship constitutes spatial semantic information, which is used to reflect the layout and mutual relationship of the text boxes in the document; S4, fusion of text features and spatial semantic information: combining the text features generated by OCR in S2 with the spatial semantic information to form a spatiotemporal feature matrix; the fusion method is: projecting the spatial semantic information through a multi-layer perceptron so that it is expressed in the same feature space as the text features, thereby achieving effective fusion of spatial layout information and text content; S5. Input the spatiotemporal feature matrix after the fusion of text features and spatial semantic information into the multi-head attention module, use the multi-head attention module to calculate the dependency between text boxes, and highlight the key semantic entities and ignore irrelevant information by allocating attention weights; S6. Semantic entity recognition: The text features processed by the multi-head attention module are classified through the classifier, the semantic category of each text box is identified, and its corresponding confidence is output.

2. The method according to claim 1, characterized in that The step of making a data set comprises: Annotate medical documents from different sources, including patient information, examination items, diagnosis results, and drug information, ensuring that the semantic label and location annotation of each text box are accurate; The dataset contains multiple semantic entity categories, which at least include patient name, gender, diagnosis results, and examination items; Data enhancement is achieved through image rotation enhancement, image scaling and noise injection to improve the generalization ability of subsequent models. The specific implementation methods are as follows: Image rotation: By randomly rotating the original medical document image to simulate the deviation of different shooting angles or scanning angles in actual use, the model is ensured to have good robustness when processing texts in different directions; Image scaling: Randomly scale the original medical image to generate medical images of different sizes, improving the model's adaptability to diverse document resolutions; Noise injection: Inject random noise into the original medical image.

3. The method according to claim 1, characterized in that The optical character recognition in S1 specifically includes: Use DBNet for text detection and identify the bounding boxes of different text areas; The text area of ​​the identified bounding box is used to identify the text content through CRNN.

4. The method according to claim 1, characterized in that: The spatial semantic information includes the multi-dimensional spatial relationship between text boxes, specifically including: Distance matrix: represents the normalized Euclidean distance between text boxes, which is used to quantify the relative distance between each pair of text boxes; Angle matrix: represents the relative angle between every two text boxes, used to capture the directional characteristics of the text boxes; The above-mentioned spatial semantic information is projected through a multi-layer perceptron, so that the spatial semantic information and text features are jointly expressed in the same dimensional space.

5. The method according to claim 4, characterized in that The multi-layer perceptron receives spatial semantic information and text features output by a document understanding pre-training model, and projects them into the same high-dimensional feature space for subsequent joint processing.

6. The method according to claim 1, characterized in that The multi-head attention module calculates the dependencies between text boxes, dynamically allocates attention weights, highlights important text boxes, and achieves accurate semantic entity recognition.

7. The method according to claim 2, characterized in that: The classifier performs multi-category classification based on the softmax function, specifically including: The text features processed by the multi-head attention module are input into the classifier, which is used to identify the semantic entity category of each text box; The classification result includes the predicted category of each text box and its corresponding confidence. The confidence is used to measure the credibility of the model for the classification result. The higher the confidence, the more reliable the classification result. The classifier supports the recognition of multiple semantic entity categories; The output of the classifier is used to identify the final semantic entity category and provide basic data for subsequent medical information processing and automated analysis.

8. A medical document semantic entity recognition system based on spatial semantic enhancement, characterized in that: The system is used to execute the method described in any one of claims 1 to 7, and the system comprises: The OCR module is used to detect and recognize the text area in the medical document image, output the content of the text box and its corresponding spatial coordinate information to obtain the text features; A spatial semantic information extraction module is used to calculate the relative distance and angle between text boxes based on the spatial coordinate information of the text boxes, and generate spatial semantic information; The feature fusion module is used to fuse the text features obtained by the OCR module with the spatial semantic information to generate spatiotemporal features; The attention mechanism module processes the fused spatiotemporal features through a multi-head attention mechanism, highlights important text box information, and generates the final semantic features; The classification module is used to classify the processed semantic features and output the category and corresponding confidence of each text box.

Images