Text recognition method, device and equipment of medical image and storage medium

Abstract

The application discloses a medical image text recognition method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: obtaining a medical image to be recognized; classifying the medical image to obtain an image type of the medical image; obtaining recognition prompt information corresponding to the image type of the medical image; performing a text recognition task on the medical image based on the recognition prompt information by using a multi-modal large language model to obtain a text recognition result of the medical image; and in the case where the image type of the medical image meets a preset condition, generating a test recognition result set based on the text recognition result by using the large language model, wherein the test recognition result set comprises test item recognition results of at least one test item, and the test item recognition result of the test item comprises a test result of the test item and a test type to which the test item belongs. The method effectively improves the accuracy of text recognition of different image types.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for text recognition of medical images. Background Technology

[0002] With the application of computer technology in the medical field, the digital processing of medical data has become an important part of the construction of medical systems.

[0003] During medical procedures, a large number of medical images are generated. In current technology, medical personnel manually organize the medical information from these images into electronic data and store it in a medical information management system. However, this method of processing medical images is inefficient. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for text recognition of medical images. The technical solution provided by this application is as follows: According to one aspect of the embodiments of this application, a text recognition method for medical images is provided, the method comprising: Acquire the medical image to be identified; The medical images are classified to obtain their image types; Obtain recognition prompt information corresponding to the image type of the medical image, wherein the recognition prompt information is used to instruct a text recognition task for the medical image; The text recognition task is performed on the medical image using a multimodal large language model based on the recognition prompt information, to obtain the text recognition result of the medical image; When the image type of the medical image meets the preset conditions, a set of laboratory identification results is generated based on the text recognition results using a large language model. The set of laboratory identification results includes the identification results of at least one laboratory item, and the identification results of the laboratory item include the laboratory result of the laboratory item and the laboratory type to which the laboratory item belongs.

[0005] According to one aspect of the embodiments of this application, a text recognition device for medical images is provided, the device comprising: The first acquisition module is used to acquire the medical image to be identified; A classification module is used to classify the medical images to obtain the image type of the medical images; The second acquisition module is used to acquire recognition prompt information corresponding to the image type of the medical image, and the recognition prompt information is used to indicate the text recognition task for the medical image; The recognition module is used to perform the text recognition task on the medical image based on the recognition prompt information using a multimodal large language model, and obtain the text recognition result of the medical image; The generation module is used to generate a set of laboratory identification results based on the text recognition results using a large language model when the image type of the medical image meets preset conditions. The set of laboratory identification results includes the identification results of at least one laboratory item, and the identification results of the laboratory item include the laboratory result of the laboratory item and the laboratory type to which the laboratory item belongs.

[0006] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described text recognition method for medical images.

[0007] According to one aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described method for text recognition of medical images.

[0008] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being loaded and executed by a processor to implement the above-described text recognition method for medical images.

[0009] The technical solutions provided in this application have at least the following beneficial effects: By classifying the medical images to be recognized, the system determines the corresponding recognition prompts based on the image type. Then, a multimodal large-scale language model performs text recognition based on these prompts, yielding the text recognition result. This approach effectively improves the accuracy of text recognition for different image types by using a multimodal large-scale language model to perform different text recognition tasks according to the image type. Furthermore, when a medical image meets preset conditions, the large-scale language model processes the text recognition results output by the multimodal large-scale language model to generate a set of laboratory test results. This enables accurate extraction of information related to laboratory tests from medical images of the laboratory report type, contributing to increased system automation. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a computer system provided in one embodiment of this application; Figure 2 This is a flowchart of a text recognition method for medical images provided in one embodiment of this application; Figure 3 This is a schematic diagram of a laboratory report image provided in one embodiment of this application; Figure 4 This is a schematic diagram of a desensitized laboratory report image provided in one embodiment of this application; Figure 5 This is a block diagram of a text recognition device for medical images provided in one embodiment of this application; Figure 6 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0012] Please refer to Figure 1 This illustration shows a schematic diagram of a computer system provided in one embodiment of this application. The computer system may include: a terminal device 10 and a server 20.

[0013] Terminal device 10 is an electronic device with data computing, processing, and storage functions. In some embodiments, terminal device 10 is used as a client to run a target application that has text recognition requirements for medical images. Optionally, the target application may be an application that needs to be downloaded and installed, or it may be in the form of a webpage or an app; this application does not limit this. Optionally, the target application is an application that has text recognition requirements for medical images. For example, the target application is implemented as a medical information management system for recognizing, parsing, and managing medical information.

[0014] In some embodiments, the terminal device 10 may include, but is not limited to, at least one of the following: mobile phone, tablet computer, personal computer, vehicle terminal, smart wearable device, smart TV, smart voice interaction device, multimedia playback device, etc., and may also include other electronic devices, which are not limited in this application embodiment.

[0015] Server 20 is a computer system specifically designed to provide services, resources, or functions. In some embodiments, server 20 is used to provide background services to clients of a target application running on terminal device 10.

[0016] In some embodiments, the server may include, but is not limited to, at least one of the following: physical server, cloud server, edge server, server cluster, etc., and may also include other types of servers, which are not limited in this application embodiment.

[0017] Terminal device 10 and server 20 communicate with each other via a network. This network can be a wired network or a wireless network.

[0018] Please refer to Figure 2 This document illustrates a flowchart of a text recognition method for medical images according to an embodiment of this application. The execution entity for each step of this method can be a computer device; for example, the computer device could be... Figure 1 The terminal device 10 or server 20 in the computer system shown. The method may include at least one of the following steps (210-250): Step 210: Obtain the medical image to be identified.

[0019] Medical images refer to image data generated or used in medical activities. Medical images to be identified refer to medical images that require text recognition.

[0020] In some embodiments, medical images may include, but are not limited to, at least one of the following: laboratory report images, medical record images, prescription images, examination report images, medical images, etc., and may also include other medical images, which are not limited in this application embodiment.

[0021] Lab report images refer to images containing test results issued by the laboratory or testing department. Medical record images refer to images containing a patient's condition, treatment process, and medical procedures. Prescription images refer to images containing medication receipts issued by doctors. Examination report images refer to images containing diagnostic reports issued after a patient's examination. Medical images refer to images generated by medical imaging equipment for diagnostic purposes.

[0022] Optionally, the acquisition methods for medical images may include, but are not limited to, at least one of the following: shooting, scanning, screenshotting, generation by imaging equipment, etc., and may also include other acquisition methods, which are not limited in this application embodiment.

[0023] For example, medical images can be obtained by a user taking a picture of a medical record displayed on the screen of a medical information management system.

[0024] Step 220: Classify the medical images to obtain the image types of the medical images.

[0025] The image type of a medical image is used to distinguish images containing different medical information. Optionally, in this embodiment, the image type may include, but is not limited to, at least one of the following: laboratory report type, medical record type, prescription type, examination report type, medical image type, etc., and may also include other image types, which are not limited in this embodiment. It is understood that a laboratory report type medical image is a laboratory report image, a medical record type medical image is a medical record image, a prescription type medical image is a prescription image, an examination report type medical image is an examination report image, and a medical image type medical image is a medical image.

[0026] Medical information refers to information related to medical activities. In some embodiments, medical information may include, but is not limited to, at least one of the following: patient identity information, health status information, diagnostic information, laboratory test information, treatment information, medical management information, etc., and may also include other medical information, which is not limited in this application embodiment.

[0027] Patient identity information refers to the basic information used to identify or locate a patient's personal identity. Optionally, patient identity information may include, but is not limited to, at least one of the following: name, gender, date of birth, ID card number, medical card number, medical record number, contact number, home address, etc., and may also include other information related to the patient's personal identity. This application embodiment does not limit this.

[0028] Health status information refers to information reflecting a patient's current or past physical condition. Optionally, health status information may include, but is not limited to, at least one of the following: chief complaint, present medical history, past medical history, allergy history, family history, body temperature, blood pressure, pulse, etc., and may also include other information related to the patient's physical condition. This application embodiment does not limit this.

[0029] Diagnostic information is used to instruct medical personnel on disease judgments based on examination results and clinical manifestations. In some embodiments, diagnostic information may include, but is not limited to, at least one of the following: disease name, preliminary diagnosis, clinical diagnosis, final diagnosis, ICD (International Classification of Diseases) code, etc., and may also include other information related to disease judgment, which is not limited in this application embodiment.

[0030] Laboratory information refers to information generated during the testing process that relates to the test items and their results. Optionally, laboratory information may include, but is not limited to, at least one of the following: test item name, test result, reference range, abnormality indicator, etc., and may also include other information related to the test items and their results. This application embodiment does not limit this.

[0031] Treatment information refers to information related to the treatment measures taken by medical personnel for patients. Optionally, treatment information may include, but is not limited to, at least one of the following: drug name, dosage, usage, name of surgery, treatment plan, doctor's orders, etc., and may also include other information related to treatment measures, which are not limited in this application embodiment.

[0032] Medical management information refers to information related to medical business processes, administration, or medical structures. Optionally, medical management information may include, but is not limited to, at least one of the following: consultation time, department name, physician name, hospital name, changes in medical records, costs, and charging items, and may also include other medical management information, which is not limited in this embodiment.

[0033] In some embodiments, medical images and their type information are stored in a first storage space. Optionally, the first storage space may be a local storage system or a remote storage system.

[0034] In some embodiments, preprocessing operations are performed on the medical images to obtain preprocessed medical images. These preprocessed medical images are then used to classify the medical images as described above to determine their image types. Preprocessing refers to standardizing, enhancing, or optimizing the original medical images before they are recognized.

[0035] Optionally, the preprocessing operations may include, but are not limited to, at least one of the following: scaling, normalization, denoising, and demoiring, and may also include other preprocessing operations, which are not limited in this embodiment. Scaling refers to adjusting the size of the medical image to meet a specified size standard. Normalization refers to mapping the pixel values ​​of the medical image to a preset numerical range. Denoising refers to removing random noise generated by shooting, scanning, or compression in the medical image. Demoiring refers to removing periodic stripe interference generated by shooting an electronic screen or printing / scanning in the medical image.

[0036] In some embodiments, medical images are classified using an image classification model to obtain type information, which indicates the image type of the medical image. In this embodiment, the image classification model is an artificial intelligence (AI) model specifically designed for classifying medical images.

[0037] Optionally, the image classification model can be trained based on any of the following models: CNN (Convolutional Neural Network), ViT (Vision Transformer), Support Vector Machine (SVM), Decision Tree, Random Forest, K-Nearest Neighbor (KNN), etc., and can also be implemented based on other models. This application embodiment does not limit this.

[0038] In some embodiments, the training process of the image classification model is as follows: obtaining classification training samples, which include classification sample images and classification sample labels, wherein the classification sample labels are used to indicate the image type to which the classification sample images belong; classifying the classification sample images using the image classification model to obtain predicted type information, wherein the predicted type information is used to indicate the predicted image type of the classification sample images; and adjusting the parameters of the image classification model based on the classification sample labels and predicted type information to obtain the trained image classification model.

[0039] Optionally, the number of classification training samples can be multiple (two or more). Optionally, the parameters of the image classification model can be adjusted multiple times based on the classification training samples to obtain the trained image classification model.

[0040] Optionally, a first loss function value is determined based on the classification sample labels and prediction type information; based on the first loss function value, the parameters of the image classification model are adjusted to obtain the trained image classification model. Optionally, the calculation method of the first loss function value may include at least one of the following: cross-entropy loss function, weighted cross-entropy function, mean squared error loss function, etc., and may also include other calculation methods, which are not limited in this embodiment.

[0041] Step 230: Obtain the recognition prompt information corresponding to the image type of the medical image. The recognition prompt information is used to instruct the text recognition task for the medical image.

[0042] Recognition prompts are task instructions used to guide or constrain the text recognition process for medical text. Text recognition tasks for medical images refer to the process of detecting, extracting, recognizing, and structuring textual information within medical images.

[0043] In some embodiments, the content of the identification prompt information may include, but is not limited to, at least one of the following: task description information, text recognition rules, output specification information, etc., and may also include other content, which is not limited in this application embodiment.

[0044] Task description information refers to descriptive information used to explain the task objectives and recognition scope of this text recognition task. Optionally, task description information may include, but is not limited to, at least one of the following: recognition fields, recognition objects, task type, etc., and may also include other content, which is not limited in this embodiment.

[0045] For example, the task description information could be "extract laboratory items and their corresponding laboratory results from medical images".

[0046] Text recognition rules refer to a set of rules that constrain or limit the processing logic of text recognition in medical images. Optionally, text recognition rules may include, but are not limited to, at least one of the following: field matching rules, keyword constraint rules, regular expression rules, medical terminology dictionary matching rules, etc., and may also include other rules, which are not limited in this embodiment.

[0047] For example, text recognition rules may include "paragraphs containing the keyword 'diagnosis' are preferentially extracted as diagnostic information".

[0048] Output specification information refers to the constraint information that standardizes the output format or data type of the recognition results of medical images. Optionally, the content of the output specification information may include, but is not limited to, at least one of the following: output format, output field name, field order, data type, etc., and may also include other content, which is not limited in this embodiment. Optionally, the output format may be a structured format. For example, structured formats may include, but are not limited to, at least one of the following: JSON (JavaScript Object Notation), MsgPack (MessagePack), XML (Extensible Markup Language), YAML (YAML Ain't MarkupLanguage), etc., and may also include other structured formats, which are not limited in this embodiment.

[0049] For example, the output specification information can be as follows: { Test Type: Routine Blood Test "Test Item Name": "Hb", Test results: 120 Unit: g / L Reference range: 110-150 } In some embodiments, different image types correspond to different recognition prompts. That is, different methods are used to recognize medical images depending on the image type to which the medical image belongs.

[0050] In some embodiments, a set of recognition prompts is obtained, which includes recognition prompt information corresponding to at least one image type; based on the image type to which the medical image belongs, recognition prompt information corresponding to the image type of the medical image is obtained from the set of recognition prompts.

[0051] In some embodiments, recognition prompts corresponding to the image type of the medical image are generated based on the image type of the medical image. Optionally, recognition prompts corresponding to the image type of the medical image are generated using a first language model. The first language model is a language model with text understanding and generation capabilities. Optionally, the first language model can be implemented using any of the following: a pre-trained language model, an instruction-fine-tuned language model, a rule-enhanced language generation model, a lightweight text generation model, a large language model, etc., and can also be implemented in other ways, which are not limited in this embodiment.

[0052] Step 240: Using a multimodal large language model based on recognition prompts, perform a text recognition task on the medical image to obtain the text recognition result of the medical image.

[0053] A multimodal large language model refers to a model capable of understanding input data in multiple modalities (two or more). A modality refers to the form in which information is expressed or perceived. For example, a modality may include, but is not limited to, at least one of the following: text, image, speech, structured data, etc. In the embodiments of this application, the multimodal large language model has the ability to understand text (recognition prompts) and images (medical images).

[0054] In this embodiment, the identified prompt information serves as a prompt word for the multimodal large language model, guiding it to perform text recognition on medical images. Optionally, the multimodal large language model is obtained after fine-tuning a corpus in the medical field. In other words, the multimodal large language model possesses the ability to understand medical-related information.

[0055] In some embodiments, image features of the medical image are extracted; text features of the recognition prompt information are extracted; and based on the image features of the medical image and the text features of the recognition prompt information, the text recognition result of the medical image is obtained. The image features of the medical image refer to the high-dimensional semantic feature representation of the medical image.

[0056] In some embodiments, the multimodal large language model includes a visual encoder and a language model; image features of the medical image are extracted by the visual encoder; text features of the recognition prompt information are extracted by the language model; and the text recognition result of the medical image is obtained by the language model based on the image features of the medical image and the text features of the recognition prompt information.

[0057] A visual encoder is a neural network model used to convert pixel data of medical images into high-dimensional semantic representations. Optionally, the visual encoder can be implemented based on any of the following: CNN, ViT, hybrid coding structure, etc., or based on other structures or models, which are not limited in this application embodiment.

[0058] A language model is a neural network model used to perform semantic understanding of recognition prompts and to jointly infer text recognition results based on the image features of medical images and the text features of recognition prompts. Optionally, the language model can be implemented based on any of the following: Transformer architecture, decoder-based autoregressive model, encoder-decoder structure model, multimodal fusion language model, etc., or it can be implemented based on other structures or models. This application embodiment does not limit this.

[0059] Optionally, the language model fuses the image features of the medical image with the text features of the recognition prompts through a cross-modal attention mechanism, and generates the text recognition result of the medical image based on the fused features.

[0060] Step 250: If the image type of the medical image meets the preset conditions, a set of laboratory identification results is generated based on the text recognition results using a large language model. The set of laboratory identification results includes the identification results of at least one laboratory item. The identification results of the laboratory item include the laboratory result of the laboratory item and the laboratory type to which the laboratory item belongs.

[0061] A large language model refers to a deep neural network model that is pre-trained on large-scale training data and possesses strong semantic understanding and text generation capabilities. Optionally, the large language model was fine-tuned using medical corpus.

[0062] After a multimodal large language model performs text recognition on medical images of laboratory reports, the output text recognition results contain issues such as mismatches and disordered order of test items and results. Therefore, further processing of the text recognition results output by the multimodal large language model is required, namely, summarizing the test items and results in the text recognition results to generate a set of test recognition results. Optionally, the set of test recognition results can be implemented in at least one of the following formats: JSON format, list structure, array structure, etc., and can also be implemented in other structures, which are not limited in this embodiment.

[0063] Preset conditions refer to the conditions that trigger further processing of the output of the multimodal large language model. In other words, when the output of the multimodal large language model meets the preset conditions, the large language model is invoked to further process the output of the multimodal large language model, thereby obtaining a more accurate final text recognition result.

[0064] In some embodiments, the preset conditions may include at least one of the following: image type conditions, text recognition completeness conditions, field structure feature conditions, keyword matching conditions, data format matching conditions, etc., and may also include other conditions, which are not limited in this application embodiment.

[0065] Image type conditions refer to the conditions that the image type of the medical image must meet. Text recognition completeness conditions refer to the conditions that the overall recognition confidence of the medical image must meet. Field structure feature conditions refer to the conditions that the text recognition result contains elements conforming to a specified field format. Keyword matching conditions refer to the conditions that the text recognition result contains a specified keyword. Data format matching conditions refer to the conditions that the text recognition result contains numerical information conforming to a specified data format.

[0066] In some embodiments, the preset conditions include: the image type of the medical image is a laboratory report type.

[0067] For example, such as Figure 3 As shown, it illustrates a schematic diagram of a laboratory report image provided in one embodiment of this application. Figure 3 The lab report image 300 shown is obtained by photographing lab report 310; therefore, lab report image 300 contains lab report 310. After the multimodal large language model and the large language model perform text recognition on the lab report image, the resulting set of lab recognition results can be as follows: [ … { "Serial Number": "2", Test Item Name: Neutrophil Classification Test result: 47.6 Unit: "%" Reference range: 40-75 Test type: Complete blood count (CBC) }, { "Serial Number": "3", Test Item Name: Lymphocyte Differentiation Test result: 43.8 Unit: "%" Reference range: 20-50 Test type: Complete blood count (CBC) }, { "Serial Number": "4", Test Item Name: Monocyte Differential Diagnosis Test result: 4.9 Unit: "%" Reference range: 3-10 Test type: Complete blood count (CBC) } … ] The above only shows the identification results of test item 311. The identification results of other test items are similar, and will not be shown one by one in this embodiment.

[0068] A test item refers to an independent test item on a test report; it is the smallest functional unit of the report. The test item recognition result is obtained by summarizing the information related to that test item from the text recognition results.

[0069] In some embodiments, at least one test item is obtained by determining the test item from the text recognition result using a large language model; for each test item, a test item recognition result is generated based on the text recognition result using a large language model.

[0070] The test result of a laboratory item refers to the detection result and related information corresponding to the laboratory item. Optionally, the test result of a laboratory item may include, but is not limited to, at least one of the following: laboratory item name, detection result, reference range, unit, abnormality indicator, etc., and may also include other information related to the laboratory item. This application embodiment does not limit this. The laboratory item name is used to uniquely identify the detection item. Optionally, the form of the laboratory item name may include, but is not limited to, at least one of the following: Chinese name, English abbreviation, full name in English, combination form, etc., and may also include other forms. This application embodiment does not limit this. The detection result refers to the actual detection value obtained for the detection item of the laboratory item. The reference range refers to the normal value range of the laboratory item in a statistical sample of healthy people. The unit refers to the unit of measurement of the measurement standard that identifies the detection value. The abnormality indicator refers to the status indicator generated based on the comparison between the test result and the reference range.

[0071] The test type refers to the category to which the test item belongs in the medical classification system. Optionally, the test type may include, but is not limited to, at least one of the following: complete blood count, complete urine count, complete stool count, liver function, kidney function, blood lipids, blood glucose, electrolytes, myocardial enzymes, coagulation function, thyroid function, immune function, infectious disease screening, tumor markers, hormone level detection, trace element detection, inflammatory markers, autoantibody detection, pathogen detection, blood gas analysis, etc., and may also include other test types, which are not limited in this application embodiment.

[0072] In some embodiments, the test results of the test items in the text recognition results are extracted by a large language model to obtain the test results of at least one test item; for each test item, the test type to which the test item belongs is determined by the large language model based on the text recognition results and the test results of the test item; and a set of test recognition results is generated based on the test results of each test item and the test type to which the test item belongs.

[0073] In this process, the large language model performs semantic understanding on the text recognition results, identifies at least one test item from the text recognition results, and extracts the information related to the test item as the test result for that test item. For each test item, the large language model matches and summarizes the information related to that test item from the text recognition results based on medical domain knowledge, and stores the information related to that test item in a specified format to obtain the test result for that test item.

[0074] In some embodiments, for each test item, if the text recognition result includes the test type of the test item, the test type of the test item is extracted from the text recognition result using a large language model; or, if the text recognition result does not include the test type of the test item, the test type of the test item is predicted based on the text recognition result using a large language model; and a test item recognition result of the test item is generated based on the test result and the test type of the test item.

[0075] Even if the text recognition result does not include the test type of the test item, since the large language model has medical-related knowledge, it can infer and predict the test type of the test item based on the contextual information related to the test item in the text recognition result.

[0076] In some embodiments, the format of the test item identification result may include, but is not limited to, at least one of the following: JSON, MsgPack, XML, YAML, custom format, etc., and may also include other formats, which are not limited in this application embodiment.

[0077] In some embodiments, the multimodal large language model and the large language model are the same model; or, the multimodal large language model and the large language model are different models.

[0078] When the multimodal large language model and the large language model are the same model, the multimodal large language model generates a set of test identification results based on the text recognition results. In this case, the text encoder of the multimodal large language model can understand and analyze the text recognition results to generate the test identification results. When the multimodal large language model and the large language model are different models, the text recognition results output by the multimodal large language model are input into the large language model; the large language model then generates a set of test identification results based on the text recognition results. By supporting two implementation methods—whether the multimodal large language model and the large language model are the same model or different models—the system can flexibly choose the model deployment scheme according to the actual application scenario, computing resources, and task complexity, thereby improving the overall system adaptability.

[0079] The above method uses a large language model to semantically analyze the text recognition results, extracts the test results of each test item, determines the test type to which each test item belongs, and then generates a structured set of test recognition results. This method improves the accuracy of extracting information related to test items, enhances the adaptability to test reports with different layouts and formats, realizes the automatic conversion of test reports with complex structures into structured information, and reduces the cost of manual intervention.

[0080] In some embodiments, the test result of a test item includes the test item name; a standardized test item set is obtained, the standardized test item set including at least one standardized test item name; for each test item in the test identification result set, the standardized test item name corresponding to the test item is determined in the standardized test item set based on the test item name of the test item using a large language model; the test item identification result of the test item is adjusted based on the standardized test item name corresponding to the test item to obtain the standardized test item identification result of the test item.

[0081] Because the same test result may have different names in different hospitals or institutions (e.g., due to abbreviations, aliases, language differences), it is necessary to store and manage test results using a unified, standardized name within the same system. Therefore, it is necessary to standardize test results from different sources so that test results with the same meaning but different names can be mapped to a unified, standardized name.

[0082] For example, the test item name is "HGB" and the standardized test item name is "hemoglobin".

[0083] The standardized test item set includes a set of standardized test item names. Optionally, the source of the standardized test item set may include, but is not limited to, at least one of the following: medical standard terminology database, medical information coding system, hospital internal unified dictionary, industry standards, etc., and may also include other sources, which are not limited in this embodiment.

[0084] For example, the standardized test item set can be as follows: { … Hemoglobin, White blood cell count, Alanine aminotransferase (ALT) Creatinine … } In some embodiments, textual features of the names of each standardized test item are extracted using a large language model; for each test item, textual features of the test item name are extracted using a large language model; and the large language model matches the textual features of the test item name with the textual features of the names of each standardized test item to determine the standardized test item name corresponding to the test item.

[0085] Optionally, based on the text features of the test item name and the text features of each standardized test item name, a text similarity of at least one standardized test item name is obtained. The text similarity of the standardized test item name is used to indicate the degree of similarity between the text features of the test item name and the text features of the standardized test item name. The standardized test item name with the highest text similarity is determined as the standardized test item name corresponding to that test item.

[0086] Optionally, the text similarity of standardized test item names can be calculated using any of the following methods: cosine similarity, dot product similarity, Euclidean distance to similarity, etc., or other methods. This application embodiment does not limit this.

[0087] Adjusting the test item identification result based on the standardized test item name corresponding to the test item means replacing the test item name with the standardized test item name corresponding to the test item to obtain the standardized test item identification result of the test item.

[0088] In some embodiments, post-processing prompts for laboratory report types are obtained. These prompts indicate the post-processing tasks for the text recognition results of medical images of laboratory report types. Based on the post-processing prompts for laboratory report types and the names of the laboratory tests, a large language model determines the standardized test names corresponding to each test item from a standardized test item set. In other words, the post-processing prompts for laboratory report types serve as prompts for the large language model, guiding it to standardize the test names of each test item.

[0089] It is important to note that technicians can design different post-processing prompts based on the text recognition results for different types of medical images. This allows the large language model to post-process the text recognition results output by the multimodal large language model. By using image-type-specific post-processing prompts to guide the large language model in post-processing the text recognition results, differentiated post-processing can be applied to different image types, thereby improving the accuracy of text recognition for different image types.

[0090] This approach first classifies medical images and generates corresponding recognition prompts based on image type. This allows a multimodal large language model to perform targeted text recognition tasks for different image types, thereby improving the recognition accuracy of complex medical documents. Furthermore, under preset conditions, semantic processing of the text recognition results through the large language model can improve the extraction accuracy of key information in specific types of medical documents and enhance the system's automated processing capabilities.

[0091] The above method, by inputting the standardized test item set and the test results of the test items into the large language model, and having the large language model standardize the mapping of the test item names, can eliminate data inconsistencies caused by naming differences, language differences, abbreviation differences or spelling differences, thereby achieving unified data management across institutions and systems.

[0092] In summary, the technical solution provided in this application classifies the medical image to be identified, determines the corresponding recognition prompt information based on the image type, and then uses a multimodal large language model to perform text recognition on the medical image based on the recognition prompt information, obtaining the text recognition result of the medical image. This approach effectively improves the accuracy of text recognition for different image types by having the multimodal large language model perform different text recognition tasks according to the image type of the medical image. Furthermore, when the medical image meets preset conditions, the text recognition result output by the large language model is processed to generate a set of laboratory test results, enabling accurate extraction of information related to laboratory items from medical images of the laboratory report type, thus improving the system's automation level.

[0093] The following is the process of desensitizing medical images.

[0094] Because medical images may contain information about patients or medical staff, it is necessary to desensitize the medical images to be identified in order to protect the privacy of patients and medical staff.

[0095] In some embodiments, sensitive information in a medical image is identified to obtain location information of at least one sensitive information, which is used to indicate the location of the sensitive information in the medical image; based on the location information of at least one sensitive information, the medical image is desensitized to obtain a desensitized medical image.

[0096] Sensitive information refers to content involving privacy or protected information. Optionally, sensitive information may include, but is not limited to, at least one of the following: patient's name, medical staff's name, ID number, mobile phone number, hospitalization number, medical record number, medical card number, address, medical insurance number, ward number, etc. It may also include other content involving privacy or protected information, which is not limited in this application embodiment.

[0097] In some embodiments, text regions in a medical image are identified to obtain location information of at least one text region, which is used to indicate the location of the text region in the medical image; based on the location information of at least one text region, the text content of each text region is extracted; sensitive information in the text content of each text region is detected to obtain a sensitivity detection result for each text region, which is used to indicate whether the text content of the text region contains sensitive information; based on the sensitivity detection results and location information of each text region, the text regions containing sensitive information in the medical image are desensitized to obtain a desensitized medical image; and a text recognition task is performed on the desensitized medical image using a multimodal large language model based on recognition prompts to obtain the text recognition result of the medical image.

[0098] Text regions in medical images refer to areas within the medical image that contain text. Each text region corresponds to a location information point, used to indicate the respective areas containing text within the medical image. In some embodiments, the location information can be represented by bounding box coordinates. That is, each text region corresponds to a bounding box, and the coordinates of this bounding box are used to represent the position of the text region in the medical image. Optionally, the representation of bounding box coordinates can include at least one of the following: top-left and bottom-right corner representation, top-left corner width and height representation, four-point coordinate representation, etc., and may also include other representation methods, which are not limited in this embodiment.

[0099] In some embodiments, a text target detection model is used to identify text regions in a medical image to obtain at least one text region. The text target detection model is used to identify text regions in an image, that is, a target detection model that treats text regions as detection targets. Optionally, the text target detection model is designed or trained by those skilled in the art.

[0100] In other words, the process involves detecting whether at least one text region contains sensitive information. This detection process is as follows: extract the text content from the text region, detect whether the text content contains sensitive information, and generate a sensitivity detection result for that text region. Then, based on the sensitivity detection results for each text region, the text regions containing sensitive information are desensitized to obtain the desensitized medical image.

[0101] In some embodiments, an Optical Character Recognition (OCR) model is used to extract the text content of each text region based on the location information of at least one text region. The OCR model is a computer vision model used to convert text information in an image into editable and computable character text. Optionally, the OCR model is designed or trained by relevant technical personnel.

[0102] The sensitivity detection result of a text region indicates whether the text region contains sensitive information. Optionally, the sensitivity detection result of a text region may include, but is not limited to, at least one of the following: whether it contains sensitive information, the category of sensitive information, the location information of sensitive information, etc., and may also include other content, which is not limited in this embodiment.

[0103] In some embodiments, the text region is desensitized according to the category of sensitive information in the sensitivity detection results of the text region; based on the location information of the sensitive information in the sensitivity detection results of the text region and the text region desensitization method, the text region is desensitized to obtain the desensitized text region.

[0104] In some embodiments, the desensitization method may include, but is not limited to, at least one of the following: rectangular block occlusion, mosaic processing, character replacement, occlusion symbol coverage, Gaussian blur, etc., and may also include other methods, which are not limited in this application embodiment.

[0105] For example, such as Figure 3 and 4 As shown, Figure 4 This illustration shows a schematic diagram of a desensitized laboratory report image provided in one embodiment of this application, with black rectangular blocks used to mask the image. Figure 3 The lab report image shown was desensitized to obtain... Figure 4 The image shown is a desensitized lab report image (400). Figure 4 In the desensitized lab report image 400 shown, similar black rectangular blocks are used to mask sensitive information 410.

[0106] The above method identifies and extracts text regions from medical images, then detects sensitive information within each text region, and desensitizes the medical image based on the detection results. This approach improves the accuracy of text extraction and sensitive information detection by accurately locating text regions and their sensitive information.

[0107] In some embodiments, for each text region, sensitive information in the text content of the text region is detected based on sensitive filtering rules to obtain a first detection result for the text region. The first detection result of the text region is used to indicate whether the text content of the text region contains sensitive information, and the sensitive filtering rules are used to indicate the judgment conditions for identifying sensitive information. If the first detection result of the text region indicates that the text content of the text region contains sensitive information, the first detection result of the text region is determined as the sensitive detection result of the text region. If the first detection result of the text region indicates that the text content of the text region does not contain sensitive information, a second detection result of the text region is obtained based on the text content of the text region through a text classification model. The second detection result is used to indicate whether the text content of the text region contains sensitive information, and the second detection result of the text region is determined as the sensitive detection result of the text region.

[0108] Sensitive filtering rules refer to a set of judgment conditions or algorithmic logic used to determine whether text content contains sensitive information. They are typically based on rule matching to quickly screen text content. In some embodiments, the logic of the judgment conditions can be implemented by at least one of the following: regular expression matching, keyword matching, pattern matching, etc., and can also be implemented by other methods, which are not limited in this application embodiment.

[0109] For example, the regular expression for an ID number is as follows: \d{17}[\dXx], which represents a string containing 18 digits or a string containing 17 digits ending with X.

[0110] The first detection result refers to the detection result obtained by detecting sensitive information in the text content of the text region through sensitive filtering rules. Optionally, the first detection result may include, but is not limited to, at least one of the following: whether it contains sensitive information, the category of sensitive information, the location information of sensitive information, etc., and may also include other content, which is not limited in this embodiment of the application.

[0111] The text content of the text area is initially screened by sensitive filtering rules to screen for relatively simple sensitive information and improve the detection efficiency of sensitive information.

[0112] A text classification model is used to understand the semantics of the text content in a text region in order to identify implicit sensitive information or sensitive information not covered by sensitive filtering rules. In other words, the text classification model further analyzes the text content of a text region to determine whether the text region contains sensitive information. Optionally, the text classification model can be implemented based on machine learning or deep learning; this embodiment of the application does not limit this.

[0113] The second detection result refers to the detection result obtained by detecting sensitive information in the text content of the text region through a text classification model. Optionally, the second detection result may include, but is not limited to, at least one of the following: whether it contains sensitive information, the category of sensitive information, the location information of sensitive information, etc., and may also include other content, which is not limited in this embodiment of the application.

[0114] For each text region, sensitive information in the text content is first detected based on sensitive filtering rules to obtain the first detection result of the text region. If the first detection result indicates that it contains sensitive information, it is used as the sensitive detection result of the text region. If the first detection result indicates that it does not contain sensitive information, the text content is further analyzed through a text classification model to obtain the second detection result, which is then used as the sensitive detection result of the text region, thereby generating the final sensitive detection result for each text region.

[0115] The above method uses a two-stage detection strategy to detect sensitive information in various text regions of medical images. The first stage quickly identifies obvious sensitive information through sensitive filtering rules, and the second stage further identifies sensitive information that is not covered by the rules or is implicit through a text classification model, thereby improving the overall detection accuracy and reducing missed detections and false detections.

[0116] The following describes the process of verifying the recognition results.

[0117] In some embodiments, the medical image is verified based on the location information of each text region to obtain a recognition verification result. The recognition verification result is used to indicate unrecognized text regions in the medical image, which are text regions where text recognition failed. Based on the recognition verification result and the location information of the unrecognized text regions, supplementary prompt information is generated. The supplementary prompt information is used to instruct the text recognition task for the unrecognized text regions in the medical image. Based on the supplementary prompt information, a multimodal large language model is used to perform a text recognition task on the unrecognized text regions in the medical image to obtain supplementary recognition results for the unrecognized text regions. Based on the supplementary recognition results for the unrecognized text regions, the text recognition results of the medical image are updated to obtain the updated text recognition results of the medical image.

[0118] In some embodiments, a multimodal large language model is used to perform a text recognition task on a medical image based on the location information and recognition prompt information of each text region, and the text recognition result of the medical image is obtained. The text recognition result of the medical image includes the text recognition result of at least one text region. If the text recognition result of the first text region is not found in the text recognition result of the medical image, the first text region is determined as an unrecognized text region.

[0119] In other words, ideally, the text recognition results of medical images include the text recognition results of all text regions. By traversing the text recognition results of each text region in the text recognition results of the medical image, the unrecognized text regions are identified.

[0120] In some embodiments, supplementary prompts are generated using a large language model based on the recognition verification results and the location information of unrecognized text regions. It should be noted that the large language model used here can be the same as or a different large language model used to generate the test result set.

[0121] In this embodiment, supplementary prompt information is used as prompt words in a multimodal large language model to re-perform text recognition tasks on unrecognized text regions in medical images, thereby supplementing the text recognition results of medical images and obtaining more complete updated text recognition results for medical images.

[0122] The above method, by recognizing and verifying the location information of each text region, can automatically detect unrecognized text regions, thereby avoiding text omissions caused by initial recognition failures and improving the coverage and completeness of medical image text recognition results.

[0123] The following section introduces the multimodal large language model and the training process of the large language model.

[0124] Due to the complexity of medical image document formats, even with a multimodal large-scale model, it's difficult to guarantee correct processing of all samples. Therefore, multimodal large-scale language models and / or large-scale language models need to learn from medical images with text recognition errors to improve their text recognition accuracy.

[0125] In some embodiments, at least one revised recognition result and the original medical image corresponding to the at least one revised recognition result are obtained, wherein the revised recognition result includes a revised set of text recognition results and / or a revised set of laboratory recognition results; at least one training sample is generated based on the at least one revised recognition result and the original medical image corresponding to the at least one revised recognition result; if the at least one training sample satisfies a first training condition, a multimodal large language model is trained based on the at least one training sample to obtain a trained multimodal large language model; and / or, if the at least one training sample satisfies a second training condition, a large language model is trained based on the at least one training sample to obtain a trained large language model.

[0126] It is understandable that the revised recognition result is obtained by revising the recognition result output by the multimodal large language model or the large language model. Optionally, the revised recognition result can be obtained automatically by the review system, or it can be obtained by relevant technical personnel (such as medical personnel, data labelers).

[0127] Optionally, after revising the original recognition result, the revised recognition result and the original recognition result are stored in a first database. The first database is used to store the revised recognition result and the original recognition result. It is understood that the original recognition result is a set of text recognition results and / or laboratory recognition results. That is, the first database stores at least one pair of revised recognition results and original recognition results.

[0128] In some embodiments, according to a first period, at least one pair of revised identification results and original identification results are obtained from a first database; for each pair of revised identification results and original identification results, if the revised identification results and original identification results are different, training information is generated based on the revised identification results and original identification results, and the training information is used to indicate the revised identification results and original identification results used to construct training samples. The first period is the period for generating training samples.

[0129] Optionally, the training information may include at least one of the following: identification information of the medical image, recognition time, identification information of the original recognition result, identification information of the revised recognition result, error type, etc., and may also include other information, which is not limited in this embodiment. In some embodiments, the training information may be implemented in a relational table format.

[0130] In some embodiments, at least one training sample is generated based on at least one piece of training information. Optionally, the at least one piece of training information can be implemented as a training data table. That is, the training data table includes at least one piece of training information.

[0131] The first training condition is the condition that triggers the training of the multimodal large language model. The second training condition is the condition that triggers the training of the large language model. It can be understood that, given at least one training sample that simultaneously satisfies both the first and second training conditions, the multimodal large language model and the large language model are trained based on at least one training sample, resulting in the trained multimodal large language model and the trained large language model.

[0132] The training methods for multimodal large language models and large language models may include, but are not limited to, at least one of the following: self-supervised pre-training, supervised fine-tuning, instruction fine-tuning, etc., and may also be trained in other ways. This application embodiment does not limit this.

[0133] In some embodiments, test samples are obtained, including revised recognition results and the original medical images corresponding to the revised recognition results; a first loss function value of the multimodal large language model is determined based on the test samples using the trained multimodal large language model; and / or, a second loss function value of the large language model is determined based on the test samples using the trained large language model; if the first loss function value is less than or equal to a first preset threshold, the trained multimodal large language model is deployed to the application environment; and / or, if the second loss function value is less than or equal to a second preset threshold, the trained large language model is deployed to the application environment. The first loss function value is used to indicate the accuracy of text recognition by the trained multimodal large language model. The second loss function value is used to indicate the accuracy of text recognition by the trained large language model.

[0134] The application environment refers to the environment in which the trained multimodal large language model and / or the large language model is put into use.

[0135] In some embodiments, if at least one training sample satisfies a first training condition, the multimodal large language model is trained based on the at least one training sample using a first training script to obtain a trained multimodal large language model, wherein the first training script is a training control program for training the multimodal large language model; and / or, if at least one training sample satisfies a second training condition, the large language model is trained based on the at least one training sample using a second training script to obtain a trained large language model, wherein the second training script is a training control program for training the large language model.

[0136] The above method automatically iteratively trains the multimodal large language model and / or the large language model to continuously learn the layout differences and detail changes of medical images based on the recognition results output by the large language model, thereby improving the recognition quality and adaptability of diverse medical images.

[0137] In some embodiments, the training samples include a revised identification result, the original medical image corresponding to the revised identification result, and an error type, wherein the error type is used to indicate the reason for the revision of the revised identification result; the first training condition includes: at least one training sample includes a training sample with an error type associated with a multimodal large language model; the second training condition includes: at least one training sample includes a training sample with an error type associated with a large language model.

[0138] Error types are key factors that trigger the training of multimodal large language models and / or large language models. Optionally, error types may include, but are not limited to, at least one of the following: text recognition error, unrecognized text, test type error, output format error, test result error for test item, etc., and may also include other error types, which are not limited in this embodiment. A text recognition error refers to the inconsistency between the text recognition result of a medical image and the content in the medical image. Unrecognized text refers to the presence of unrecognized text content in the medical image. A test type error refers to the incorrect classification of test items. A test result error for a test item refers to a matching error when summarizing relevant information of test items.

[0139] Optionally, the error types associated with the multimodal large language model may include, but are not limited to, at least one of the following: text recognition error, output format error, unrecognized text, etc., and may also include other error types, which are not limited in this embodiment.

[0140] Optionally, the error types related to the large language model may include, but are not limited to, at least one of the following: test type error, output format error, test result error of test item, etc., and may also include other error types, which are not limited in this embodiment.

[0141] In some embodiments, the first training condition further includes: the number of training samples being greater than or equal to a third preset threshold, reaching a second period, etc. The second period is the training period for the multimodal large language model. In some embodiments, the second training condition further includes: the number of training samples being greater than or equal to a third preset threshold, reaching a third period, etc. The third period is the training period for the large language model.

[0142] It should be noted that the aforementioned first preset threshold, second preset threshold, or third preset threshold is preset by relevant technical personnel.

[0143] The above approach avoids training the entire system by only triggering the training process of the corresponding model (multimodal large language model and large language model) for different types of errors, thereby reducing the consumption of computing resources and improving training efficiency and model update speed.

[0144] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0145] Please refer to Figure 5This diagram illustrates a block diagram of a text recognition device for medical images according to an embodiment of this application. The device has the functions described above, which can be implemented in hardware or by hardware executing corresponding software. The device can be the terminal device 10 or server 20 described above, or it can be located within the terminal device 10 or server 20. Figure 5 As shown, the device 500 may include a first acquisition module 510, a classification module 520, a second acquisition module 530, an identification module 540, and a generation module 550.

[0146] The first acquisition module 510 is used to acquire the medical image to be identified.

[0147] The classification module 520 is used to classify the medical image to obtain the image type of the medical image.

[0148] The second acquisition module 530 is used to acquire recognition prompt information corresponding to the image type of the medical image, and the recognition prompt information is used to indicate the text recognition task for the medical image.

[0149] The recognition module 540 is used to perform the text recognition task on the medical image based on the recognition prompt information using a multimodal large language model, and obtain the text recognition result of the medical image.

[0150] The generation module 550 is used to generate a set of laboratory identification results based on the text recognition results using a large language model when the image type of the medical image meets preset conditions. The set of laboratory identification results includes the identification results of at least one laboratory item, and the identification results of the laboratory item include the laboratory result of the laboratory item and the laboratory type to which the laboratory item belongs.

[0151] In some embodiments, the generation module 550 is further configured to extract the test results of the test items in the text recognition results through the large language model to obtain the test results of the at least one test item; for each test item, the large language model determines the test type to which the test item belongs based on the text recognition results and the test results of the test item; and generates the test recognition result set based on the test results of each test item and the test type to which the test item belongs.

[0152] In some embodiments, the test results of the test item include the test item name; the device 500 further includes a standardization module (in Figure 5(Not shown in the image) is used to obtain a standardized test item set, which includes at least one standardized test item name; for each test item in the test identification result set, the large language model determines the standardized test item name corresponding to the test item in the standardized test item set based on the test item name of the test item; based on the standardized test item name corresponding to the test item, the test item identification result of the test item is adjusted to obtain the standardized test item identification result of the test item.

[0153] In some embodiments, the device 500 further includes a desensitization module (in Figure 5 (Not shown in the image) is used to identify text regions in the medical image to obtain location information of at least one text region, the location information of which indicates the position of the text region in the medical image; based on the location information of the at least one text region, extract the text content of each text region; detect sensitive information in the text content of each text region to obtain a sensitivity detection result for each text region, the sensitivity detection result of which indicates whether the text content of the text region contains the sensitive information; based on the sensitivity detection results and location information of each text region, perform desensitization processing on the text regions in the medical image containing the sensitive information to obtain a desensitized medical image; the recognition module 540 is used to perform the text recognition task on the desensitized medical image using the multimodal large language model based on the recognition prompt information to obtain the text recognition result of the medical image.

[0154] In some embodiments, the desensitization module is further configured to, for each text region, detect sensitive information in the text content of the text region based on a sensitive filtering rule, and obtain a first detection result for the text region. The first detection result of the text region is used to indicate whether the text content of the text region contains the sensitive information, and the sensitive filtering rule is used to indicate the determination condition for identifying the sensitive information. If the first detection result of the text region indicates that the text content of the text region contains the sensitive information, the first detection result of the text region is determined as the sensitive detection result of the text region. If the first detection result of the text region indicates that the text content of the text region does not contain the sensitive information, a second detection result of the text region is obtained based on the text content of the text region using a text classification model. The second detection result is used to indicate whether the text content of the text region contains the sensitive information, and the second detection result of the text region is determined as the sensitive detection result of the text region.

[0155] In some embodiments, the device 500 further includes a supplementary module (in Figure 5 (Not shown in the image) is used to verify the medical image based on the location information of each of the text regions, and obtain a recognition verification result. The recognition verification result is used to indicate unrecognized text regions in the medical image, which are text regions where text recognition failed. Based on the recognition verification result and the location information of the unrecognized text regions, supplementary prompt information is generated. The supplementary prompt information is used to instruct the text recognition task for the unrecognized text regions in the medical image. Based on the supplementary prompt information, the multimodal large language model performs the text recognition task on the unrecognized text regions in the medical image to obtain supplementary recognition results for the unrecognized text regions. Based on the supplementary recognition results for the unrecognized text regions, the text recognition results of the medical image are updated to obtain the updated text recognition results of the medical image.

[0156] In some embodiments, the multimodal large language model and the large language model are the same model; or, the multimodal large language model and the large language model are different models.

[0157] In some embodiments, the device 500 further includes a training module (in Figure 5 (Not shown in the image) is used to obtain at least one revised recognition result and the original medical image corresponding to the at least one revised recognition result, wherein the revised recognition result includes a revised set of text recognition results and / or a revised set of laboratory recognition results; based on the at least one revised recognition result and the original medical image corresponding to the at least one revised recognition result, at least one training sample is generated; if the at least one training sample satisfies a first training condition, the multimodal large language model is trained based on the at least one training sample to obtain a trained multimodal large language model; and / or, if the at least one training sample satisfies a second training condition, the large language model is trained based on the at least one training sample to obtain a trained large language model.

[0158] In some embodiments, the training samples include the revision recognition result, the original medical image corresponding to the revision recognition result, and the error type, wherein the error type is used to indicate the reason for the revision recognition result being revised; the first training condition includes: the at least one training sample includes a training sample with an error type related to the multimodal large language model; the second training condition includes: the at least one training sample includes a training sample with an error type related to the large language model.

[0159] In summary, the technical solution provided in this application classifies the medical image to be identified, determines the corresponding recognition prompt information based on the image type, and then uses a multimodal large language model to perform text recognition on the medical image based on the recognition prompt information, obtaining the text recognition result of the medical image. This approach effectively improves the accuracy of text recognition for different image types by having the multimodal large language model perform different text recognition tasks according to the image type of the medical image. Furthermore, when the medical image meets preset conditions, the text recognition result output by the large language model is processed to generate a set of laboratory test results, enabling accurate extraction of information related to laboratory items from medical images of the laboratory report type, thus improving the system's automation level.

[0160] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0161] Please refer to Figure 6 This diagram illustrates a structural block diagram of a computer device 600 provided in one embodiment of this application. The computer device 600 may be... Figure 1 The terminal device 10 in the computer system shown can also be Figure 1 The server 20 in the computer system shown is used to implement the text recognition method for medical images provided in the above embodiments. Specifically: Typically, computer device 600 includes a processor 610 and a memory 620.

[0162] Processor 610 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 610 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 610 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 610 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 610 may also include an AI processor for handling computational operations related to machine learning.

[0163] The memory 620 may include one or more computer-readable storage media, which may be non-transitory. The memory 620 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 620 is used to store a computer program configured to be executed by one or more processors to implement the above-described text recognition method for medical images.

[0164] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0165] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the above-described text recognition method for medical images. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0166] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned text recognition method for medical images.

[0167] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0168] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for text recognition in medical images, characterized in that, The method includes: Acquire the medical image to be identified; The medical images are classified to obtain their image types; Obtain recognition prompt information corresponding to the image type of the medical image, wherein the recognition prompt information is used to instruct a text recognition task for the medical image; The text recognition task is performed on the medical image using a multimodal large language model based on the recognition prompt information, to obtain the text recognition result of the medical image; When the image type of the medical image meets the preset conditions, a set of laboratory identification results is generated based on the text recognition results using a large language model. The set of laboratory identification results includes the identification results of at least one laboratory item, and the identification results of the laboratory item include the laboratory result of the laboratory item and the laboratory type to which the laboratory item belongs.

2. The method according to claim 1, characterized in that, The step of generating a set of laboratory identification results based on the text recognition results using a large language model includes: The test results of the test items in the text recognition results are extracted by the large language model to obtain the test results of at least one test item; For each of the test items, the test type to which the test item belongs is determined by the large language model based on the text recognition result and the test result of the test item; Based on the test results of each test item and the test type to which the test item belongs, the test identification result set is generated.

3. The method according to claim 2, characterized in that, The test results for the aforementioned test items include the test item name; The method further includes: Obtain a standardized test item set, wherein the standardized test item set includes at least one standardized test item name; For each test item in the test identification result set, the standardized test item name corresponding to the test item is determined in the standardized test item set by the large language model based on the test item name of the test item; Based on the standardized test item name corresponding to the test item, the test item identification result of the test item is adjusted to obtain the standardized test item identification result of the test item.

4. The method according to claim 1, characterized in that, The method further includes: The text region in the medical image is identified to obtain the location information of at least one text region, and the location information of the text region is used to indicate the position of the text region in the medical image. Based on the location information of the at least one text region, extract the text content of each text region; Sensitive information is detected in the text content of each text region to obtain a sensitivity detection result for each text region. The sensitivity detection result of the text region is used to indicate whether the text content of the text region contains the sensitive information. Based on the sensitivity detection results of each text region and the location information of each text region, the text regions containing the sensitive information in the medical image are desensitized to obtain the desensitized medical image. The step of performing the text recognition task on the medical image based on the recognition prompt information using a multimodal large language model to obtain the text recognition result of the medical image includes: The text recognition task is performed on the desensitized medical image using the multimodal large language model based on the recognition prompt information, and the text recognition result of the medical image is obtained.

5. The method according to claim 4, characterized in that, The process of detecting sensitive information in the text content of each of the text regions and obtaining the sensitivity detection results for each of the text regions includes: For each text region, based on the sensitive filtering rule, sensitive information in the text content of the text region is detected to obtain a first detection result of the text region. The first detection result of the text region is used to indicate whether the text content of the text region contains the sensitive information. The sensitive filtering rule is used to indicate the determination condition for identifying the sensitive information. If the first detection result of the text region indicates that the text content of the text region contains the sensitive information, the first detection result of the text region shall be determined as the sensitive detection result of the text region; If the first detection result of the text region indicates that the text content of the text region does not contain the sensitive information, a second detection result of the text region is obtained based on the text content of the text region through a text classification model. The second detection result is used to indicate whether the text content of the text region contains the sensitive information. The second detection result of the text region is determined as the sensitive detection result of the text region.

6. The method according to claim 4, characterized in that, The method further includes: Based on the location information of each of the text regions, the medical image is verified to obtain a recognition verification result. The recognition verification result is used to indicate the unrecognized text regions in the medical image. The unrecognized text regions are text regions where text recognition failed. Based on the recognition verification result and the location information of the unrecognized text region, supplementary prompt information is generated. The supplementary prompt information is used to instruct the text recognition task for the unrecognized text region in the medical image. Based on the supplementary prompt information, the multimodal large language model performs the text recognition task on the unrecognized text regions in the medical image to obtain the supplementary recognition results for the unrecognized text regions; Based on the supplementary recognition results of the unrecognized text regions, the text recognition results of the medical image are updated to obtain the updated text recognition results of the medical image.

7. The method according to claim 1, characterized in that, The multimodal large language model and the large language model are the same model; or, the multimodal large language model and the large language model are different models.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Acquire at least one revised recognition result and the original medical image corresponding to the at least one revised recognition result, wherein the revised recognition result includes a set of revised text recognition results and / or revised laboratory recognition results; Based on the at least one revised identification result and the original medical image corresponding to the at least one revised identification result, at least one training sample is generated; If at least one training sample satisfies the first training condition, the multimodal large language model is trained based on the at least one training sample to obtain the trained multimodal large language model; and / or, If at least one training sample satisfies the second training condition, the large language model is trained based on the at least one training sample to obtain the trained large language model.

9. The method according to claim 8, characterized in that, The training samples include the revision recognition result, the original medical image corresponding to the revision recognition result, and the error type, wherein the error type is used to indicate the reason why the revision recognition result was revised; The first training condition includes: the at least one training sample includes training samples of error types related to the multimodal large language model; The second training condition includes: the at least one training sample includes training samples of error types related to the large language model.

10. A text recognition device for medical images, characterized in that, The device includes: The first acquisition module is used to acquire the medical image to be identified; A classification module is used to classify the medical images to obtain the image type of the medical images; The second acquisition module is used to acquire recognition prompt information corresponding to the image type of the medical image, and the recognition prompt information is used to indicate the text recognition task for the medical image; The recognition module is used to perform the text recognition task on the medical image based on the recognition prompt information using a multimodal large language model, and obtain the text recognition result of the medical image; The generation module is used to generate a set of laboratory identification results based on the text recognition results using a large language model when the image type of the medical image meets preset conditions. The set of laboratory identification results includes the identification results of at least one laboratory item, and the identification results of the laboratory item include the laboratory result of the laboratory item and the laboratory type to which the laboratory item belongs.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as described in any one of claims 1 to 9.

Images