Abnormality detection method for physical examination report based on OCR and natural language model technology

Abstract

The present application provides a method for detecting physical examination report anomalies based on OCR and natural language model technology. It includes: obtaining physical examination report materials to be identified; using a layout analysis algorithm to classify and process the physical examination report materials to be identified based on the layout type to obtain classified materials, wherein the layout type includes images, text and tables; according to the layout type of the classified materials, using OCR recognition technology and natural language models to analyze and process the classified materials, generate physical examination report detection results, and determine whether there are physical examination abnormalities in the physical examination report materials to be identified based on the physical examination report detection results. Through the above-mentioned physical examination report anomaly detection method, the digitization process of the physical examination report can be greatly accelerated, and at the same time, the abnormal data in the physical examination report can be quickly screened, and key information can be delivered to medical personnel. It solves the problem that the data extracted from the report by the existing technology is incomplete, resulting in a poor user experience.

Description

Technical Field

[0001] The present application relates to the technical field of abnormality detection in physical examination reports, and in particular to a method, device, computer-readable storage medium and electronic device for detecting abnormality in physical examination reports based on OCR and natural language model technology. Background Art

[0002] The current OCR recognition technology for physical examination reports mainly targets the detection of table-type data, that is, the different test items and their values ​​in the physical examination report are detected and extracted and input into the formatted template. However, this method ignores the diversity of layouts and types in physical examination reports, which contain not only table-type data, but also data displayed in pictures (such as B-ultrasound images, electrocardiograms, etc.), as well as plain text information such as symptom descriptions and summaries. Extracting only the table data in the physical examination report will result in an incomplete digitization of the physical examination report, and may also ignore important text summaries, resulting in missing key information in the physical examination report and a poor user experience. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, computer-readable storage medium and electronic device for detecting abnormalities in physical examination reports based on OCR and natural language model technology, so as to at least solve the problem that the data extracted from the report in the prior art is incomplete, resulting in a poor user experience.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for detecting physical examination report abnormalities based on OCR and natural language model technology is provided, including: obtaining physical examination report materials to be identified, wherein the physical examination report materials to be identified are in the form of pictures; classifying the physical examination report materials to be identified based on the layout type using a layout analysis algorithm to obtain classified materials, wherein the layout type includes images, text and tables; according to the layout type of the classified materials, using OCR recognition technology and a natural language model to analyze and process the classified materials to generate a physical examination report detection result, and determining whether the physical examination report materials to be identified have any physical examination abnormalities based on the physical examination report detection result.

[0005] Optionally, according to the layout type of the classified material, OCR recognition technology and a natural language model are used to analyze and process the classified material to generate a physical examination report detection result, including: when it is determined that the layout type of the classified material is the image, using the OCR recognition technology to extract the context information of the classified material in the physical examination report material to be identified; inputting the context information into the natural language model to generate the physical examination report detection result.

[0006] Optionally, according to the layout type of the classified material, OCR recognition technology and a natural language model are used to analyze and process the classified material to generate a physical examination report test result, including: when it is determined that the layout type of the classified material is the table, the OCR recognition technology is used to identify and process each test item in the classified material to obtain each test item information; each test item information is matched with standard test item information to generate a matching result; abnormal items in the test item information are marked according to the matching result to generate the physical examination report test result.

[0007] Optionally, according to the layout type of the classified material, OCR recognition technology and a natural language model are used to analyze and process the classified material to generate a physical examination report detection result, including: when it is determined that the layout type of the classified material is text, the OCR recognition technology is used to extract and process the classified material to generate text information; the natural language model is used to extract and process abnormal text descriptions in the text information to generate abnormal description information; the content of the text information is marked according to the abnormal description information to generate the physical examination report detection result.

[0008] Optionally, according to the layout type of the classified materials, OCR recognition technology and natural language model are used to analyze and process the classified materials to generate physical examination report detection results, including: according to the layout type of each classified material, OCR recognition technology and natural language model are used to analyze and process the classified materials to generate image detection results, table detection results and text detection results; the image detection results, the table detection results and the text detection results are integrated to generate the physical examination report detection results.

[0009] Optionally, after analyzing and processing the classified materials according to the layout type of the classified materials using OCR recognition technology and a natural language model to generate a physical examination report test result, the method also includes: sending the physical examination report test result to the review end; upon receiving the review pass information sent by the review end, sending the physical examination report test result and the physical examination report materials to be identified to a designated user end.

[0010] Optionally, after sending the physical examination report test results to the review end, the method also includes: obtaining the review time for sending the physical examination report test results to the review end, and determining whether the review time exceeds the preset time; if the review time exceeds the preset time, sending the physical examination report materials to be identified to the designated user end.

[0011] According to another aspect of the present application, a physical examination report abnormality detection device based on OCR and natural language model technology is provided, including: an acquisition unit, used to acquire physical examination report materials to be identified, wherein the physical examination report materials to be identified are in the form of pictures; a classification processing unit, used to classify the physical examination report materials to be identified based on the layout type using a layout analysis algorithm to obtain classified materials, wherein the layout type includes images, text and tables; an analysis processing unit, used to analyze and process the classified materials according to the layout type of the classified materials using OCR recognition technology and a natural language model to generate a physical examination report detection result, and determine whether the physical examination report materials to be identified have physical examination abnormalities based on the physical examination report detection result.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the physical examination report abnormality detection methods based on OCR and natural language model technology.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the physical examination report abnormality detection methods based on OCR and natural language model technology.

[0014] Applying the technical solution of the present application, by obtaining the physical examination report materials to be identified, wherein the physical examination report materials to be identified are in the form of pictures; using the layout analysis algorithm based on the layout type to classify and process the physical examination report materials to be identified, to obtain classified materials, wherein the layout types include images, text and tables; according to the layout type of the classified materials, using OCR recognition technology and natural language models to analyze and process the classified materials, generate physical examination report detection results, and determine whether there are physical examination abnormalities in the physical examination report materials to be identified based on the physical examination report detection results. Through the above-mentioned physical examination report abnormality detection method, the digitization process of the physical examination report can be greatly accelerated, and at the same time, the abnormal data in the physical examination report can be quickly screened, and key information can be delivered to medical personnel. It solves the problem that the data extracted from the report by the existing technology is incomplete, resulting in a poor user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for detecting abnormalities in a physical examination report based on OCR and natural language model technology provided in an embodiment of the present application is shown;

[0017] Figure 2 A flow chart of a method for detecting abnormalities in a physical examination report based on OCR and natural language model technology according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of a process for detecting abnormalities in a physical examination report based on OCR and natural language model technology according to an embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram of a processing flow in which the medical examination report material to be identified is a picture according to an embodiment of the present application is shown;

[0020] Figure 5 A schematic diagram of a processing flow in which the physical examination report material to be identified is a table according to an embodiment of the present application is shown;

[0021] Figure 6 A schematic diagram of a processing flow for a medical examination report material to be identified as text according to an embodiment of the present application is shown;

[0022] Figure 7 A structural block diagram of a physical examination report abnormality detection device based on OCR and natural language model technology provided according to an embodiment of the present application is shown.

[0023] The above drawings include the following reference numerals:

[0024] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] As introduced in the background technology, the data extracted in the report by the prior art is incomplete, resulting in a poor user experience. In order to solve the problem that the data extracted in the report by the prior art is incomplete, resulting in a poor user experience, the embodiments of the present application provide a method, device, computer-readable storage medium and electronic device for detecting anomalies in physical examination reports based on OCR and natural language model technology.

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for detecting abnormalities in a physical examination report based on OCR and natural language model technology according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the physical examination report abnormality detection method based on OCR and natural language model technology in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific example of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a method for detecting abnormalities in physical examination reports based on OCR and natural language model technology is provided, which runs on a mobile terminal, a computer terminal or a similar computing device. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 1 is a flow chart of a method for detecting abnormalities in a physical examination report based on OCR and natural language model technology according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0034] Step S201, obtaining the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture;

[0035] Specifically, the required physical examination report pictures can be extracted through the database, or obtained locally through the physical examination report pictures. The obtained pictures are classified according to the UID (user) number and ID (number). Each user has a different number of physical examination pictures independently, and the physical examination report pictures of each user are distinguished by numbers.

[0036] Step S202, classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables;

[0037] Specifically, layout analysis, also called layout parsing or document image analysis, is a technology used to extract structured information from scanned document images.

[0038] Step S203, according to the layout type of the classified materials, the OCR recognition technology and the natural language model are used to analyze and process the classified materials, generate a physical examination report detection result, and determine whether there are any physical examination abnormalities in the physical examination report materials to be identified according to the physical examination report detection result.

[0039] Through this embodiment, by obtaining the physical examination report materials to be identified, wherein the above-mentioned physical examination report materials to be identified are in the form of pictures; based on the layout type, the layout analysis algorithm is used to classify and process the above-mentioned physical examination report materials to be identified, and classified materials are obtained, wherein the above-mentioned layout types include images, texts and tables; according to the above-mentioned layout types of the above-mentioned classified materials, OCR recognition technology and natural language models are used to analyze and process the above-mentioned classified materials to generate physical examination report detection results, and according to the above-mentioned physical examination report detection results, it is determined whether the above-mentioned physical examination report materials to be identified have physical examination abnormalities. Through the above-mentioned physical examination report abnormality detection method, the digitization process of the physical examination report can be greatly accelerated, and at the same time, the abnormal data in the physical examination report can be quickly screened, and key information can be delivered to medical personnel. It solves the problem that the data extracted from the report by the existing technology is incomplete, resulting in a poor user experience.

[0040] During the specific implementation process, according to the above-mentioned layout type of the above-mentioned classified materials, OCR recognition technology and natural language model are used to analyze and process the above-mentioned classified materials to generate the physical examination report detection result, including: when it is determined that the above-mentioned layout type of the above-mentioned classified materials is the above-mentioned image, the above-mentioned OCR recognition technology is used to extract the context information of the above-mentioned classified materials in the above-mentioned physical examination report materials to be identified; the above-mentioned context information is input into the above-mentioned natural language model to generate the above-mentioned physical examination report detection result.

[0041] For the text before the image, the method uses the OCR optical character extraction technology to extract the relevant information of the physical examination item description and inputs it into the large language model; for the text after the image, the method uses the OCR optical character extraction technology to extract the test results under the physical examination item, and inputs the text characters obtained by OCR into the large language model; the large language model is used to first classify the image into the corresponding part of the output result according to the description of the physical examination item, and at the same time refine the expression of the test result. If there is no abnormal description of the physical examination result, "no abnormality" is output, otherwise the abnormal result is extracted and incorporated into the test result of the physical examination report.

[0042] Specifically, according to the above-mentioned layout type of the above-mentioned classified materials, OCR recognition technology and natural language model are used to analyze and process the above-mentioned classified materials to generate physical examination report detection results, including: when it is determined that the above-mentioned layout type of the above-mentioned classified materials is the above-mentioned table, the above-mentioned OCR recognition technology is used to identify and process each detection item in the above-mentioned classified materials to obtain each detection item information; each of the above-mentioned detection item information is matched with the standard detection item information to generate a matching result; abnormal items of the above-mentioned detection item information are marked according to the above-mentioned matching results to generate the above-mentioned physical examination report detection results.

[0043] This method uses OCR to identify each detection item detection name and the corresponding formatting template to compare the preset physical examination index name in the template to determine whether the two are the same. If the two are the same, the match is successful, and if the two are not the same, the match is unsuccessful. For physical examination indicators that may have multiple names, other possible names corresponding to them can be recorded when presetting the physical examination index name. When comparing, the detected name is compared with all preset names for consistency. If one of them is the same as the preset name, it can be considered that the match is successful; for physical examination index items that have been successfully matched, search the column representing the index result and the reference value column in the same row to extract the results of the two items. The result is compared with the reference value. If the result is within the normal range of the reference value, it indicates that the user's indicator is normal and no separate mark is required. If the result is outside the normal range of the reference value, it indicates that the user's indicator is abnormal and the relevant medical staff should pay special attention to it. Therefore, the test item needs to be marked separately as one of the abnormal items in the physical examination report. After the table obtained by the layout analysis is extracted and the abnormal detection is marked separately, the results need to be integrated and input into the formatting template, and the columns of each test indicator in the template are filled in according to the test results. For physical examination indicators that fail to match, they are marked and sent for manual review.

[0044] More specifically, according to the above-mentioned layout type of the above-mentioned classified materials, OCR recognition technology and natural language model are used to analyze and process the above-mentioned classified materials to generate physical examination report detection results, including: when it is determined that the above-mentioned layout type of the above-mentioned classified materials is the above-mentioned text, the above-mentioned OCR recognition technology is used to extract and process the above-mentioned classified materials to generate text information; the above-mentioned natural language model is used to extract and process the abnormal text description in the above-mentioned text information to generate abnormal description information; the content of the above-mentioned text information is marked according to the above-mentioned abnormal description information to generate the above-mentioned physical examination report detection results.

[0045] This method uses OCR technology to extract text characters from text layouts and converts the optical images therein into text characters. For text characters that have been extracted and detected by OCR, a pre-trained natural language model is used to determine whether the text contains text describing abnormal physical examination results. The natural language model can be trained from scratch or fine-tuned based on an existing Chinese model based on pre-labeled data, including complete physical examination results and descriptions of abnormal physical examinations. The data set used for training includes original text and extracted text. The original text is input into the model, and the model learns to extract descriptions of abnormal physical examinations during the training process.

[0046] According to the above-mentioned model for extracting descriptions of abnormal physical examination results, the descriptions of abnormal physical examination results in the text characters are extracted and marked separately, wherein the text descriptions of abnormal physical examination results are usually information that relevant medical personnel should pay special attention to, which can be used to remind relevant medical personnel and prevent omission of key reports. The complete text obtained by the above OCR and the text descriptions of abnormal physical examination results processed and output by the natural language model are integrated to generate an output, wherein the results output by the natural language model are separately marked and accompanied by the text source of the abnormal part.

[0047] Furthermore, according to the above-mentioned layout type of the above-mentioned classified materials, OCR recognition technology and natural language model are used to analyze and process the above-mentioned classified materials to generate physical examination report detection results, including: according to the above-mentioned layout type of each of the above-mentioned classified materials, OCR recognition technology and natural language model are used to analyze and process the above-mentioned classified materials to generate image detection results, table detection results and text detection results; the above-mentioned image detection results, the above-mentioned table detection results and the above-mentioned text detection results are integrated and processed to generate the above-mentioned physical examination report detection results.

[0048] This method can greatly accelerate the digitization process of physical examination reports, solving the problem that the existing technology of extracting report data is incomplete, resulting in poor user experience.

[0049] Furthermore, after analyzing and processing the above-mentioned classified materials according to the above-mentioned layout type of the above-mentioned classified materials by using OCR recognition technology and natural language model to generate physical examination report detection results, the above-mentioned method also includes: sending the above-mentioned physical examination report detection results to the review end; upon receiving the review pass information sent by the above-mentioned review end, sending the above-mentioned physical examination report detection results and the above-mentioned physical examination report materials to be identified to the designated user end.

[0050] The method can also quickly screen abnormal data in the physical examination report and deliver key information to the review end, thereby improving the detection speed of the physical examination report, wherein the review end can be a medical staff.

[0051] Specifically, after sending the above-mentioned physical examination report test results to the review end, the above-mentioned method also includes: obtaining the review time for sending the above-mentioned physical examination report test results to the above-mentioned review end, and determining whether the above-mentioned review time exceeds the preset time; if the above-mentioned review time exceeds the above-mentioned preset time, the above-mentioned physical examination report materials to be identified are sent to the specified above-mentioned user end.

[0052] In addition, for medical examination report test results that fail the review or take too long to review, the original report image can be output to the terminal, and users can get the health status in time based on the image, avoiding the large amount of time required for repeated manual review, improving processing efficiency and reducing manual operations.

[0053] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the physical examination report abnormality detection method based on OCR and natural language model technology of the present application will be described in detail below in combination with specific embodiments.

[0054] This embodiment relates to a specific method for detecting abnormalities in physical examination reports based on OCR and natural language model technology. Figure 3 As shown, the following steps are included:

[0055] Step S1: Acquire pictures for abnormality detection in physical examination reports;

[0056] In this embodiment, the required physical examination report pictures can be extracted through the database, or they can be obtained locally through the physical examination report pictures. The obtained pictures are classified according to the UID (user) number and ID (number). Each user has a different number of physical examination pictures, and the physical examination report pictures of each user are distinguished by numbers.

[0057] Step S2: Preprocess the image to be identified. The shooting light and angle of each image may be different, and the location of the physical examination report is also different. The images are cropped, rotated, etc. to make them in the same format, which is convenient for subsequent OCR detection.

[0058] Step S3: Analyze the layout of the pre-processed physical examination report images and divide the layout into three categories: images, tables and texts;

[0059] Layout analysis, also known as layout parsing or document image analysis, is a technology used to extract structured information from scanned document images;

[0060] Specifically, after obtaining the preprocessed image, the physical examination image can be subjected to layout analysis using an existing layout analysis algorithm to obtain layouts of different parts.

[0061] As an exemplary embodiment, for the medical examination report image to be detected, the connected areas in the image can be first identified to determine the text area and the non-text area and segment them. The segmented areas are classified into three areas: paragraphs, pictures, and tables. The characteristics of each area are identified to classify them, such as the baseline, character spacing, and geometric features of the text line; the grid structure and cell structure of the table; the color features, edge density, and texture features of the picture.

[0062] After the text, pictures and tables are segmented, they are analyzed and processed separately.

[0063] Step S4, after obtaining the table recognition result and the text recognition result of the embodiment, they are combined with the pictures obtained during the layout analysis and classified by the large language model.

[0064] Step S5, generate an electronic version of the physical examination report and a result report of the abnormality detection of the physical examination report, and submit the result to manual review to check whether there are any recognition errors and factual errors. If the review is correct, upload the result to the terminal.

[0065] At the same time, for physical examination test result reports that fail the review, the original report image can be output to the terminal, and users can obtain their health status in a timely manner based on the image, avoiding the large amount of time required for repeated manual review, improving processing efficiency and reducing manual operations.

[0066] When the segmented layout is a picture, the above large language model is used to identify the context of the picture and extract its key information, such as Figure 4 As shown, in one embodiment, optionally, in step S4, when the layout is a picture, the steps include:

[0067] Step S401, for the text before the picture, use the OCR optical character extraction technology to extract the relevant information of the physical examination item description and input it into the large language model;

[0068] Step S402: For the text behind the image, use the optical character extraction (OCR) technology to extract the test results of the physical examination item, and input the text characters obtained by OCR into the large language model;

[0069] Step S403, using a large language model to first classify the image into the corresponding part of the output result according to the description of the physical examination item, and at the same time refine the description of the test result. If there is no description of abnormal physical examination results, "no abnormality" is output, otherwise the abnormal results are extracted and incorporated into the latest report.

[0070] Step S404: summarize and output the above generated results, and end the process.

[0071] like Figure 5 As shown, in one embodiment, optionally, in step S4, when the layout is a table, the steps include:

[0072] Step S501, using OCR technology to identify various test items, and matching the identified test items with various test items in the formatting template;

[0073] Step S502, respectively processing the detection items that successfully match the formatted template and the items that fail to match;

[0074] Specifically, the detection name of each detection item identified by OCR is compared with each preset physical examination indicator name in the corresponding formatting template to determine whether the two are the same. If the two are the same, the match is successful, and if the two are not the same, the match is unsuccessful. For physical examination indicators that may have multiple names, other possible names corresponding to them can be recorded when presetting the physical examination indicator name. When comparing, the detected name is compared with all preset names for consistency. If one of them is the same as the preset name, it can be considered that the match is successful;

[0075] For the physical examination indicators that are successfully matched, S503 and S504 are executed according to the subsequent process. For the physical examination indicators that are not successfully matched, they are marked and sent for manual review.

[0076] Step S503, OCR detection extracts the results of each detection index that has been successfully matched, and processes according to the results and their normal intervals;

[0077] Specifically, for the physical examination index items that have been successfully matched, search the column representing the index result and the reference value column in the same row, and extract the results of the two items. Compare the result with the reference value. If the result is within the normal range of the reference value, it means that the user's index is normal and no separate mark is made; if the result is outside the normal range of the reference value, it means that the user's index is abnormal, and the relevant medical staff should pay special attention to it. Therefore, the test item needs to be marked separately as one of the abnormal items in the physical examination report.

[0078] Step S504, after the table obtained from the layout analysis is extracted and the abnormal detection is separately marked, the results need to be integrated and input into the formatting template, and the columns of each detection indicator in the template are filled in according to the detection results. At the same time, the abnormal parts of the results judged in the above step S503 are integrated to generate the output.

[0079] The formatted template can be maintained in the database as a format for filling in the recognition result. As shown in Table 1:

[0080] Table 1

[0081]

[0082]

[0083] like Figure 6 As shown, in one embodiment, optionally, in step S4, when the layout is text, the steps include:

[0084] Step S601, using OCR technology to extract text characters from the text layout, and converting the optical image therein into text characters.

[0085] Step S602: for the extracted text, determine whether there is a text description indicating that the physical examination result is abnormal. The part describing the abnormal physical examination result is extracted and marked separately.

[0086] Specifically, for text characters that have been extracted and detected by OCR, a pre-trained natural language model is used to determine whether the text contains text that describes abnormal physical examination results.

[0087] The natural language model can be trained from scratch or fine-tuned based on an existing Chinese model based on pre-labeled data, including complete physical examination results and descriptions of physical examination abnormalities. The training data set includes original text and extracted text. The original text is input into the model, and the model learns to extract descriptions of physical examination abnormalities during training.

[0088] Step S603, according to the above-mentioned abnormal physical examination result description extraction model, the description of the abnormal physical examination result class in the text characters is extracted and marked separately, wherein the text description of the abnormal physical examination result is usually information that relevant medical personnel should pay special attention to, which can facilitate reminding relevant medical personnel and prevent omission of key reports.

[0089] Step S604, integrating the complete text obtained by the above OCR and the text description of the abnormal physical examination result output by the natural language model processing to generate an output, wherein the result output by the natural language model is individually marked and accompanied by the text source of its abnormal part.

[0090] After obtaining the table recognition result and the text recognition result of the embodiment, they are combined with the pictures obtained during the layout analysis and classified by the large language model.

[0091] Generate an electronic version of the physical examination report and a result report of the abnormality detection of the physical examination report, and submit the result to manual review to check whether there are any recognition errors and factual errors. If the review is correct, upload the result to the terminal.

[0092] At the same time, for physical examination test result reports that fail the review, the original report image can be output to the terminal, and users can obtain their health status in a timely manner based on the image, avoiding the large amount of time required for repeated manual review, improving processing efficiency and reducing manual operations.

[0093] This embodiment solves the problem of imperfect detection schemes for various types of layouts in the existing physical examination report OCR technology. It fails to distinguish between tables, text and pictures, resulting in a large amount of manual labor still required in the process of electronicization of physical examination reports. This embodiment proposes a method for detecting anomalies in physical examination reports based on OCR and natural language model technology, so as to achieve the purpose of improving the efficiency and automation of physical examination report recognition.

[0094] The embodiment of the present application also provides a physical examination report abnormality detection device based on OCR and natural language model technology. It should be noted that the physical examination report abnormality detection device based on OCR and natural language model technology in the embodiment of the present application can be used to execute the physical examination report abnormality detection method based on OCR and natural language model technology provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0095] The following is an introduction to the physical examination report abnormality detection device based on OCR and natural language model technology provided in an embodiment of the present application.

[0096] Figure 7 Schematic diagram of a medical examination report abnormality detection device based on OCR and natural language model technology according to an embodiment of the present application. Figure 7 As shown, the device comprises:

[0097] An acquisition unit 71 is used to acquire the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture;

[0098] A classification processing unit 72, configured to classify the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables;

[0099] The analysis and processing unit 73 is used to analyze and process the above-mentioned classified materials according to the above-mentioned layout type of the above-mentioned classified materials using OCR recognition technology and natural language model, generate a physical examination report detection result, and determine whether the above-mentioned physical examination report materials to be identified have any physical examination abnormalities based on the above-mentioned physical examination report detection result.

[0100] In this embodiment, the acquisition unit is used to acquire the physical examination report materials to be identified, wherein the physical examination report materials to be identified are in the form of pictures; the classification processing unit is used to classify the physical examination report materials to be identified based on the layout type using a layout analysis algorithm to obtain classified materials, wherein the layout type includes images, text and tables; the analysis processing unit is used to analyze and process the classified materials according to the layout type of the classified materials using OCR recognition technology and a natural language model to generate a physical examination report test result, and determine whether the physical examination report materials to be identified have physical examination abnormalities based on the physical examination report test result. In this way, the digitization process of the physical examination report can be greatly accelerated, and at the same time, the abnormal data in the physical examination report can be quickly screened, and key information can be delivered to medical personnel. It solves the problem that the data extracted from the report by the existing technology is incomplete, resulting in a poor user experience.

[0101] As an optional scheme, the analysis and processing unit includes a first extraction module and a first generation module; the first extraction module is used to extract the context information of the above-mentioned classified material in the above-mentioned physical examination report material to be identified by using the above-mentioned OCR recognition technology when determining that the above-mentioned layout type of the above-mentioned classified material is the above-mentioned image; the first generation module is used to input the above-mentioned context information into the above-mentioned natural language model to generate the above-mentioned physical examination report detection result.

[0102] An optional solution, the analysis and processing unit includes a recognition processing module, a second generation module and a third generation module; the recognition processing module is used to, when determining that the layout type of the above-mentioned classified material is the above-mentioned table, use the above-mentioned OCR recognition technology to identify and process each detection item in the above-mentioned classified material to obtain each detection item information; the second generation module is used to match the above-mentioned detection item information with the standard detection item information to generate a matching result; the third generation module is used to mark the abnormal items of the above-mentioned detection item information according to the above-mentioned matching result to generate the above-mentioned physical examination report detection result.

[0103] An optional solution, the analysis and processing unit includes a second extraction module, a second generation module and a third generation module; the second extraction module is used to use the above-mentioned OCR recognition technology to extract and process the above-mentioned classification material when it is determined that the above-mentioned layout type of the above-mentioned classification material is the above-mentioned text, so as to generate text information; the second generation module is used to use the above-mentioned natural language model to extract and process the abnormal text description in the above-mentioned text information, and generate abnormal description information; the third generation module is used to mark the content of the above-mentioned text information according to the above-mentioned abnormal description information, and generate the above-mentioned physical examination report detection result.

[0104] An optional solution, the analysis and processing unit also includes an analysis and processing module and an integration processing module; the analysis and processing module is used to analyze and process the above-mentioned classified materials according to the above-mentioned layout type of each of the above-mentioned classified materials, respectively, using OCR recognition technology and natural language model to generate image detection results, table detection results and text detection results; the integration processing module is used to integrate the above-mentioned image detection results, the above-mentioned table detection results and the above-mentioned text detection results to generate the above-mentioned physical examination report detection results.

[0105] An optional solution, the device also includes a first sending unit and a second sending unit; the first sending unit is used to analyze and process the above-mentioned classified materials according to the above-mentioned layout type of the above-mentioned classified materials using OCR recognition technology and a natural language model, and after generating a physical examination report test result, the physical examination report test result is sent to the review end; the second sending unit is used to send the above-mentioned physical examination report test result and the above-mentioned physical examination report materials to be identified to a designated user end when receiving the review pass information sent by the above-mentioned review end.

[0106] An optional solution, the device also includes a determination unit and a third sending unit; the determination unit is used to obtain the review time for sending the above-mentioned physical examination report test results to the above-mentioned review end after sending the above-mentioned physical examination report test results to the above-mentioned review end, and determine whether the above-mentioned review time exceeds the preset time; the third sending unit is used to send the above-mentioned physical examination report materials to be identified to the specified above-mentioned user end when the above-mentioned review time exceeds the above-mentioned preset time.

[0107] The above-mentioned physical examination report abnormality detection device based on OCR and natural language model technology includes a processor and a memory. The above-mentioned acquisition unit, classification processing unit and analysis processing unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in the form of any combination.

[0108] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of incomplete data extraction report in the prior art, resulting in poor user experience, can be solved by adjusting kernel parameters.

[0109] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0110] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the physical examination report abnormality detection method based on OCR and natural language model technology.

[0111] Specifically, the physical examination report anomaly detection method based on OCR and natural language model technology includes:

[0112] Step S201, obtaining the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture;

[0113] Step S202, classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables;

[0114] Step S203, according to the layout type of the classified materials, the OCR recognition technology and the natural language model are used to analyze and process the classified materials, generate a physical examination report detection result, and determine whether there are any physical examination abnormalities in the physical examination report materials to be identified according to the physical examination report detection result.

[0115] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the physical examination report abnormality detection method based on OCR and natural language model technology when running.

[0116] Specifically, the physical examination report anomaly detection method based on OCR and natural language model technology includes:

[0117] Step S201, obtaining the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture;

[0118] Step S202, classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables;

[0119] Step S203, according to the layout type of the classified materials, the OCR recognition technology and the natural language model are used to analyze and process the classified materials, generate a physical examination report detection result, and determine whether there are any physical examination abnormalities in the physical examination report materials to be identified according to the physical examination report detection result.

[0120] An embodiment of the present invention provides an electronic device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0121] Step S201, obtaining the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture;

[0122] Step S202, classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables;

[0123] Step S203, according to the layout type of the classified materials, the OCR recognition technology and the natural language model are used to analyze and process the classified materials, generate a physical examination report detection result, and determine whether there are any physical examination abnormalities in the physical examination report materials to be identified according to the physical examination report detection result.

[0124] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0125] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes at least the following method steps: Step S201, obtaining a medical examination report material to be identified, wherein the medical examination report material to be identified is in the form of a picture;

[0126] Step S202, classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables;

[0127] Step S203, according to the layout type of the classified materials, the OCR recognition technology and the natural language model are used to analyze and process the classified materials, generate a physical examination report detection result, and determine whether there are any physical examination abnormalities in the physical examination report materials to be identified according to the physical examination report detection result.

[0128] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0138] 1) A method for detecting abnormalities in physical examination reports based on OCR and natural language model technology in the present application includes: obtaining physical examination report materials to be identified, wherein the physical examination report materials to be identified are in the form of pictures; using a layout analysis algorithm based on the layout type to classify and process the physical examination report materials to be identified, and obtaining classified materials, wherein the layout types include images, text, and tables; according to the layout type of the classified materials, using OCR recognition technology and a natural language model to analyze and process the classified materials, generate physical examination report detection results, and determine whether there are physical examination abnormalities in the physical examination report materials to be identified based on the physical examination report detection results. In this way, the digitization process of the physical examination report can be greatly accelerated, and at the same time, the abnormal data in the physical examination report can be quickly screened, and key information can be delivered to medical personnel. It solves the problem that the data extracted from the report by the existing technology is incomplete, resulting in a poor user experience.

[0139] 2) A physical examination report abnormality detection device based on OCR and natural language model technology of the present application includes: an acquisition unit, used to acquire physical examination report materials to be identified, wherein the physical examination report materials to be identified are in the form of pictures; a classification processing unit, used to classify and process the physical examination report materials to be identified based on the layout type using a layout analysis algorithm to obtain classified materials, wherein the layout types include images, text and tables; an analysis processing unit, used to analyze and process the classified materials according to the layout type of the classified materials using OCR recognition technology and a natural language model, generate a physical examination report detection result, and determine whether there is a physical examination abnormality in the physical examination report materials to be identified based on the physical examination report detection result. In this way, the digitization process of the physical examination report can be greatly accelerated, and at the same time, the abnormal data in the physical examination report can be quickly screened, and key information can be delivered to medical personnel. It solves the problem that the data extracted from the report in the prior art is incomplete, resulting in a poor user experience.

[0140] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting abnormalities in physical examination reports based on OCR and natural language model technology, characterized in that: include: Obtaining the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture; Classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables; According to the layout type of the classified material, the classified material is analyzed and processed using OCR recognition technology and a natural language model to generate a physical examination report detection result, and based on the physical examination report detection result, it is determined whether the physical examination report material to be identified has any physical examination abnormalities.

2. The method according to claim 1, characterized in that: According to the layout type of the classified material, the classified material is analyzed and processed using OCR recognition technology and a natural language model to generate a physical examination report test result, including: In the case where it is determined that the layout type of the classified material is the image, using the OCR recognition technology to extract context information of the classified material in the physical examination report material to be recognized; The context information is input into the natural language model to generate the physical examination report detection result.

3. The method according to claim 1, characterized in that According to the layout type of the classified material, the classified material is analyzed and processed using OCR recognition technology and a natural language model to generate a physical examination report test result, including: When it is determined that the layout type of the classified material is the table, the OCR recognition technology is used to identify and process each detection item in the classified material to obtain information of each detection item; Matching each of the test item information with the standard test item information to generate a matching result; The abnormal items in the detection item information are marked according to the matching results to generate the detection results of the physical examination report.

4. The method according to claim 1, characterized in that According to the layout type of the classified material, the classified material is analyzed and processed using OCR recognition technology and a natural language model to generate a physical examination report test result, including: When it is determined that the layout type of the classified material is text, the classified material is extracted and processed by using the OCR recognition technology to generate text information; The natural language model is used to extract and process the abnormal text description in the text information to generate abnormal description information; The content of the text information is marked according to the abnormal description information to generate the physical examination report detection result.

5. The method according to claim 1, characterized in that According to the layout type of the classified material, the classified material is analyzed and processed using OCR recognition technology and a natural language model to generate a physical examination report test result, including: According to the layout type of each of the classified materials, respectively, the classified materials are analyzed and processed using OCR recognition technology and natural language model to generate image detection results, table detection results and text detection results; The image detection result, the table detection result and the text detection result are integrated and processed to generate the physical examination report detection result.

6. The method according to claim 1, characterized in that After analyzing and processing the classified materials according to the layout type of the classified materials by using OCR recognition technology and natural language model to generate a physical examination report test result, the method further includes: Sending the test results of the physical examination report to the review end; When the audit passing information sent by the audit terminal is received, the physical examination report detection result and the physical examination report materials to be identified are sent to the designated user terminal.

7. The method according to claim 6, characterized in that After sending the test results of the physical examination report to the review end, the method further includes: Obtaining the review time for sending the test results of the physical examination report to the review end, and determining whether the review time exceeds a preset time; When the review time exceeds the preset time, the physical examination report materials to be identified are sent to the designated user terminal.

8. A device for detecting abnormalities in physical examination reports based on OCR and natural language model technology, characterized in that: include: An acquisition unit, used for acquiring the physical examination report material to be identified, wherein the physical examination report material to be identified is in the form of a picture; A classification processing unit, used for classifying the physical examination report materials to be identified by using a layout analysis algorithm based on the layout type to obtain classified materials, wherein the layout type includes images, texts and tables; The analysis and processing unit is used to analyze and process the classified materials according to the layout type of the classified materials by using OCR recognition technology and natural language model, generate a physical examination report detection result, and determine whether there are any physical examination abnormalities in the physical examination report materials to be identified according to the physical examination report detection result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for detecting abnormalities in physical examination reports based on OCR and natural language model technology as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the physical examination report abnormality detection method based on OCR and natural language model technology as described in any one of claims 1 to 7.

Images