Character acquisition, page processing and knowledge graph construction method, device and medium

Abstract

A method and device for acquiring character information in a picture, a non-transitory storage medium, a page processing method, and a knowledge graph construction method. The method for acquiring character information in a picture comprises: acquiring a picture and extracting at least one character information included in the picture; and verifying the at least one character information based on a knowledge graph. The method for acquiring character information in a picture can improve the accuracy of the acquired character information by verifying the at least one character information extracted based on a knowledge graph.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a method and device for acquiring character information in a picture, a storage medium, a page processing method, and a knowledge graph construction method. Background Art

[0002] Patients (or examinees) will do a lot of medical examinations to confirm whether they are healthy. The results of medical examinations serve as auxiliary information for doctors to make diagnoses. However, after obtaining paper documents (test reports, prescriptions, medical records, physical examination reports, etc.), patients may throw away paper documents or store them in a corner of their home. Therefore, the preservation rate of patients' historical medical examination data is low and difficult to reuse. In addition, paper documents and their images are unstructured data, which is not easy to view and manage.

[0003] Optical Character Recognition (OCR) technology can directly convert text content on images and photos into editable text. For example, OCR technology can extract text content on images in JPG, PNG, GIF, BMP, DOC and other formats. Summary of the invention

[0004] At least one embodiment of the present disclosure provides a method for acquiring character information in a picture, which includes: acquiring a picture and extracting at least one item of character information included in the picture; and verifying the at least one item of character information based on a knowledge graph.

[0005] For example, in at least one example of the acquisition method, the verifying of at least one item of character information based on the knowledge graph includes: confirming that there is erroneous character information in the at least one item of character information based on the knowledge graph; and correcting the erroneous character information based on the knowledge graph.

[0006] For example, in at least one example of the acquisition method, the at least one item of character information includes multiple items of character information. The confirming that there is an error in the at least one item of character information based on the knowledge graph includes: obtaining multiple entities according to the multiple items of character information of the image, and selecting an entity to be verified from the multiple entities to determine whether there is an error in the character information corresponding to the entity to be verified; and confirming whether there is an error in the character information corresponding to the entity to be verified according to the hierarchical structure of the knowledge graph, and if there is an error in the character information corresponding to the entity to be verified, confirming that the character information corresponding to the entity to be verified is the character information with the error.

[0007] For example, in at least one example of the acquisition method, confirming whether there is an error in the character information corresponding to the entity to be verified based on the hierarchical structure of the knowledge graph includes: grading the multiple entities according to the hierarchical structure of the knowledge graph; determining the level of the entity to be verified in the hierarchical structure of the knowledge graph; calculating the similarity between the entity to be verified and all entities that have the same level and the same relationship with the entity to be verified in the knowledge graph to obtain multiple entity similarities related to the entity to be verified; and when the maximum value among the multiple entity similarities is less than a predetermined entity similarity threshold, determining that the entity to be verified is an entity to be verified with an error and that the character information corresponding to the entity to be verified has an error.

[0008] For example, in at least one example of the acquisition method, a minimum compilation distance algorithm is used to obtain the similarity between the entity to be verified and all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified.

[0009] For example, in at least one example of the acquisition method, the correction of the erroneous character information based on the knowledge graph includes: determining the number of all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified to obtain the number of entities. When the number of entities is equal to 1, directly replace the character information corresponding to the entity to be verified with the character information corresponding to the entity with the same level and the same relationship with the entity to be verified, or calculate the probability that the entity with the same level and the same relationship with the entity to be verified is the entity to be verified with the error to obtain the entity probability, and when the entity probability is greater than a predetermined entity probability threshold, replace the character information corresponding to the entity to be verified with the character information corresponding to the entity with the same level and the same relationship with the entity to be verified. When the number of entities is greater than 1, perform the following steps: determine at least two candidate entities based on the multiple entity similarities; calculate the probability that each of the at least two candidate entities is the entity to be verified with the error to obtain the candidate probability of each of the at least two candidate entities; and replace the character information corresponding to the entity to be verified with the character information corresponding to the candidate entity corresponding to the maximum candidate probability.

[0010] For example, in at least one example of the acquisition method, determining at least two candidate entities based on the multiple entity similarities includes: sorting in descending order all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified based on the multiple entity similarities, and selecting a predetermined number of entities at the front of the sequence as the at least two candidate entities; or sorting in ascending order all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified based on the multiple entity similarities, and selecting a predetermined number of entities at the end of the sequence as the at least two candidate entities.

[0011] For example, in at least one example of the acquisition method, the calculating the probability that each of the at least two candidate entities is the entity to be verified with errors includes: determining whether the entity to be verified includes a next-level entity to be verified, so as to obtain a first determination result, wherein the next-level entity to be verified is all next-level entities subordinate to the entity to be verified; determining whether the entity to be verified corresponds to a related entity to be verified, so as to obtain a second determination result, wherein the related entity to be verified is all entities that are located at the same level as the entity to be verified in the hierarchical structure of the knowledge graph, are associated with the entity to be verified, and have different relationships with the upper-level entity to which the entity to be verified belongs; and selecting a method for calculating the probability that each of the at least two candidate entities is the entity to be verified with errors based on the first determination result and the second determination result.

[0012] For example, in at least one example of the acquisition method, the first judgment result is the first judgment sub-result when it is determined that the entity to be verified includes the next-level entity to be verified, or the second judgment sub-result when it is determined that the entity to be verified does not include the next-level entity to be verified; and the second judgment result is the third judgment sub-result when it is determined that the entity to be verified corresponds to the related entity to be verified, or the fourth judgment sub-result when it is determined that the entity to be verified does not correspond to the related entity to be verified.

[0013] For example, in at least one example of the acquisition method, the method for selecting based on the first judgment result and the second judgment result for calculating the probability that each of the at least two candidate entities is the entity to be verified with errors includes: when the first judgment result is a first judgment sub-result and the second judgment result is a third judgment sub-result, using the next-level entity to be verified and / or the related entity to be verified to calculate the candidate probability of each of the at least two candidate entities; when the first judgment result is a second judgment sub-result and the second judgment result is a third judgment sub-result, using the related entity to be verified to calculate the candidate probability of each of the at least two candidate entities; when the first judgment result is a first judgment sub-result and the second judgment result is a fourth judgment sub-result, using the next-level entity to be verified to calculate the candidate probability of each of the at least two candidate entities; and when the first judgment result is a second judgment sub-result and the second judgment result is a fourth judgment sub-result, using the entity similarity of each of the at least two candidate entities as the probability that each of the at least two candidate entities is the entity to be verified with errors.

[0014] For example, in at least one example of the acquisition method, the use of the relevant entities to be verified to calculate the candidate probability of each of the at least two candidate entities includes: obtaining the relevant entities to be verified corresponding to the entity to be verified; performing the following method for each entity included in the relevant entity to be verified: obtaining relevant candidate entities, wherein the relevant candidate entity and each of the at least two candidate entities are located at the same level in the hierarchical structure of the knowledge graph, are associated with each of the at least two candidate entities, and the relationship between the relevant candidate entity and the upper-level entity to which each of the at least two candidate entities belongs is equal to the relationship between each of the entities included in the relevant entity to be verified and the entity to be verified, calculating the similarity between each entity included in the relevant entity to be verified and the relevant candidate entity; obtaining the relevant similarity of each of the at least two candidate entities based on the similarity between each of the entities included in the relevant entity to be verified and the relevant candidate entity; and obtaining the candidate probability of each of the at least two candidate entities based on the relevant similarity and entity similarity of each of the at least two candidate entities.

[0015] For example, in at least one example of the acquisition method, the acquiring the relevant similarity of each of the at least two candidate entities based on the similarity between each of the entities included in the relevant entities to be verified and the relevant candidate entity includes: determining the number of entities included in the relevant entities to be verified; when the number of entities included in the relevant entities to be verified is equal to 1, taking the similarity between the entities included in the relevant entities to be verified and the relevant candidate entity as the relevant similarity of each of the at least two candidate entities; when the number of entities included in the relevant entities to be verified is greater than 1, taking the weighted sum of multiple similarities between the entities included in the relevant entities to be verified and the relevant candidate entity as the relevant similarity of each of the at least two candidate entities; the acquiring the candidate probability of each of the at least two candidate entities based on the relevant similarity and entity similarity of each of the at least two candidate entities includes: taking the weighted sum of the relevant similarity and entity similarity of each of the at least two candidate entities as the candidate probability of each of the at least two candidate entities.

[0016] For example, in at least one example of the acquisition method, the calculation of the candidate probability of each of the at least two candidate entities using the next-level entities to be verified includes: obtaining the next-level entities to be verified included in the entities to be verified; dividing the next-level entities to be verified into at least one group of next-level entity groups to be verified based on the relationship between the entities to be verified and each entity in the next-level entities to be verified; performing the following method for each of the at least one group of next-level entity groups to be verified: obtaining a next-level candidate entity group, wherein the next-level candidate entity group is all entities that belong to each of the at least two candidate entities and whose relationship with each of the at least two candidate entities is equal to the relationship between each of the entities included in the next-level entity group to be verified and the entity to be verified; calculating the similarity between each group of the at least one group of next-level entity groups to be verified and the next-level candidate entity group to obtain at least one group similarity corresponding to the at least one group of next-level entity groups to be verified; obtaining the next-level similarity of each of the at least two candidate entities based on the at least one group similarity; and calculating the candidate probability of each of the at least two candidate entities based on the next-level similarity of each of the at least two candidate entities and the entity similarity.

[0017] For example, in at least one example of the acquisition method, the acquisition of at least one group similarity corresponding to the at least one group of next-level entity groups to be verified includes: acquiring the maximum similarity of each of the multiple entities included in each group of the at least one group of next-level entity groups to be verified, so as to acquire multiple maximum similarities of the multiple entities included in each group of the at least one group of next-level entity groups to be verified, wherein the maximum similarity is the largest similarity among the similarities between each of the multiple entities included in each group of the at least one group of next-level entity groups to be verified and all entities included in the next-level candidate entity group; and acquiring at least one group similarity corresponding to the at least one group of next-level entity groups to be verified based on the multiple maximum similarities.

[0018] For example, in at least one example of the acquisition method, acquiring at least one group similarity corresponding to the at least one group of next-level entity groups to be verified based on the multiple maximum similarities includes: taking the weighted sum of the multiple maximum similarities as the at least one group similarity corresponding to the at least one group of next-level entity groups to be verified.

[0019] For example, in at least one example of the acquisition method, the acquiring of the next-level similarity of each of the at least two candidate entities based on the at least one group similarity includes: determining the number of next-level entity groups included in the next-level entity to be verified, and when the number of the next-level entity groups to be verified is equal to 1, using the group similarity of the next-level entity groups included in the next-level entity to be verified as the next-level similarity of each of the at least two candidate entities; and when the number of the next-level entity groups to be verified is greater than 1, using the weighted sum of multiple group similarities of the next-level entity groups included in the next-level entity to be verified as the next-level similarity of each of the at least two candidate entities; the calculating of the candidate probability of each of the at least two candidate entities based on the next-level similarity of each of the at least two candidate entities and the entity similarity includes: using the product of the next-level similarity based on each of the at least two candidate entities and the entity similarity as the candidate probability of each of the at least two candidate entities.

[0020] For example, in at least one example of the acquisition method, the calculation of the candidate probability of each of the at least two candidate entities using the next-level entity to be verified and the related entity to be verified includes: obtaining the related entity to be verified corresponding to the entity to be verified; performing the following method for each entity included in the related entity to be verified: obtaining the related candidate entity, wherein the related candidate entity and each of the at least two candidate entities are at the same level in the hierarchical structure of the knowledge graph, are associated with each of the at least two candidate entities, and the relationship between the related candidate entity and the upper-level entity to which each of the at least two candidate entities belongs is equal to the relationship between each of the entities included in the related entity to be verified and the entity to be verified, calculating the similarity between each of the entities included in the related entity to be verified and the related candidate entity; obtaining the related similarity of each of the at least two candidate entities based on the similarity between each of the entities included in the related entity to be verified and the related candidate entity; obtaining the next-level entity included in the entity to be verified The method comprises the steps of: obtaining a next-level candidate entity group, wherein the next-level candidate entity group is composed of all entities belonging to each of the at least two candidate entities and having a relationship with each of the at least two candidate entities equal to a relationship between each of the entities included in the next-level entity group and the entity to be verified; calculating a similarity between each of the at least one group of next-level entity groups and the next-level candidate entity group to obtain at least one group similarity corresponding to the at least one group of next-level entity groups to be verified; obtaining a next-level similarity of each of the at least two candidate entities based on the at least one group similarity; and obtaining a candidate probability of each of the at least two candidate entities based on the relevant similarity, the next-level similarity and the entity similarity of each of the at least two candidate entities.

[0021] At least one embodiment of the present disclosure further provides a device for acquiring character information in a picture, comprising: a character information extraction device and a verification and correction device. The character information extraction device is configured to acquire a picture and extract at least one item of character information included in the picture; and the verification and correction device is configured to verify the at least one item of character information based on a knowledge graph.

[0022] At least one embodiment of the present disclosure further provides a device for acquiring character information in a picture, comprising: a processor and a memory. The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes any acquisition method provided by at least one embodiment of the present disclosure.

[0023] At least one embodiment of the present disclosure further provides a non-transitory storage medium, which includes computer program instructions stored thereon. When the computer program instructions are executed by a processor, the computer executes any acquisition method provided by at least one embodiment of the present disclosure.

[0024] At least one embodiment of the present disclosure also provides a page processing method, which includes: obtaining a picture of a page; and obtaining the verified character information of the picture of the page by executing any acquisition method provided by at least one embodiment of the present disclosure; and structuring the verified character information of the picture of the page to obtain a structured data combination.

[0025] For example, in at least one example of the page processing method, the page processing method also includes: confirming, based on the knowledge graph, that the picture of the page includes a medical image; and adding the medical image to the structured data combination.

[0026] For example, in at least one example of the page processing method, confirming that the picture of the page includes a medical image based on the knowledge graph includes: determining that the picture of the page includes the medical image based on character information extracted from the picture of the page and verified based on the knowledge graph, including text corresponding to the medical image.

[0027] For example, in at least one example of the page processing method, the page processing method further includes: based on the knowledge graph and the multiple entities corresponding to the verified character information, adding descriptions corresponding to the multiple entities in the structured data combination.

[0028] At least one embodiment of the present disclosure also provides a knowledge graph construction method, which includes: obtaining the data structure of a medical knowledge graph; exporting corresponding entities from at least a medical image archiving and communication system and a laboratory information management system; aligning the entities exported from the medical image archiving and communication system with the entities exported from the laboratory information management system to obtain multiple entities after entity alignment; and filling the multiple aligned entities into the medical knowledge graph based on the data structure of the medical knowledge graph.

[0029] For example, in at least one example of the construction method, the knowledge graph construction method also includes: obtaining sub-item descriptions of clinical examination sub-items involved in the laboratory information management system; and processing the sub-item descriptions and filling them into the medical knowledge graph.

[0030] For example, in at least one example of the construction method, the knowledge graph construction method also includes: exporting corresponding entities from a hospital information system; aligning the entities exported from the hospital information system with multiple entities after the entity alignment; and filling the entities exported from the hospital information system and aligned with multiple entities after the entity alignment into the medical knowledge graph based on the data structure of the medical knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0032] Figure 1 is an exemplary flow chart of a method for acquiring character information in a picture provided by at least one embodiment of the present disclosure;

[0033] Figure 2A A schematic diagram showing a process of extracting at least one character information included in a picture using optical character recognition technology;

[0034] Figure 2B Another schematic diagram showing a process of extracting at least one item of character information included in a picture using optical character recognition technology;

[0035] Figure 3 is a data structure of a medical knowledge graph provided by at least one embodiment of the present disclosure;

[0036] Figure 4 This is an example of a blood test report;

[0037] Figure 5A is a schematic flow chart of verifying at least one item of character information based on a knowledge graph provided by at least one embodiment of the present disclosure;

[0038] Figure 5B is a schematic flowchart of parsing based on a knowledge graph provided by at least one embodiment of the present disclosure;

[0039] Fig. 6A is a schematic flowchart of verifying character information in an image and adding explanations based on a knowledge graph provided by at least one embodiment of the present disclosure;

[0040] Figure 6B is a schematic flowchart of medical record analysis based on knowledge graph provided by at least one embodiment of the present disclosure;

[0041] Fig. 7A This is a picture corresponding to a part of the blood routine examination form;

[0042] Figure 7B yes Fig. 7A The picture shown is of a portion of the blood test sheet after it has been contaminated;

[0043] Figure 7C A software program that uses a method for obtaining character information in a picture provided by at least one embodiment of the present disclosure to obtain character information in a picture Figure 7B The result obtained by character information in the picture;

[0044] Figure 8 A device for acquiring character information in a picture provided by at least one embodiment of the present disclosure;

[0045] Fig. 9 Another device for acquiring character information in a picture provided by at least one embodiment of the present disclosure;

[0046] Fig.10 is a non-transitory storage medium provided by at least one embodiment of the present disclosure;

[0047] Fig.11 is an exemplary flow chart of a page processing method provided by at least one embodiment of the present disclosure;

[0048] Fig.12 A method for constructing a medical knowledge graph provided by at least one embodiment of the present disclosure;

[0049] Fig.13 An exemplary scene diagram of a system for acquiring character information in a picture provided by at least one embodiment of the present disclosure is shown; and

[0050] Fig.14 A computing device provided by at least one embodiment is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Unless otherwise defined, the technical terms or scientific terms used in this disclosure should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0053] The inventors of the present disclosure have noticed in their research that although optical character recognition (OCR) technology can be used to extract text content from images, the text content extracted from images by OCR technology is only the characters themselves, and parsing technology (e.g., page parsing technology) is also needed to integrate the extracted characters into a format consistent with the image content and convenient for users to view according to rules, so as to obtain information with specific meanings. For example, ordinary optical character recognition technology and parsing technology can only extract text content from images, and convert the text content in the image into a text format, so that the text content in the image can be stored in a memory in a text format (that is, ordinary OCR technology and parsing technology only solve the problem of electronicization and storage of medical examination reports). However, the computer has a poor understanding ability of the text content in the above text format, and cannot effectively retrieve and display (e.g., display using a graphical user interface) the above content, so that the patient's medical examination report still cannot be effectively used.

[0054] In addition, the inventors of the present disclosure also noticed in their research that paper documents in the medical field (test reports, prescriptions, medical records, physical examination reports, etc.) involve uncommon characters, uncommon words, etc., and the characters of these uncommon characters and uncommon words are complex. Therefore, the OCR module trained using the usual method (that is, the ordinary OCR module) may have difficulty in recognizing these uncommon characters and uncommon words. The inventors of the present disclosure also noticed in their research that the relevant medical OCR module has poor recognition performance for non-medical texts. Therefore, on the whole, the accuracy of the ordinary OCR module and the medical OCR module in extracting character information from pictures corresponding to paper documents in the medical field is low. Since the accuracy of the extracted characters is closely related to the physical health of the patient, there is an urgent need for a method to improve the accuracy of the characters obtained by extraction.

[0055] Knowledge graph is a frontier research issue of intelligent big data. It conforms to the development of the information age with its unique technical advantages. For example, knowledge graph has the following advantages: incremental data model design; good data integration capability; supported by existing standards such as Resource Description Framework (RDF) and Web Ontology Language (OWL); semantic search and knowledge reasoning capabilities, etc. For example, knowledge graph has better performance in querying complex related information, can understand user intentions from a semantic level, and can have a strong descriptive ability for data. Knowledge graph can be used to describe various entities and concepts that exist in the real world. For example, knowledge graph can be used to describe the association between two entities. For example, "Yao Ming was born in Shanghai" involves two entities "Yao Ming" and "Shanghai" and the relationship between the two entities "birthplace".

[0056] In the field of medicine, with the development of regional health informatization and medical information systems, a huge amount of medical data has been accumulated. How to extract information from these data and manage, share and apply them is a key issue in promoting medical intelligence, and is also the basis for medical knowledge retrieval, clinical diagnosis, medical quality management, and intelligent processing of electronic medical records and health records. Medical knowledge graphs are a way to achieve smart medical care, which is conducive to more efficient and accurate medical services. Based on the reasoning ability of knowledge graphs, medical knowledge graphs have the advantages of strong interpretability and stability. However, due to the following problems, the application scope of medical graphs is relatively small (for example, only used for knowledge question answering and retrieval): there are few existing Chinese medical knowledge graphs, medical terminology in different regions is not unified, and the construction of medical knowledge graphs requires the deep participation of medical experts, resulting in low efficiency, many restrictions, and poor scalability of medical knowledge graphs.

[0057] Edit distance refers to the minimum number of edit operations required to transform one string into another. Allowed edit operations include replacing one character with another, inserting a character, and deleting a character. Generally speaking, the smaller the edit distance between two strings, the greater the similarity between the two strings.

[0058] For example, the minimum edit distance algorithm can use edit[i][j] to represent the edit distance between a string consisting of i characters of string A and a string consisting of j characters of string B. edit[i][j] represents the edit distance between a string consisting of the 0th character to the ith character of string A and a string consisting of the 0th character to the jth character of string B. For example, when string A is "graph" and string B is "grape", edit[5[5] represents the edit distance between the string "graph" and the string "grape"; edit[2[3] represents the edit distance between the string "gr" and the string "gra". For example, i and j can be greater than or equal to 1.

[0059] For example, the edit distance between string A and string B can also be represented by dis[i][j]. For example, dis[0][0] means that when string A and string B are both empty, their edit distance is 0. Obviously, dis[0][j] is the case where string A is empty and string B is j in length. At this time, their edit distance is j, that is, the minimum edit distance from an empty string to string B after adding j characters is j; similarly, dis[i][0] means that when string A is i in length and string B is empty, string A needs to delete i characters to be empty, so the minimum edit distance from string A to string B is i.

[0060] Through the above description, we can get the following dynamic programming equation.

[0061]

[0062] Among them, the effective number of replacement flags can be expressed using the following equation.

[0063]

[0064] The min() function (i.e., the function used to calculate the minimum value) in the above dynamic programming equation contains three expressions, namely, edit[i-1][j]+1, edit[i][j-1]+1, and edit[i-1][j-1]+flag. The above three expressions represent the operation of inserting characters, deleting characters, and replacing characters, respectively, as described below.

[0065] edit[i-1][j]+1 is equivalent to inserting the last character of string A after the last character of string B. The above character insertion operation increases the compilation distance by 1. Then edit[i-1][j] is calculated to calculate edit[i-1][j]+1 as part of the calculation of the compilation distance edit[i][j].

[0066] edit[i][j-1]+1 is equivalent to deleting the last character of string B. The above character deletion operation increases the compilation distance by 1. Then edit[i][j-1] is calculated to calculate edit[i][j-1]+1 as part of the calculation of the compilation distance edit[i][j].

[0067] edit[i-1][j-1]+flag is equivalent to replacing the last character of string B with the last character of string A. Here, flag is used to mark the effective number of character replacements.

[0068] For example, the similarity of the two strings can be obtained by using the edit distance of the two strings calculated using the minimum edit distance algorithm. For example, the similarity of the two strings can be calculated using the following formula and the edit distance of the two strings: S(A, B) = 1-(DIS(A, B) / max(Len(A), Len(B)). Here, DIS(A, B) represents the edit distance between string A and string B, Len(A) and Len(B) represent the string length of A and the string length of B respectively, and max(Len(A), Len(B)) is equal to the maximum value of the string length of A and the string length of B.

[0069] At least one embodiment of the present disclosure provides a method and device for acquiring character information in a picture, a non-transitory storage medium, a page processing method, and a method for constructing a medical knowledge graph. The method for acquiring character information in a picture includes: acquiring a picture and extracting at least one character information included in the picture; and verifying the at least one character information based on a knowledge graph. The method for acquiring character information in a picture verifies the at least one character information extracted based on a knowledge graph, thereby improving the accuracy of the acquired character information.

[0070] The following is a non-restrictive description of the method for obtaining character information in a picture provided according to an embodiment of the present disclosure through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present disclosure.

[0071] Figure 1is an exemplary flow chart of a method for obtaining character information in a picture provided by at least one embodiment of the present disclosure. Figure 1 As shown, the method for obtaining character information in the picture includes the following steps S10 and S20.

[0072] Step S10: Acquire a picture and extract at least one item of character information included in the picture.

[0073] Step S20: verify at least one item of character information based on the knowledge graph.

[0074] In step S10, the specific method for acquiring the picture can be set according to the actual application requirements, and at least one embodiment of the present disclosure does not specifically limit this. In one example, acquiring the picture may include acquiring (reading) the picture stored in the memory from the memory. In another example, acquiring the picture may also include remotely acquiring the picture, such as using an information sending and receiving device to receive the picture. For example, the information sending and receiving device can receive the picture from a client (e.g., a mobile phone terminal or a PC).

[0075] For example, in step S10, the acquired picture may be a picture after picture preprocessing or a picture before picture preprocessing. For example, in the case where the acquired picture is a picture before picture preprocessing, the picture may be further preprocessed before extracting at least one item of character information included in the picture. For example, picture preprocessing includes, but is not limited to, at least one of detection and extraction of the main area of ​​the picture, picture correction, picture segmentation, text direction correction of the picture, picture binarization, and picture denoising.

[0076] For example, the detection and extraction of the main area of ​​the image includes detecting the main area of ​​the above-mentioned image (valid area, area containing useful information), and extracting the detected main area as the image for subsequent processing. For example, the main area of ​​the image can be detected by locating the area where text exists in the image (Text Detection), that is, finding the bounding box (Bounding Box) of the word or text line (Word / Line-level). For example, in the detection and extraction of the main area of ​​the image, the user can confirm the detected main area before extraction (that is, manually select the valid area by the user) to improve the accuracy of detection and extraction of the main area of ​​the image.

[0077] For example, picture correction (eg, trapezoidal correction) is used to correct the deformation of a picture (eg, a picture taken by a mobile phone). For example, gamma correction can be used to implement picture correction (eg, trapezoidal correction).

[0078] For example, image segmentation is used to segment text (e.g., text in a physical examination report) from a picture containing a background (e.g., a photo taken by an ordinary mobile phone) to improve the accuracy of text (e.g., character information) extracted from the picture. For example, text can be segmented from a picture containing a background by the following method.

[0079] First, the image is horizontally projected to find the upper and lower limits of each row; second, the rows are cut along the upper and lower limits of each row to obtain multiple rows; third, for each row obtained by cutting, a vertical projection is performed to find the left and right boundaries of each character; fourth, a single character is cut along the left and right boundaries of each character. For example, horizontal projection is used to count the elements of each row of a picture (that is, count in the horizontal direction), and then, a statistical result graph can be drawn based on the above statistical results, thereby determining the starting point and end point of each row in the column direction, that is, the upper and lower limits of each row. For example, the projection direction of the vertical projection is downward, which is used to count the number of elements in each column and the projection position of the elements in each column, thereby determining the left and right limits of each column.

[0080] For example, the horizontal projection of the picture can be achieved by the following method: first, set a straight line (axis) extending in the vertical direction; second, project the pixels on the picture onto the straight line (axis) in a direction perpendicular to the straight line (axis). After the projection, multiple pixels on the picture correspond to multiple black dots on the straight line (axis); third, count the number of black dots of the pixels on the picture perpendicular to the vertical straight line, and add up the sum as the value of the position of the axis. After completing the above horizontal projection, cutting based on the picture projection can be performed. Specifically, after the picture is mapped and the relevant features are obtained, the cutting position (coordinates) of the picture is determined based on the above features, and the above coordinates are used to cut the above picture to obtain the target picture (for example, a picture of a single text obtained by segmenting the picture). For example, vertical projection of a picture is similar to horizontal projection of a picture, except that a straight line extending in the vertical direction is set, and the pixels on the picture are projected onto the straight line extending in the vertical direction.

[0081] The following is an example to illustrate horizontal projection. For example, horizontal projection can be achieved by the following method. First, binarize the image so that the text is black (for example, the pixel value corresponding to the text is Lb, for example, Lb=0) and the background is white (for example, the pixel value corresponding to the text is Lw, for example, Lw=255). Secondly, perform the following steps for each row of the image: determine in turn whether the pixel value of each column is equal to Lb. If the pixel value is equal to Lb, it is determined that the pixel located in the row and column is very similar to text; count the number of all pixels in the row that are very similar to text. Assuming that there are n pixels in the row whose pixel values ​​are equal to Lb, set the row from the first column to the nth column to black.

[0082] For example, the text direction correction of an image is used to correct the tilt of the text in the image caused by the image shooting process. For example, the image can be corrected using the Fast Fourier Transformation (FFT) technology, and then the text with the direction correction is obtained by inversely transforming the corrected image.

[0083] For example, image binarization can be used to improve the contrast of text. For example, image binarization can be achieved by the following method. First, a threshold is set; then, for multiple pixels of the image, the values ​​of pixels whose pixel values ​​(for example, the grayscale value of the pixel) are greater than or equal to the threshold are set to 1 (that is, the pixel is rendered white), and the values ​​of pixels whose pixel values ​​are less than the threshold are set to 0 (that is, the pixel is rendered black). For example, adaptive binarization can be used (different thresholds are used for each pixel in the comparison process). For example, a two-dimensional matrix corresponding to the pixel values ​​of multiple pixels of the image (that is, a matrix for storing the pixel values ​​of multiple pixels of the image) can be converted into a vector, and then each pixel is compared with a threshold value (that is, an adaptive threshold value) calculated based on the pixels within a certain range.

[0084] For example, image denoising is used to reduce the noise of an image to improve the accuracy of text extracted from the image. For example, after the received image is converted into a binary image, there may be a lot of noise in the binary image; in this regard, median filtering or mean filtering can be used to denoise the binary image.

[0085] For example, the character information obtained from the image may include single character information or multiple character information. For example, each character information includes a single character or a string. For example, the character information may be text information or symbol information. For example, the symbol information may be an upward arrow symbol, a downward arrow symbol, etc.

[0086] For example, optical character recognition technology may be used to extract at least one item of character information included in the image. Figure 2AA schematic diagram showing a process of extracting at least one item of character information included in a picture using optical character recognition technology.

[0087] For example, Figure 2A As shown, the process of extracting at least one character information included in the picture by optical character recognition technology includes: text detection, text recognition and result output. For example, text detection is used to detect a single character in the picture. For example, text recognition is used to convert the detected text into a text format. For example, text recognition includes text feature extraction and contrast recognition, or text recognition can also be performed using, for example, a trained neural network (for example, a connected text proposal network or a convolutional recurrent neural network for detecting text in a natural image). For example, contrast recognition is used to compare the features obtained by text feature extraction with a comparison database or a feature database (containing a character set of all texts to be recognized), thereby finding text in a text format corresponding to the text in the picture based on the features of the text in the picture. For example, the specific method of using optical character recognition technology for text detection and text recognition can be found in the relevant technology, which will not be repeated here. For example, text detection can be achieved by detecting the coordinates of the upper and lower boundaries of the text rows in the picture and the coordinates of the left and right boundaries of the text columns. For example, CTPN (Detecting Text in Natural Image with Connectionist Text Proposal Network) and / or CRNN (Convolutional Recurrent Neural Network) can be used for text recognition, and an OCR recognition open source library (e.g., Tesseract digital recognition library) can be used for Chinese character recognition training.

[0088] Figure 2B Another schematic diagram of the process of extracting at least one character information included in a picture using optical character recognition technology is shown. Figure 2B As shown, the process of extracting at least one item of character information included in the image by the optical character recognition technology may also include: image input and image processing. For example, image processing refers to image preprocessing. The specific method of image preprocessing can be found in the relevant description of step S10, which will not be repeated here. For example, when the optical character recognition technology also includes image processing (for example, image preprocessing), the image obtained in step S10 may be an image before image preprocessing, and no additional image preprocessing is required before extracting at least one item of character information included in the image.

[0089] For example, in step S20, by verifying at least one item of character information based on the knowledge graph (for example, checking and correcting), it can be confirmed whether there are errors in the character information extracted from the image and the erroneous character information can be corrected, thereby improving the accuracy of the character information obtained by the method for obtaining character information in the image provided by at least one embodiment of the present disclosure.

[0090] For example, in step S20, the knowledge graph is a medical knowledge graph. For example, the medical knowledge graph may be a medical examination and testing knowledge graph. For example, the medical examination and testing knowledge graph may be used to provide relevant knowledge of test indicators and test orders. For example, the medical examination and testing knowledge graph may provide the possibility for accurate identification and analysis.

[0091] For example, the architecture (framework or data structure) of the medical knowledge graph can be used Figure 3 For example, the medical knowledge graph construction method provided by at least one embodiment of the present disclosure can be used and based on Figure 3 The medical knowledge graph is constructed by using the data structure of the medical knowledge graph shown in the figure, and the medical knowledge graph obtained by the construction can be applied to step S20. The method for constructing a medical knowledge graph provided by at least one embodiment of the present disclosure will be Fig.12 The embodiment shown is described in detail and will not be repeated here.

[0092] For example, based on Figure 3 The medical knowledge graph obtained by the data structure of the medical knowledge graph shown may include multiple entities and relationships derived from a medical picture archiving and communication (PACS) system and a laboratory information management (LIS) system. For example, by making the medical knowledge graph include entities derived from the PACS system, it is possible to confirm whether a picture contains a medical image based on the medical knowledge graph, and save the medical image if the picture contains the medical image.

[0093] For example, entities exported from the PACS system include clinical examination items, clinical examination sub-items, and entities related to medical imaging examinations in clinical examination sheets. For example, entities exported from the PACS system include: computer X-ray tomography images, nuclear magnetic resonance images (NMRI), ultrasound images, electrocardiograms, electroencephalograms, etc. For example, entities exported from the LIS system include clinical examination items, clinical examination sub-items, and entities not related to medical imaging examinations in clinical examination sheets. For example, entities exported from the LIS system include various laboratory examination items. For example, laboratory examination items include blood examination items, urine examination items, stool examination items, etc. For example, blood examination items include routine blood examination items, blood lipid examination items, blood sugar examination items, etc. For example, routine blood examination items include platelet count (PLT) sub-items, platelet distribution width (PDW) sub-items, and platelet hematocrit (PCT) sub-items.

[0094] For example, based on Figure 3 The medical knowledge graph obtained by the data structure of the medical knowledge graph shown may also include entities derived from the hospital information system and / or entities related to the sub-item description of the clinical examination sub-item. For example, the entities derived from the hospital information system include entities related to patient information and entities related to medical advice. For example, the entity related to the sub-item description of the clinical examination sub-item gives the description of the clinical examination sub-item and the meaning of the abnormality when the clinical examination sub-item results are abnormal, so that the patient can understand the examination results more clearly.

[0095] It should be noted that the data structure of the knowledge graph used in step S20 is not limited to Figure 3 For example, the data structure of the knowledge graph used in step S20 may also not include Figure 3 The data structure of the knowledge graph shown includes the part corresponding to the hospital information system and / or the part corresponding to the medical significance of the clinical examination sub-items.

[0096] For example, in step S20, verifying at least one item of character information based on the knowledge graph includes the following step S210.

[0097] Step S210: confirming that there is erroneous character information in at least one item of character information based on the knowledge graph.

[0098] For example, the at least one item of character information includes multiple items of character information, and the erroneous character information includes character information that is incorrectly recognized during the recognition process (for example, "water" is incorrectly recognized as "small") and / or character information that is not recognized during the recognition process.

[0099] For example, in step S210, confirming that there is erroneous character information in at least one item of character information based on the knowledge graph includes the following steps S211 and 212.

[0100] Step S211: obtaining a plurality of entities according to the plurality of character information of the image, and selecting an entity to be checked from the plurality of entities to determine whether the character information corresponding to the entity to be checked has errors.

[0101] Figure 4 is an example of a blood test report. Figure 4 The picture of the blood routine test report shown in the figure, according to the multiple character information of the picture, the multiple entities obtained respectively may include: the name of the report, the sub-item name of the blood routine test (item name), the abbreviation of the sub-item (code), the sub-item test result (result), the unit of the sub-item test result (unit), whether the sub-item test result is abnormal (indicated by an upward or downward arrow), and the reference value of the sub-item test result (item reference value). For example, some sub-item names and corresponding abbreviations of the blood routine test include: platelet count (PLT), platelet distribution width (PDW) and platelet packed volume (PCT).

[0102] For example, after obtaining multiple entities according to multiple character information of the image, the multiple entities can be verified one by one until all entities included in the multiple character information of the image are verified. For example, platelet count can be selected as the entity to be verified.

[0103] Step 212: According to the hierarchical structure of the knowledge graph, confirm whether there are any errors in the character information corresponding to the entity to be verified, and if there are any errors in the character information corresponding to the entity to be verified, confirm that the character information corresponding to the entity to be verified is the character information with errors.

[0104] For example, in step 212, according to the hierarchical structure of the knowledge graph, confirming whether there are errors in the character information corresponding to the entity to be verified includes the following steps 2121-2124.

[0105] Step 2121: Classify multiple entities according to the hierarchical structure of the knowledge graph.

[0106] For example, in step 2121, multiple entities can be graded by analyzing the table structure in the image and the layout, size, boldness, etc. of the text in the image. For example, optical recognition technology can be used to determine whether there is a table in the image. If there is a table, the table structure in the image is further analyzed (or parsed). For example, the specific method of table structure analysis can be found in Figure 5A and Figure 5BFor example, the (relative) size of the text in the picture, whether it is bold, etc. can also be obtained through picture analysis, for example, partially implemented by using a classifier or a neural network.

[0107] For example, for Figure 4 The multiple entities in the picture of the blood routine test report shown can be graded using the following method.

[0108] First, the entity corresponding to the character string "blood routine test" located in the first line of the report, centered and in a larger font (for simplicity, the subsequent description in this disclosure directly uses "blood routine test" to represent it, and other entities are treated similarly) can be regarded as the first-level entity in the report; the entities corresponding to the "code", "item name (that is, the sub-item name of the blood routine test)", "result", "item reference value", etc. located in the first line of the table can be regarded as the second-level entities in the report; all entities located in the table and outside the first line of the table can be regarded as the third-level entities in the report.

[0109] Then, the level in the report can be associated with the level of the entity in the hierarchical structure of the knowledge graph. For example, if in the hierarchical structure of the knowledge graph, "blood routine examination", "item name (that is, the name of the sub-item of blood routine examination)" and "platelet count" are at the fifth, sixth and seventh levels respectively, then Figure 4 The "blood routine test" in the picture of the blood routine test report is at the fifth level in the hierarchical structure of the knowledge graph; Figure 4 The “code”, “item name (i.e., the sub-item name of the blood routine test)”, “result”, and “item reference value” in the picture of the blood routine test report are at the sixth level in the hierarchical structure of the knowledge graph; Figure 4 The "platelet count" in the picture of the blood routine test report is at the seventh level in the hierarchical structure of the knowledge graph.

[0110] For example, based on the knowledge graph, it can be known that the entities corresponding to "bed number" in the blood routine examination report do not belong to the entities related to the blood routine examination results. Therefore, the entities corresponding to "bed number" will be ignored in the process of grading multiple entities.

[0111] Step 2122: Determine the level of the entity to be verified in the hierarchical structure of the knowledge graph.

[0112] For example, in step 2122, the level of the entity to be verified in the hierarchical structure of the knowledge graph can be determined based on the grading result in step 2121. For example, when "platelet count" is selected as the entity to be verified, based on the grading result in step 2121, it can be determined that the level of the entity to be verified in the hierarchical structure of the knowledge graph is the seventh level.

[0113] Step 2123: Calculate the similarity between the entity to be verified and all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified, so as to obtain multiple entity similarities related to the entity to be verified.

[0114] For example, the similarity between the entity to be verified and all entities with the same level and the same relationship as the entity to be verified in the knowledge graph is between 0 and 1.

[0115] For example, in step 2123, "having the same relationship" means that the relationship between all entities with the same level and the same relationship as the entity to be verified in the knowledge graph and their parent entities is equal to the relationship between the entity to be verified and its parent entity.

[0116] For example, when "platelet count" is selected as the entity to be verified, all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified include "platelet count", "platelet distribution width", "platelet hematocrit" and "neutrophil percentage" etc.

[0117] For example, if there is no error in "platelet count", the similarity between the entity to be verified (i.e., "platelet count") and the "platelet count" in the knowledge graph is 1; the similarities between the entity to be verified (i.e., "platelet count") and the "platelet distribution width", "platelet hematocrit" and "neutrophil percentage" in the knowledge graph are, for example, 0.5, 0.6 and 0, respectively.

[0118] For example, if "platelet count" is mistakenly identified as "platelet counter X", the similarity between the entity to be verified (i.e., "platelet counter X") and the "platelet count", "platelet distribution width", "platelet hematocrit" and "neutrophil percentage" in the knowledge graph (i.e., the entity similarity of "platelet count", "platelet distribution width", "platelet hematocrit" and "neutrophil percentage") is, for example, 0.8, 0.5, 0.6 and 0, respectively. It should be noted that the "X" in "platelet counter X" indicates that the character is misidentified, and the "X" in other characters or entities also has a similar meaning, which will not be repeated.

[0119] For example, in step 2123, a minimum compilation distance algorithm may be used to obtain the similarity between the entity to be verified and all entities in the knowledge graph that have the same level and the same relationship as the entity to be verified.

[0120] Step 2124: When the maximum value among the multiple entity similarities is less than the predetermined entity similarity threshold, it is determined that the entity to be verified is an entity to be verified with an error and that the character information corresponding to the entity to be verified has an error.

[0121] For example, the predetermined entity similarity threshold can be set according to actual application requirements. For example, the predetermined entity similarity threshold can be set to 1. For another example, the predetermined entity similarity threshold can also be set to 0.95. For example, when the predetermined entity similarity threshold is set to 1, as long as all entities with the same level and the same relationship as the entity to be verified in the knowledge graph do not include the entity to be verified, the entity to be verified is determined to be an entity to be verified with an error and the character information corresponding to the entity to be verified is incorrect.

[0122] For example, when the similarities between the entity to be verified (i.e., "Platelet Counter X") and the "Platelet Count", "Platelet Distribution Width", "Platelet Hematocrit" and "Neutrophil Percentage" in the knowledge graph are, for example, 0.8, 0.5, 0.6 and 0 respectively, since the maximum value of multiple entity similarities (i.e., 0.8) is less than the predetermined entity similarity threshold (e.g., 1), it can be determined that there is an error in the character information corresponding to the entity to be verified and the entity to be verified is an erroneous entity to be verified.

[0123] For example, when the maximum value among the multiple entity similarities is greater than or equal to a predetermined entity similarity threshold, it is determined that there is no error in the character information corresponding to the entity to be verified.

[0124] For example, when the similarities between the entity to be verified (i.e., "platelet count") and the "platelet count", "platelet distribution width", "platelet hematocrit" and "neutrophil percentage" in the knowledge graph are 1, 0.5, 0.6 and 0, respectively, since the maximum value of multiple entity similarities (i.e., 1) is greater than or equal to the predetermined entity similarity threshold (e.g., 1), it can be determined that there is no error in the character information corresponding to the entity to be verified.

[0125] For example, in step S20, verifying at least one item of character information based on the knowledge graph also includes the following step 220.

[0126] Step S220: Correcting erroneous character information based on the knowledge graph.

[0127] For example, correcting erroneous character information based on the knowledge graph includes: determining the number of all entities in the knowledge graph that have the same level and the same relationship as the entity to be verified, to obtain the number of entities.

[0128] For example, correcting erroneous character information based on the knowledge graph also includes: when the number of entities is equal to 1, directly replacing the character information corresponding to the erroneous entity to be verified with the character information corresponding to the entity at the same level and the same relationship with the entity to be verified in the knowledge graph, or calculating the probability that the entity at the same level and the same relationship with the entity to be verified is the erroneous entity to be verified to obtain the entity probability, and when the entity probability is greater than a predetermined entity probability threshold, replacing the character information corresponding to the erroneous entity to be verified with the character information corresponding to the entity at the same level and the same relationship with the entity to be verified.

[0129] For example, the predetermined entity probability threshold can be set according to actual application requirements. For example, the predetermined entity probability threshold can be set to 0.5, 0.8, 0.9 or other applicable values. For example, when the entity probability of the candidate entity is less than or equal to the predetermined entity probability threshold, the erroneous character information can be manually corrected.

[0130] For example, correcting erroneous character information based on the knowledge graph also includes: when the number of entities is greater than 1, executing the following steps S221-S222.

[0131] Step S221: Determine at least two candidate entities based on multiple entity similarities.

[0132] For example, in step S221, determining at least two candidate entities based on multiple entity similarities includes: sorting in descending order all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified based on multiple entity similarities, and selecting a predetermined number of entities at the front of the sequence as at least two candidate entities; or sorting in ascending order all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified based on multiple entity similarities, and selecting a predetermined number of entities at the end of the sequence as at least two candidate entities.

[0133] For example, the predetermined number may be set according to actual application requirements, such as 2, 3, 10 or other applicable numbers.

[0134] For example, in step S221, “platelet count”, “platelet distribution width”, “hematocrit” and “neutrophil percentage” in the knowledge graph can be arranged in descending order based on multiple entity similarities (i.e., 0.8, 0.5, 0.6 and 0) into the following sequence, i.e., “platelet count”, “hematocrit”, “platelet distribution width” and “neutrophil percentage”. For example, when the predetermined number is set to 2, the two entities arranged at the front of the sequence, i.e., “platelet count” and “hematocrit”, can be used as at least two candidate entities.

[0135] For another example, in step S221, “platelet count”, “platelet distribution width”, “hematocrit” and “neutrophil percentage” in the knowledge graph can be sorted in ascending order based on multiple entity similarities (i.e., 0.8, 0.5, 0.6 and 0) into the following sequence, i.e., “neutrophil percentage”, “platelet distribution width”, “hematocrit” and “platelet count”. For example, when the predetermined number is set to 2, the two entities arranged at the end of the sequence, i.e., “hematocrit” and “platelet count” can be used as at least two candidate entities.

[0136] Step S222: Calculate the probability that each of the at least two candidate entities is an entity to be verified with errors, so as to obtain a candidate probability associated with each of the at least two candidate entities.

[0137] For example, the probability that each of at least two candidate entities is an entity to be verified with errors refers to the probability that the character information corresponding to each of the at least two candidate entities is equivalent to the character information corresponding to the entity to be verified with errors when it is not incorrectly recognized. For example, the probability that "hematocrit" is the entity to be verified with errors "platelet count X" refers to the probability that the character information corresponding to "hematocrit" is equivalent to the character information corresponding to "platelet count X" when it is not incorrectly recognized.

[0138] For example, in step S222, calculating the probability that each of the at least two candidate entities is an entity to be verified with errors includes the following steps S2221 to S2223.

[0139] Step S2221: Determine whether the entity to be verified includes a next-level entity to be verified to obtain a first determination result. The next-level entity to be verified is all next-level entities subordinate to the entity to be verified.

[0140] For example, for Figure 4In the picture of the blood routine examination report shown, when "Platelet Count X" is selected as the entity to be verified, it can be determined that the entity to be verified does not include the next level entity to be verified; when "Item Name X" is selected as the entity to be verified, it can be determined that the entity to be verified includes the next level entity to be verified, for example, "Item Name X" includes the next level entities to be verified including "Platelet Count", "Platelet Distribution Width", "Platelet Hematocrit" and "Neutrophil Percentage" etc.

[0141] For example, the first determination result is the first determination sub-result when it is determined that the entity to be verified includes the entity to be verified at the next level, or the second determination sub-result when it is determined that the entity to be verified does not include the entity to be verified at the next level. For example, when "project name X" is selected as the entity to be verified, the first determination result is the first determination sub-result; when "platelet count X" is selected as the entity to be verified, the first determination result is the second determination sub-result.

[0142] Step S2222: Determine whether the entity to be verified corresponds to the related entity to be verified to obtain a second determination result. The related entities to be verified are all entities that are at the same level as the entity to be verified in the hierarchical structure of the knowledge graph, are associated with the entity to be verified, and have different relationships with the upper-level entity to which the entity to be verified belongs. For example, the related entity to be verified is associated with the entity to be verified, which means that the related entity to be verified and the entity to be verified are used to describe different aspects of the same subject. For example, "platelet count (the full name of the platelet count inspection sub-item)" and "PLT (abbreviation of the platelet count inspection sub-item)", "100-300 (reference value of the platelet count inspection sub-item) are used to describe different aspects of the platelet count inspection sub-item, therefore, the entity to be verified "platelet count" is related to "PLT" and "100-300".

[0143] For example, for Figure 4 In the picture of the blood routine examination report, when "platelet count X" is selected as the entity to be verified, it can be determined that the entity to be verified corresponds to the relevant entity to be verified. For example, the relevant entities to be verified corresponding to "platelet count X" include "PLT (abbreviation of platelet count)", "100-300 (reference value of platelet count)", etc. When "project name X" is selected as the entity to be verified, it can be determined that the entity to be verified does not correspond to the relevant entity to be verified.

[0144] For example, the second determination result is the third determination sub-result when it is determined that the entity to be verified corresponds to the relevant entity to be verified or the fourth determination sub-result when it is determined that the entity to be verified does not correspond to the relevant entity to be verified. For example, when "project name X" is selected as the entity to be verified, the second determination result is the fourth determination sub-result; when "platelet count X" is selected as the entity to be verified, the second determination result is the third determination sub-result.

[0145] Step S2223: Selecting a method for calculating the probability that each of the at least two candidate entities is an entity to be verified with an error based on the first determination result and the second determination result.

[0146] For example, in step S2223, the method for calculating the probability that each of the at least two candidate entities is an entity to be verified with an error based on the first determination result and the second determination result includes one of the following steps S231-S234.

[0147] Step S232: when the first determination result is the second determination sub-result and the second determination result is the third determination sub-result, calculating the candidate probability of each of the at least two candidate entities using the relevant entity to be checked.

[0148] For example, when "Platelet Counter X" is selected as the entity to be verified, the first judgment result is the second judgment sub-result, and the second judgment result is the third judgment sub-result (that is, "Platelet Counter X" does not have a next-level entity to be verified, but corresponds to a related entity to be verified), and the candidate probability of each of at least two candidate entities can be calculated using the related entity to be verified.

[0149] For example, the candidate entities of the entity to be verified (i.e., "platelet counter X") include "platelet count", "platelet hematocrit", etc. It should be noted that, for the sake of clarity, it is assumed here that the candidate entities of the entity to be verified (i.e., "platelet counter X") only include "platelet count" and "platelet hematocrit".

[0150] For example, calculating the candidate probability of each of at least two candidate entities using the relevant entity to be checked includes the following steps S241 to S244.

[0151] Step S241: Obtain the related entities to be verified corresponding to the entity to be verified.

[0152] For example, the related entities to be verified corresponding to the entity to be verified (ie, "platelet count X") include "PLT", "100-300", etc.

[0153] Step S242: Execute the following steps S2421-S2422 for each of the entities included in the relevant entities to be verified.

[0154] Step S2421: Obtain relevant candidate entities. The relevant candidate entity is at the same level as each of the at least two candidate entities in the hierarchical structure of the knowledge graph, is associated with each of the at least two candidate entities, and has a relationship with the upper-level entity to which each of the at least two candidate entities belongs that is equal to a relationship between each of the entities included in the relevant entity to be verified and the entity to be verified.

[0155] For example, for the relevant entity to be verified "PLT", the relevant candidate entity corresponding to the candidate entity "platelet count" is "PLT", and the relevant candidate entity corresponding to the candidate entity "platelet hematocrit" is "PCT". For example, for the relevant entity to be verified "100-300", the relevant candidate entity corresponding to the candidate entity "platelet count" is "100-300", and the relevant candidate entity corresponding to the candidate entity "platelet hematocrit" is "0.17-0.35".

[0156] Step S2422: Calculate the similarity between each entity included in the relevant entity to be verified and the relevant candidate entity.

[0157] For example, the similarity between the relevant entity to be verified "PLT" and the relevant candidate entity "PLT" corresponding to the candidate entity "platelet count" is 1, while the similarity between the relevant entity to be verified "PLT" and the relevant candidate entity "PCT" corresponding to the candidate entity "platelet hematocrit" is 0.65.

[0158] For example, the similarity between the relevant entity to be verified "100-300" and the relevant candidate entity "100-300" corresponding to the candidate entity "platelet count" is 1; the similarity between the relevant entity to be verified "100-300" and the relevant candidate entity "0.17-0.35" corresponding to the candidate entity "platelet hematocrit" is 0.

[0159] Step S243: obtaining the relevant similarity of each of at least two candidate entities based on the similarity between each of the entities included in the relevant entity to be verified and the relevant candidate entity.

[0160] For example, in step S243, obtaining the relevant similarity of each of at least two candidate entities based on the similarity between each of the entities included in the relevant entity to be verified and the relevant candidate entity includes the following steps S2431 to S2433.

[0161] Step S2431: Determine the number of entities included in the relevant entities to be verified.

[0162] For example, when the related entities to be verified corresponding to the entity to be verified (i.e., "platelet counter X") only include "PLT", it can be determined that the number of entities included in the related entities to be verified is 1. For another example, when the related entities to be verified corresponding to the entity to be verified (i.e., "platelet counter X") include "PLT" and "100-300", it can be determined that the number of entities included in the related entities to be verified is 2.

[0163] Step S2432: when the number of entities included in the relevant entities to be verified is equal to 1, the similarity between the entities included in the relevant entities to be verified and the relevant candidate entities is used as the relevant similarity of each of the at least two candidate entities.

[0164] For example, when the related entities to be verified corresponding to the entity to be verified (i.e., "platelet count X") only include "PLT", the similarity (i.e., 1) between the related entity to be verified "PLT" and the related candidate entity "PLT" corresponding to the candidate entity "platelet count" can be used as the related similarity of the candidate entity "platelet count"; and the similarity (0.65) between the related entity to be verified "PLT" and the related candidate entity "PCT" corresponding to the candidate entity "platelet volume" can be used as the related similarity of the candidate entity "platelet volume".

[0165] Step S2433: when the number of entities included in the relevant entities to be verified is greater than 1, a weighted sum of multiple similarities between the entities included in the relevant entities to be verified and the relevant candidate entities is used as the relevant similarity of each of the at least two candidate entities.

[0166] For example, when the number of entities included in the relevant entities to be verified is greater than 1, a weight may be set for each entity included in the relevant entities to be verified. For example, when the relevant entities to be verified corresponding to the entity to be verified (i.e., "Platelet Counter X") include "PLT" and "100-300", the weight of the sub-item abbreviation (e.g., "PLT") may be set to 0.7, and the item reference value (e.g., "100-300") may be set to 0.3.

[0167] For example, the relevant similarity of the candidate entity “platelet count” = 0.7×1+0.3×1=1, that is, the relevant similarity of the candidate entity “platelet count” = 0.7(weight of sub-item abbreviation)×1(similarity between the relevant entity to be verified “PLT” and the relevant candidate entity “PLT” corresponding to the candidate entity “platelet count”)+0.3(weight of item reference value)×1(similarity between the relevant entity to be verified “100-300” and the relevant candidate entity “100-300” corresponding to the candidate entity “platelet count”).

[0168] For example, the relevant similarity of the candidate entity "platelet count" = 0.7×0.65+0.3×0=0.455, that is, the relevant similarity of the candidate entity "platelet count" = 0.7 (weight of the sub-item abbreviation) × 0.65 (similarity between the relevant entity to be verified "PLT" and the relevant candidate entity "PCT" corresponding to the candidate entity "platelet count") + 0.3 (weight of the item reference value) × 0 (similarity between the relevant entity to be verified "100-300" and the relevant candidate entity "0.17-0.35" corresponding to the candidate entity "platelet count").

[0169] Step S244: Obtain the candidate probability of each of the at least two candidate entities based on the relevant similarity of each of the at least two candidate entities and the entity similarity.

[0170] For example, in step S244, obtaining the candidate probability of each of at least two candidate entities based on the relevant similarity and entity similarity of each of at least two candidate entities includes: taking the weighted sum of the relevant similarity and entity similarity of each of at least two candidate entities as the candidate probability of each of the at least two candidate entities.

[0171] For example, the candidate probability of a candidate entity = the relevance similarity of the candidate entity × the weight of the relevance similarity of the candidate entity + the entity similarity of the candidate entity × the weight of the entity similarity of the candidate entity. For example, the weights of the relevance similarity and the entity similarity can be set according to actual application requirements. For example, the weights of the relevance similarity and the entity similarity can be set at 0.6 and 0.4, respectively.

[0172] For example, when the entity similarities of the candidate entity “platelet count” and the candidate entity “platelet hematocrit” are 0.8 and 0.6 respectively, and the relevant similarities of the candidate entity “platelet count” and the candidate entity “platelet hematocrit” are 1 and 0.455 respectively, the candidate probability of the candidate entity “platelet count” = 1×0.6+0.8×0.4=0.92; the candidate probability of the candidate entity “platelet hematocrit” = 0.455×0.6+0.6×0.4=0.513.

[0173] In other examples, the product of the correlation similarity and entity similarity of each of the at least two candidate entities can also be used as the candidate probability of each of the at least two candidate entities. In this case, the candidate probability of the candidate entity "platelet count" = 1 × 0.8 = 0.8; the candidate probability of the candidate entity "platelet hematocrit" = 0.455 × 0.6 = 0.273.

[0174] Step S233: when the first determination result is the first determination sub-result and the second determination result is the fourth determination sub-result, the candidate probability of each of the at least two candidate entities is calculated using the next level entity to be checked.

[0175] For example, when "Project Name X" is selected as the entity to be verified, the first determination result is the first determination sub-result; the second determination result is the fourth determination sub-result (that is, the entity to be verified "Project Name X" has a next-level entity to be verified, but does not correspond to the related entity to be verified), and the next-level entity to be verified can be used to calculate the candidate probability of each of at least two candidate entities. For example, the candidate entities of "Project Name X" include "Project Name" and "Project Reference Value".

[0176] For example, calculating the candidate probability of each of at least two candidate entities using the next level entity to be checked includes the following steps S251 to S255.

[0177] Step S251: Obtain the next level entity to be verified included in the entity to be verified.

[0178] For example, the next level of entities to be verified of the entity to be verified "Project Name X" includes "platelet count" and "platelet hematocrit". It should be noted that, for the sake of clarity, it is assumed here that the next level of entities to be verified of the entity to be verified "Project Name X" only includes "platelet count" and "platelet hematocrit".

[0179] Step S252: dividing the next-level entities to be verified into at least one next-level entity group based on the relationship between the entity to be verified and each entity in the next-level entities to be verified.

[0180] For example, since the next-level entities to be verified include "platelet count" and "platelet hematocrit" which have the same relationship with the entity to be verified "Project Name X", only one group of next-level entity groups to be verified is divided here, that is, the entities included in the next-level entity group to be verified are "platelet count" and "platelet hematocrit".

[0181] Step S253: Execute the following steps S2531 to S25312 for each of at least one next-level entity group to be verified.

[0182] Step S2531: Obtain a next-level candidate entity group, which is composed of all entities that are subordinate to each of at least two candidate entities and whose relationship with each of at least two candidate entities is equal to the relationship between each of the entities included in the next-level entity group to be verified and the entity to be verified.

[0183] For example, the next level candidate entity group included in the candidate entity "project name" includes entities such as "platelet count", "platelet hematocrit", and "platelet distribution width"; the next level candidate entity group included in the candidate entity "project reference value" includes "100-300", "0.17-0.35", "9-17", etc.

[0184] Step S2532: Calculate the similarity between each of at least one next-level entity group to be verified and the next-level candidate entity group to obtain at least one group similarity corresponding to the at least one next-level entity group to be verified.

[0185] For example, in step S2532, obtaining at least one group similarity corresponding to at least one group of next-level entity groups to be verified includes:

[0186] Step 310: Obtain the maximum similarity of each of the multiple entities included in each of at least one next-level entity group to be verified, so as to obtain multiple maximum similarities of the multiple entities included in each of at least one next-level entity group to be verified. Here, the maximum similarity is the maximum similarity among the similarities between each of the multiple entities included in each of the at least one next-level entity group to be verified and all the entities included in the next-level candidate entity group.

[0187] For example, the similarities between the entity “platelet count” included in the next-level entity group to be verified and all entities of the next-level candidate entity group of the candidate entity “project name” (for example, “platelet count”, “platelet hematocrit”, “platelet distribution width”) are 1, 0.6, and 0.5, respectively. Therefore, the largest similarity among the similarities between the entity “platelet count” included in the next-level entity group to be verified and all entities of the next-level candidate entity group of the candidate entity “project name” (that is, the maximum similarity of the entity “platelet count” included in the next-level entity group to be verified for the candidate entity “project name”) is 1.

[0188] For example, the similarities between the entity "platelet count" included in the next-level entity group to be verified and all entities of the next-level candidate entity group of the candidate entity "project name" (for example, "platelet count", "platelet count", and "platelet distribution width") are 0.6, 1, and 0.5, respectively. Therefore, the largest similarity among the similarities between the entity "platelet count" included in the next-level entity group to be verified and all entities of the next-level candidate entity group of the candidate entity "project name" (that is, the maximum similarity of the entity "platelet count" included in the next-level entity group to be verified for the candidate entity "project name") is 1.

[0189] Therefore, for the candidate entity “project name”, the multiple maximum similarities of the multiple entities included in the next-level entity group to be verified (ie, “platelet count” and “platelet hematocrit”) are 1 and 1, respectively.

[0190] For example, the maximum similarity among the similarities between the entity “platelet count” included in the next-level entity group to be verified and all entities of the next-level candidate entity group of the candidate entity “project reference value” (that is, the maximum similarity of the entity “platelet count” included in the next-level entity group to be verified for the candidate entity “project reference value”) is 0; the maximum similarity among the similarities between the entity “hematocrit” included in the next-level entity group to be verified and all entities of the next-level candidate entity group of the candidate entity “project reference value” (that is, the maximum similarity of the entity “hematocrit” included in the next-level entity group to be verified for the candidate entity “project reference value”) is 0; in this case, for the candidate entity “project reference value”, the multiple maximum similarities of the multiple entities included in the next-level entity group to be verified (that is, “platelet count” and “hematocrit”) are 0 and 0, respectively.

[0191] Step 320: Obtain at least one group similarity corresponding to at least one group of entities to be verified at the next level based on the multiple maximum similarities.

[0192] For example, obtaining at least one group similarity corresponding to at least one next-level entity group to be verified based on multiple maximum similarities includes: taking a weighted sum of the multiple maximum similarities as at least one group similarity corresponding to at least one next-level entity group to be verified.

[0193] For example, the weights of the multiple entities included in the next level entity group to be verified can be set according to actual application requirements. For example, the weights of the multiple entities included in the next level entity group to be verified are all equal to 1 / the number of multiple entities. For example, the weights of the entities "platelet count" and "platelet hematocrit" in the next level entity group to be verified are both equal to 1 / 2.

[0194] For example, for the candidate entity "project name", the group similarity of the next level entity group to be verified (including entities "platelet count" and "platelet hematocrit") = 0.5×1+0.5×1=1. For example, for the candidate entity "project reference value", the group similarity of the next level entity group to be verified (including entities "platelet count" and "platelet hematocrit") = 0.5×0+0.5×0=0.

[0195] Step S254: obtaining a next level similarity of each of at least two candidate entities based on at least one group similarity.

[0196] For example, in step S254, obtaining the next-level similarity of each of at least two candidate entities based on at least one group similarity includes: determining the number of next-level entity groups included in the next-level entity to be verified; when the number of next-level entity groups is equal to 1, using the group similarity of the next-level entity groups included in the next-level entity to be verified as the next-level similarity of each of the at least two candidate entities; when the number of next-level entity groups is greater than 1, using the weighted sum of multiple group similarities of the next-level entity groups included in the next-level entity to be verified as the next-level similarity of each of the at least two candidate entities.

[0197] For example, since the next-level entities to be verified of "Project Name X" include "platelet count" and "platelet hematocrit", which have the same relationship with the entity to be verified "Project Name X", that is, the number of next-level entity groups to be verified of "Project Name X" is equal to 1, the next-level similarity of the candidate entity "Project Name" and the next-level similarity of the candidate entity "Project Reference Value" are 1 and 0, respectively.

[0198] Step S255: Calculate the candidate probability of each of the at least two candidate entities based on the next level similarity of each of the at least two candidate entities and the entity similarity.

[0199] For example, in step S255, calculating the candidate probability of each of at least two candidate entities based on the next-level similarity of each of the at least two candidate entities and the entity similarity includes: taking the product of the next-level similarity of each of the at least two candidate entities and the entity similarity as the candidate probability of each of the at least two candidate entities.

[0200] For example, when the entity similarities of "project name" and "project reference value" are 0.75 and 0.3 respectively, and the next level similarities of "project name" and "project reference value" are 1 and 0 respectively, the candidate probability of the candidate entity "project name" = 0.75×1=0.75, and the candidate probability of "project reference value" = 0.3×0=0.

[0201] Step S231: when the first determination result is the first determination sub-result and the second determination result is the third determination sub-result, the candidate probability of each of at least two candidate entities is calculated using the next level entity to be verified and / or the related entity to be verified.

[0202] For example, when the first determination result is the first determination sub-result and the second determination result is the third determination sub-result, the candidate probability of each of the at least two candidate entities can be calculated using the next-level entity to be verified or the candidate probability of each of the at least two candidate entities can be calculated using the related entity to be verified. For another example, when the first determination result is the first determination sub-result and the second determination result is the third determination sub-result, the candidate probability of each of the at least two candidate entities can be calculated using the next-level entity to be verified and the related entity to be verified.

[0203] For example, calculating the candidate probability of each of at least two candidate entities using the next-level entity to be verified and the related entity to be verified includes the above steps S241 to S244 and steps S251 to S255.

[0204] For example, calculating the candidate probability of each of at least two candidate entities using the next level entity to be verified and the related entity to be verified also includes: obtaining the candidate probability of each of the at least two candidate entities based on the related similarity, next level similarity and entity similarity of each of the at least two candidate entities.

[0205] For example, obtaining the candidate probability of each of at least two candidate entities based on the relevant similarity, next-level similarity, and entity similarity of each of at least two candidate entities includes: taking the product of the weighted sum of the relevant similarity and entity similarity of each of at least two candidate entities and the next-level similarity as the candidate probability of each of the at least two candidate entities. For example, when the relevant similarity, next-level similarity, and entity similarity of the candidate entities are 0.9, 1, and 0.8, respectively, and the weights of the relevant similarity and entity similarity are 0.6 and 0.4, respectively, the relevant similarity of the candidate entity = (0.6×0.9+0.4×0.8)×1 = 0.86.

[0206] For another example, obtaining the candidate probability of each of at least two candidate entities based on the relevant similarity, the next level similarity and the entity similarity of each of at least two candidate entities includes: taking the product of the relevant similarity, the next level similarity and the entity similarity of each of at least two candidate entities as the candidate probability of each of the at least two candidate entities. For example, when the relevant similarity, the next level similarity and the entity similarity of the candidate entities are 0.9, 1 and 0.8 respectively, the relevant similarity of the candidate entities = 0.9 × 1 × 0.8 = 0.72.

[0207] Step S234: when the first determination result is the second determination sub-result and the second determination result is the fourth determination sub-result, the entity similarity of each of the at least two candidate entities is used as the probability that each of the at least two candidate entities is an erroneous entity to be verified.

[0208] For example, in step S234, if the entity to be verified has neither a next-level entity to be verified nor a related entity to be verified, the entity similarity of each of at least two candidate entities of the entity to be verified can be used as the probability that each of the at least two candidate entities is an erroneous entity to be verified.

[0209] For example, in step S220, correcting erroneous character information based on the knowledge graph also includes: when the number of entities is greater than 1, executing the following step S223.

[0210] Step S223: Replace the character information corresponding to the entity to be verified with errors with the character information corresponding to the candidate entity with the maximum candidate probability.

[0211] For example, when it is confirmed that the candidate probability of the candidate entity "platelet count" of the entity to be verified "platelet count X" is 0.8, and the candidate probability of the candidate entity "platelet volume" of the entity to be verified "platelet count X" is 0.273, the character information corresponding to the entity to be verified "platelet count X" can be replaced with the character information corresponding to the candidate entity (that is, "platelet count") corresponding to the maximum candidate probability (that is, 0.8).

[0212] For example, when it is confirmed that the candidate probability of the candidate entity "Project Name" of the entity to be verified "Project Name X" is 0.75, and the candidate probability of the candidate entity "Project Reference Value" of the entity to be verified "Project Name X" is 0, the character information corresponding to the entity to be verified "Project Name X" can be replaced with the character information corresponding to the candidate entity (that is, "Project Name") corresponding to the maximum candidate probability (that is, 0.75).

[0213] Figure 5A It is a schematic flowchart of verifying at least one item of character information based on a knowledge graph provided by at least one embodiment of the present disclosure. Figure 5B It is a schematic flowchart of knowledge graph-based parsing provided by at least one embodiment of the present disclosure.

[0214] like Figure 5A and Figure 5B As shown, verifying at least one item of character information based on the knowledge graph includes table analysis (i.e., table structure analysis) and entity linking (Entity Linking, for example, entity linking can associate corresponding entities or / and map an entity to an associated entity). For example, the result of the table analysis can be used to grade multiple entities included in at least one item of character information extracted from an image. For example, through entity linking, multiple entities included in at least one item of character information extracted from an image can be associated with entities in the knowledge graph, thereby determining the levels of multiple entities in the knowledge graph.

[0215] For example, Figure 5A and Figure 5B As shown, after completing the entity linking, at least one character information extracted from the image can be verified based on the knowledge graph. For example, the specific method of verifying at least one character information extracted from the image based on the knowledge graph can refer to the method for obtaining character information in the image provided by at least one embodiment of the present disclosure, which will not be repeated here. For example, after completing the verification, the result can be output (that is, the verified characters are output).

[0216] For example, Figure 5B As shown, table analysis may include table recognition and segmentation. For example, a program provided by OpenCV may be used to recognize a table, so that the table can be recognized and segmented. For example, the page type is confirmed as a table through a knowledge graph.

[0217] For example, table recognition and segmentation include table horizontal line detection, table vertical line detection and table vertex detection. For example, table horizontal line detection includes horizontal straight line detection, image binarization and secondary median filtering. For example, the pixels of each row in the image can be detected, and the horizontal line that divides the two records can be identified based on the standard that the average pixel difference between adjacent pixels is greater than 120, and a list of horizontal straight lines can be obtained. For example, table vertical line detection includes detecting the grayscale image and then performing straight line detection, with a maximum spacing of 30 pixels, a minimum line segment length of 500 pixels, and a vertical straight line. For example, table vertex detection is used to identify the vertex where each cell is located.

[0218] For example, Figure 5B As shown, table analysis can also include fine reconstruction of the table. For example, fine reconstruction of the table includes at least one of row identification (no horizontal lines), column identification (no vertical lines) and cross-row identification. For example, row identification can be completed after statistics and analysis (row coordinates) through the position information returned after text recognition. For example, column identification can be completed after comprehensive statistical analysis (column coordinates) through the position information returned after text recognition. For example, the position information after row identification (upward alignment) and column identification (left alignment) can be used, and the number of rows in the column with the largest number of rows can be used as the maximum number of rows of the table. Through the text information in the same column position of adjacent rows, if there is no text, it is defined as text crossing lines, and the content is integrated into one line (when there is no information above or below, it is adjusted based on the middle).

[0219] For example, Figure 5BAs shown, the parsing based on the knowledge graph may also include at least one of page number recognition, page type recognition, line reconstruction and text block reconstruction, header and footer recognition, and image / text recognition classification. For example, page number recognition is used to use OCR to identify the page number of each report, so as to confirm and adjust the order of the report to be in positive order and continuous from front to back. For example, an error may be reported if the page is discontinuous. For example, page type recognition is used to identify whether the page contains medical images. For example, based on the results of layout analysis and OCR, an electronic document containing text information and layout information can be reconstructed, line reconstruction and text block reconstruction can be performed, and it can be standardized and output as a document in HOCR (a document format used by OCR) format. For example, header and footer recognition is used to realize the header and footer recognition of the report through a similarity recognition algorithm; for example, similarity recognition means that in a plurality of examination reports, if similar text appears multiple times, it is classified as a header and footer. For example, image / text recognition classification includes using CTPN (Detecting Text in Natural Image with Connectionist Text Proposal Network) and / or CRNN (Convolutional Recurrent Neural Network) based on a general deep learning framework, and using an OCR recognition open source library (e.g., Tesseract digital recognition library) for Chinese character recognition training.

[0220] Fig. 6A It is a schematic flowchart of verifying character information in an image and adding explanations based on a knowledge graph provided by at least one embodiment of the present disclosure.

[0221] For example, Fig. 6A As shown, at least one embodiment of the present disclosure provides a method for verifying character information in an image and adding explanations based on a knowledge graph, including: picture input; recognition result output; parsing result output; and graph verification.

[0222] For example, OCR can be used to extract character information from a picture, and the character information can be output as a recognition result; for example, the character information extracted from the picture by OCR can be parsed to convert the character information in the picture into character information in text format, and the character information in text format can be output as a parsing result. For example, the table in the picture can also be analyzed, and the table analysis result can be used to correspond the entity corresponding to the parsed character information to the entity in the knowledge graph. For example, the graph verification includes using the knowledge graph to verify the parsing result to confirm whether the character information extracted from the picture includes erroneous character information. For example, in the case where the character information extracted from the picture does not include erroneous character information, the knowledge graph can be used in the graph interpretation step to add a description (for example, a sub-item description) to the entity corresponding to the character information extracted from the picture. For example, in the case where the character information extracted from the picture includes erroneous character information, the knowledge graph can be used in the graph correction step to correct the erroneous character information, and then the knowledge graph can be used in the graph interpretation step to add a description to the entity corresponding to the corrected character information. For example, after completing the graph interpretation step, the character information with the added description can be output.

[0223] For example, the specific method of using the knowledge graph to verify the parsing results to confirm whether the character information extracted from the image includes erroneous character information, using the knowledge graph to correct the erroneous character information, and using the knowledge graph to add descriptions to the entities corresponding to the character information, for example, can refer to the relevant content of the method for obtaining character information in the image and the page processing method provided in at least one embodiment of the present disclosure, which will not be repeated here.

[0224] Figure 6B It is a schematic flowchart of medical record analysis based on knowledge graph provided by at least one embodiment of the present disclosure.

[0225] For example, Figure 6BAs shown, the medical record parsing based on the knowledge graph includes the following steps. First, obtain the document (i.e., medical record) of the relevant user, which can be a paper or electronic file (i.e., picture). If it is a paper, it is scanned to obtain a picture; for example, the medical record includes the patient's medical history, medical report, etc. Secondly, the subject (subject area) of the picture is detected, and the boundary of the subject (subject area) of the picture can be determined, and then the subject (subject area) can be extracted along the boundary of the subject (subject area). Third, perform image enhancement (e.g., denoising, improving contrast, etc.) on the subject (subject area). Fourth, perform image recognition on the enhanced subject area (i.e., extract character information in the picture) to obtain a recognition result (i.e., character information extracted from the picture). Fifth, parse the recognition result (i.e., character information extracted from the picture) to convert the extracted character information into text format. Sixth, use a knowledge graph (for example, a general knowledge graph or a medical examination and testing knowledge graph) to check the parsed character information; if it is confirmed that the parsed character information does not contain erroneous character information, output the parsed character information; if it is confirmed that the parsed character information contains erroneous character information, use the medical examination and testing knowledge graph to correct it, and then output the corrected character information.

[0226] like Figure 6B As shown, correction using the medical examination and verification knowledge graph includes: confirming the combination with the highest probability based on the relationship between the item to be checked (for example, the character information to be corrected) and the checklist, the name of the item to be checked, the abbreviation of the item to be checked, etc., and using the character information corresponding to the combination with the highest probability to cover (replace) the erroneous character information.

[0227] For example, based on Figure 3 The information extracted from the image is verified by using multiple entities in the knowledge graph shown (i.e., checklist, check item, check sub-item, English abbreviation, medical order code, medical order name, content, diagnosis, alias, etc.) and the relationship between the above multiple entities. For example, the minimum compilation distance algorithm can be used to obtain the compilation distance (Levenshtein distance) between at least one of the above multiple entities and the content to be verified (entity to be verified) to verify (check and correct) the information extracted from the image (e.g., character information).

[0228] For example, the probability that the candidate correct content (e.g., candidate character information; for example, "platelet count X") corresponding to the erroneous content (e.g., erroneous character information; for example, "platelet count") is the content corresponding to the erroneous content when it is not misidentified (e.g., "platelet count") can be determined by calculating the probability of the minimum block compilation distance, and the candidate correct content corresponding to the maximum probability will cover the erroneous content. For example, calculating the probability of the minimum block compilation distance includes: using a block calculation method to calculate the minimum compilation distance between the content to be verified (including names, abbreviations, ranges, etc.) and the entities (or entities and relationships) in the medical examination and inspection knowledge graph, and calculating the probability based on the minimum compilation distance calculated above.

[0229] For example, the block minimum edit distance probability calculation involves the following steps. First, information (e.g., inspection agency, clinical examination items, clinical examination sub-items, inspection list, inspection type, etc.) is obtained through layout analysis and table semantic recognition and division; the entities corresponding to the above information are classified according to the entity name and type in the knowledge graph, and each entity module (e.g., at least one of the entities corresponding to the above information corresponds to the entity in the knowledge graph) is calculated for the compilation distance, and the content corresponding to the highest probability combination (e.g., character information) is selected to cover the content with errors. For example, block calculation means: the entity corresponding to the inspection agency is only calculated for the compilation distance with the entity corresponding to the inspection agency in the knowledge graph; the entity corresponding to the clinical examination item is only calculated for the compilation distance with the entity corresponding to the clinical examination item in the knowledge graph; the entity corresponding to the inspection list name is only calculated for the compilation distance with the entity corresponding to the inspection list name in the knowledge graph. For example, the use of the block minimum edit distance probability algorithm can make full use of the relationship between entities in the knowledge graph (including information and included information, etc.), which can effectively reduce the scope of the minimum edit distance calculation of each entity (that is, reduce the amount of calculation), improve the efficiency of error correction and the accuracy of the content after error correction.

[0230] For example, the probability can be calculated using the following formula.

[0231]

[0232] Here, P(B) is the probability of correct content (e.g., correct character information), A i At least one of the entities corresponding to the checklist, inspection item, inspection details, word count, English abbreviation, etc. in the inspection knowledge graph.

[0233] For example, in the block minimum edit distance probability calculation, the entity corresponding to surgical examination (II) only needs to perform edit distance probability calculation with the entity of the examination item corresponding to surgical examination (II) included in the physical examination item entity in the knowledge graph, and does not need to perform compilation distance probability calculation with the relevant entities included in the physical examination institution in the knowledge graph. For example, when performing edit distance probability calculation on the entity corresponding to surgical examination (II) and the entity of the examination item corresponding to surgical examination (II) included in the physical examination item entity in the knowledge graph, the surgical examination sub-item "skin color examination" included in surgical examination (II) only needs to perform compilation distance calculation with the entity corresponding to at least one surgical examination sub-item included in surgical examination (II) in the knowledge graph (for example, full name, English abbreviation, number of words, etc.), thereby determining the correct content.

[0234] Fig. 7A This is a picture corresponding to a part of the routine blood test form. Figure 7B yes Fig. 7A Shown is a picture of part of the blood test sheet after it has become contaminated.

[0235] like Figure 7B As shown in the figure, the following character information extracted from the image corresponding to the contaminated inspection form all have errors: "average blood volume XX", "large platelets X cases" and "lymphocyte XX ratio". It should be noted that Fig. 7A and Figure 7B The character information in the first to seventh columns respectively corresponds to "sub-project serial number", "sub-project name", "sub-project abbreviation", "sub-project result", "sub-project result abnormal prompt", "sub-project result unit" and "sub-project reference value".

[0236] Figure 7C A software program that uses a method for obtaining character information in a picture provided by at least one embodiment of the present disclosure to obtain character information in a picture Figure 7B The result obtained by combining the character information in the picture. Figure 7C The characters in the solid line box are from Figure 7B The character information obtained from the picture shown in the figure, where the "name" in the solid line box represents the name of the inspection sub-item. Figure 7C As shown, a software program using a method for acquiring character information in a picture provided based on at least one embodiment of the present disclosure corrects the erroneous character information "mean blood volume XX", "X number of large platelets" and "XX ratio of lymphocytes" into "mean platelet volume", "ratio of large platelets" and "percentage of lymphocytes" respectively and then outputs them.

[0237] It should be noted that due to Figure 7C The characters outside the solid line frame are irrelevant to the technical solution of the embodiment of the present disclosure, so the meaning of the characters outside the solid line frame will not be repeated here.

[0238] At least one embodiment of the present disclosure further provides a device for acquiring character information in a picture. Figure 8 A device for acquiring character information in a picture provided by at least one embodiment of the present disclosure.

[0239] For example, Figure 8 As shown, the device for acquiring character information in the picture includes a character information extraction device and a checking and correction device.

[0240] For example, the character information extraction device is configured to obtain a picture and extract at least one character information included in the picture. For example, the verification and correction device is configured to verify at least one character information based on the knowledge graph.

[0241] For example, the specific method of obtaining a picture and extracting at least one item of character information included in the picture and the specific method of verifying at least one item of character information based on a knowledge graph can be found in at least one embodiment of the present disclosure. A method for obtaining character information in a picture is also provided, which will not be described in detail here.

[0242] For example, the character information extraction device and the inspection and correction device can be implemented by software, firmware, hardware and any combination thereof, respectively. For example, the hardware includes a Field Programmable Gate Array (FPGA) and the like.

[0243] For example, by setting a verification and correction device in the device for acquiring character information in the picture, at least one item of character information can be verified (for example, verification and correction) based on the knowledge graph, thereby confirming whether there are errors in the character information extracted from the picture and correcting the erroneous character information, thereby improving the accuracy of the character information acquired by the device for acquiring character information in the picture provided by at least one embodiment of the present disclosure.

[0244] At least one embodiment of the present disclosure further provides another device for acquiring character information in a picture. Fig. 9 At least one embodiment of the present disclosure provides another device for acquiring character information in a picture. Fig. 9 As shown, the other device for acquiring character information in a picture includes: a processor and a memory. The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes any method for acquiring character information in a picture provided by at least one embodiment of the present disclosure.

[0245] For example, the processor is, for example, a central processing unit (CPU), a graphics processor GPU, a tensor processing unit (TPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. For example, the processor can be implemented as a general-purpose processor, and can also be a single-chip microcomputer, a microprocessor, a digital signal processor, a dedicated image processing chip, or a field programmable logic array, etc. For example, the memory may include at least one of a volatile memory and a non-volatile memory, for example, the memory may include a read-only memory (ROM), a hard disk, a flash memory, etc. Accordingly, the memory may be implemented as one or more computer program products, and the computer program product may include various forms of computer-readable storage media, and one or more computer program instructions may be stored on the computer-readable storage medium. The processor may run the program instructions to execute the method for obtaining character information in any picture provided by at least one embodiment of the present disclosure. The memory may also store various other applications and various data, for example, various data used and / or generated by the application, etc.

[0246] For example, at least one embodiment of the present disclosure also provides another device for acquiring character information in a picture, which can verify (for example, check and correct) at least one item of character information based on a knowledge graph, thereby confirming whether there are errors in the character information extracted from the picture and correcting the erroneous character information, thereby improving the accuracy of character information acquired by another device for acquiring character information in a picture provided by at least one embodiment of the present disclosure.

[0247] At least one embodiment of the present disclosure also provides a non-transitory storage medium. Fig.10 is a non-transitory storage medium provided by at least one embodiment of the present disclosure. Fig.10 As shown, the non-transitory storage medium includes computer program instructions stored thereon. When the computer program instructions are executed by a processor, the computer executes any method for acquiring character information in a picture provided by at least one embodiment of the present disclosure.

[0248] For example, the non-transitory storage medium may include a magnetic storage medium, an optical storage medium, a semiconductor storage medium, etc.; for example, the non-transitory storage medium may include a read-only memory (ROM), a hard disk, a flash memory, etc. For example, Fig.10 The non-transitory storage medium shown may be used to improve the accuracy of character information obtained from an image.

[0249] At least one embodiment of the present disclosure also provides a page processing method. Fig.11 is an exemplary flow chart of a page processing method provided by at least one embodiment of the present disclosure. Fig.11As shown, the page processing method includes the following steps S510 to S530.

[0250] Step S510: Get the image of the page.

[0251] Step S520: Acquire the verified character information of the image of the page by executing any acquisition method provided by at least one embodiment of the present disclosure.

[0252] Step S530: Structuring the verified character information of the image on the page to obtain a structured data combination.

[0253] For example, in step S510, the page may be a medical report, a medical record, a prescription, etc. For example, the medical report may include at least one of the following reports: an examination report, a physical examination report, a hospitalization report, a discharge report, and the like.

[0254] For example, structured data is relational model data and can be stored in a relational database. For example, unstructured data is data without a fixed pattern and can be stored in a non-relational database. For example, structured data can be understood by a computer, so it is conducive to retrieval and display in a user graphical interface. Therefore, by adopting the page processing method provided by at least one embodiment of the present disclosure to obtain a structured data combination from the picture of the page, not only can the accuracy of the character information obtained from the picture of the page be improved, but also the medical examination and inspection data of the patient can be more effectively utilized. For example, "platelet count"-"value"-"215" is a structured data; for example, "platelet hematocrit"-"value"-"0.2" is also a structured data. For example, multiple structured data can be obtained based on the character information after verification of the picture of the page, and the above multiple structured data are associated with each other to obtain a structured data combination.

[0255] For example, the page processing method may further include the following steps S540 and S550.

[0256] Step S540: Confirm that the image on the page includes medical images based on the knowledge graph.

[0257] Step S550: adding medical images to the structured data combination.

[0258] For example, at least one embodiment of the present disclosure further provides a page processing method that can be executed in sequence in the order of step S510, step S520, step S530, step S540, and step S550.

[0259] For another example, at least one embodiment of the present disclosure further provides a page processing method that can be executed in the order of step S510, step S520, step S540, and step S530+step S550 (ie, step S530 and step S550 are executed simultaneously).

[0260] For example, in step S540, confirming that the picture of the page includes a medical image based on the knowledge graph includes: determining that the picture of the page includes a medical image based on the character information extracted from the picture of the page and verified based on the knowledge graph (for example, the verified character information obtained by step S520) including text corresponding to the medical image. For example, if the verified character information obtained by step S520 includes text such as nuclear magnetic resonance image (NMRI) and does not include text corresponding to an examination sub-item (for example, platelet count), it can be determined that the picture of the page includes a medical image, and the medical image can be extracted from the picture of the page, so that the page processing method provided by at least one embodiment of the present disclosure can effectively save more medical examination results. For example, by making the page processing method also include confirming that the picture of the page includes a medical image based on the knowledge graph, the page processing method provided by at least one embodiment of the present disclosure can also have a page type recognition function.

[0261] For example, in step S550, the medical image detected and extracted from the picture of the page may be associated with the structured data in step S530, thereby adding the medical image to the updated structured data combination.

[0262] For example, the page processing method may further include the following step S560.

[0263] Step S560: Based on the knowledge graph and the multiple entities corresponding to the verified character information, add descriptions corresponding to the multiple entities in the structured data combination.

[0264] For example, for each of the above-mentioned multiple entities, based on the level of each of the multiple entities in the knowledge graph, the examination item explanation and abnormality explanation associated with each entity (e.g., examination sub-item) corresponding to the above-mentioned multiple entities (medical significance of the examination sub-item) can be found in the knowledge graph, and the description corresponding to the multiple entities can be added to the structured data combination based on the examination item explanation and abnormality explanation. For example, by adding descriptions corresponding to the multiple entities in the structured data combination, the patient can better understand the meaning of the examination results, thereby allowing the patient to use the examination results more effectively.

[0265] For example, the following description can be added to the entity "platelet count": "Platelets are a type of cell in human blood, and platelet count refers to the number of platelets contained in a unit volume of blood. Decreased platelet count is common in aplastic anemia, acute leukemia, radiotherapy and chemotherapy, thrombocytopenic purpura, hypersplenism, etc.; increased platelet count is common in pathological myeloproliferative diseases, essential thrombocythemia, acute hemolysis, acute suppurative infection, etc."

[0266] For example, the page processing method may also include page number recognition to combine and merge structured data located on different pages.

[0267] At least one embodiment of the present disclosure also provides a method for constructing a knowledge graph (for example, a method for constructing a medical knowledge graph). Fig.12 A method for constructing a medical knowledge graph is provided by at least one embodiment of the present disclosure.

[0268] For example, Fig.12 As shown, the method for constructing the medical knowledge graph includes the following steps S610 to S640.

[0269] Step S610: Obtain the data structure of the medical knowledge graph.

[0270] For example, in step S610, the data structure of the medical knowledge graph can be Figure 3 The data structure of the medical knowledge graph shown in FIG. 1 is not limited to this. For example, the data structure of the medical knowledge graph may not include Figure 3 The data structure of the medical knowledge graph shown in the figure corresponds to the part of the hospital information system and / or the part corresponding to the medical significance of the clinical examination sub-item. For example, the data structure (medical knowledge graph framework) for constructing the medical knowledge graph includes defining the types of entities and the types of relationships between entities.

[0271] Step S620: Export corresponding entities from at least the medical image archiving and communication system and the laboratory information management system.

[0272] For example, the corresponding entity can be manually exported from the medical image archiving and communication system and the laboratory information management system. For another example, the corresponding entity can be automatically exported from the medical image archiving and communication system and the laboratory information management system using the export function of the medical image archiving and communication system and the laboratory information management system.

[0273] For example, by making the medical knowledge graph include a picture archiving and communication system (PACS), it is possible to confirm whether a picture contains a medical image, and if the picture contains a medical image, extract it from the picture on the page. The knowledge graph obtained based on the medical knowledge graph construction method provided by at least one embodiment of the present disclosure can be used to effectively preserve more medical examination results.

[0274] Step S630: aligning the entities exported from the medical image archiving and communication system with the entities exported from the laboratory information management system to obtain a plurality of entities after entity alignment.

[0275] For example, entity alignment of entities derived from the medical image archiving and communication system and entities derived from the laboratory information management system is used to associate entities derived from the medical image archiving and communication system and entities derived from the laboratory information management system that are substantially the same (have the same meaning). For example, the medical image archiving and communication system may use "abbreviation" to represent "abbreviation of project name", while the laboratory information management system may use "code" to represent "abbreviation of project name". In this case, both "abbreviation" and "code" may be associated with "abbreviation of project name" (or "abbreviation" and "code" may be directly associated) in step S630, so that the entities derived from the medical image archiving and communication system and the entities derived from the laboratory information management system are entity aligned.

[0276] For example, by aligning entities exported from a medical image archiving and communication system with entities exported from a laboratory information management system, the knowledge graph obtained by the method for constructing a medical knowledge graph provided based on at least one embodiment of the present disclosure can have wider applicability.

[0277] For example, in addition to aligning entities derived from the medical image archiving and communication system and entities derived from the laboratory information management system, other processing can be performed on the entities derived from the medical image archiving and communication system and entities derived from the laboratory information management system in combination with medical literature, guidelines, papers, etc. For example, other processing includes data cleaning (e.g., removing irrelevant entities) and data supplementation (supplementing important entities that are not included in the two systems).

[0278] Step S640: Fill the aligned multiple entities into the medical knowledge graph based on the data structure of the medical knowledge graph.

[0279] For example, when entity alignment and other processing are performed on entities exported from a medical image archiving and communication system and entities exported from a laboratory information management system, the aligned and other processed entities can be filled into the medical knowledge graph based on the data structure of the medical knowledge graph.

[0280] For example, the method for constructing the medical knowledge graph includes: exporting corresponding entities from a hospital information system; in this case, the entities exported from the hospital information system can be aligned with multiple entities after entity alignment; and the entities exported from the hospital information system and aligned with multiple entities after entity alignment are filled into the medical knowledge graph based on the data structure of the medical knowledge graph.

[0281] For example, in the case where it is necessary to export the corresponding entity from the hospital information system, the export of the corresponding entity from the hospital information system can be performed in step S620, and in step S630, the entity exported from the medical image archiving and communication system, the entity exported from the laboratory information management system, and the corresponding entity exported from the hospital information system can be aligned at the same time.

[0282] For example, the method for constructing the medical knowledge graph also includes: obtaining sub-item descriptions of clinical examination sub-items involved in the laboratory information management system; and processing the sub-item descriptions and filling them into the medical knowledge graph.

[0283] For example, the sub-item description of the clinical examination sub-item involved in the laboratory information management system recorded on the Internet obtained by crawlers (i.e., network robots) may include "low hematocrit, which is common in patients with anemia and when the patient's blood is diluted".

[0284] For example, the processing of the obtained sub-item descriptions of the clinical examination sub-items includes cleaning (i.e., removing irrelevant information). For another example, the processing of the obtained sub-item descriptions of the clinical examination sub-items may also include merging (i.e., merging descriptions obtained from multiple information sources) and cleaning (i.e., removing irrelevant or redundant information).

[0285] For example, by processing the sub-item descriptions and filling them into the medical knowledge graph, the method and apparatus for obtaining the knowledge graph using the medical knowledge graph construction method provided based on at least one embodiment of the present disclosure can have the potential to present to patients the descriptions of the clinical examination sub-items and the meaning of the abnormalities when the clinical examination sub-item results are abnormal, thereby allowing patients to understand the meaning of the examination results more clearly and to use the examination results more effectively.

[0286] Fig.13 An exemplary scene diagram of a system for acquiring character information in a picture provided by at least one embodiment of the present disclosure is shown. Fig.13 As shown, the system 300 for acquiring character information in a picture may include a user terminal 310 , a network 320 , a server 330 , and a database 340 .

[0287] For example, the user terminal 310 may be Fig.13 The computer 310-1 and the portable terminal 310-2 are shown in FIG. It is understandable that the user terminal may also be any other type of electronic device capable of receiving, processing and displaying data, which may include but is not limited to desktop computers, laptop computers, tablet computers, smart home devices, wearable devices, vehicle-mounted electronic devices, medical electronic devices, etc.

[0288] For example, the network 320 may be a single network or a combination of at least two different networks. For example, the network 320 may include but is not limited to a local area network, a wide area network, a public network, a private network, the Internet, a mobile communication network, etc., or a combination of several of them.

[0289] For example, the server 330 can be a single server or a server group, and each server in the server group is connected via a wired network or a wireless network. The wired network can communicate by means of twisted pair, coaxial cable or optical fiber transmission, for example, and the wireless network can communicate by means of 3G / 4G / 5G mobile communication network, Bluetooth, Zigbee or WiFi, for example. The present disclosure does not limit the type and function of the network. The server group can be centralized, such as a data center, or distributed. The server can be local or remote. For example, the server 330 can be a general-purpose server or a dedicated server, and can be a virtual server or a cloud server, etc.

[0290] For example, the database 340 can be used to store various data used, generated and output from the user terminal 310 and the server 330 in operation. The database 340 can be interconnected or communicated with the server 330 or a part of the server 330 via the network 320, or directly interconnected or communicated with the server 330, or interconnected or communicated with the server 330 via a combination of the above two methods. In some embodiments, the database 340 can be an independent device. In other embodiments, the database 340 can also be integrated in at least one of the user terminal 310 and the server 340. For example, the database 340 can be set on the user terminal 310, or can be set on the server 340. For another example, the database 340 can also be distributed, with a part of it being set on the user terminal 310 and another part being set on the server 340.

[0291] In one example, the user terminal 310 can obtain a picture (for example, using the camera of the user terminal to take a picture of a paper medical examination report), and send the obtained picture to the server 330 via the network 320 or other technologies (for example, Bluetooth communication, infrared communication, etc.); the server 330 can obtain the information of the knowledge graph from the database, and execute the method for obtaining character information in the picture and the page processing method provided by at least one embodiment of the present disclosure to obtain the verified character information extracted from the above picture and the structured data combination obtained by structuring the verified character information; then, the server 330 can store the above structured data combination in the database 340, and send the above structured data combination to the user terminal 310; finally, the user terminal displays (for example, through a pre-designed graphical user interface) the above structured data combination.

[0292] In some implementations, the server 330 may use an application built into the server to execute the method for acquiring character information in a picture and the method for processing a page provided by at least one embodiment of the present disclosure. In other implementations, the server 330 may execute the method for acquiring character information in a picture and the method for processing a page provided by at least one embodiment of the present disclosure by calling an application stored externally on the server.

[0293] In another example, the user terminal 310 can obtain a picture (for example, a picture obtained by taking a paper medical examination report using a camera of the user terminal), extract character information from the picture, and send the extracted character information to the server 330 via the network 320 or other technologies (for example, Bluetooth communication, infrared communication, etc.); the server 330 can obtain the information of the knowledge graph from the database, and execute part of the method for obtaining character information in the picture and the page processing method provided by at least one embodiment of the present disclosure to verify the received character information and structure the verified character information to obtain a structured data combination; then, the server 330 can store the above structured data combination in the database 340, and send the above structured data combination to the user terminal 310; finally, the user terminal displays (for example, through a pre-designed graphical user interface) the above structured data combination. In some implementations, the user terminal can extract the character information in the picture using an application built into the user terminal. In other implementations, the user terminal can extract the character information in the picture by calling an application stored externally to the user terminal.

[0294] The method, device or system according to the embodiments of the present application can also be Fig.14 The architecture of the computing device 400 shown is implemented.

[0295] Fig.14FIG. 4 shows the architecture of the computing device 400. Fig.14 As shown, the computing device 400 may include a bus 410, one or at least two CPUs 420, a read-only memory (ROM) 430, a random access memory (RAM) 440, a communication port 450 connected to a network, an input / output component 460, a hard disk 470, etc. The storage device in the computing device 400 (e.g., ROM 430 or hard disk 470) may store at least one corresponding instruction of the method for obtaining character information in a picture, the page processing method, and the method for constructing a medical knowledge graph provided by at least one embodiment of the present disclosure, as well as various related data or files. The computing device 400 may also include a human-computer user interface 480. Of course, Fig.14 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig.14 One or at least two components of the computing device shown.

[0296] Although the disclosure has been described in detail above with general descriptions and specific implementation methods, it is obvious to those skilled in the art that some modifications or improvements may be made to the embodiments of the disclosure. Therefore, these modifications or improvements made without departing from the spirit of the disclosure are within the scope of protection claimed by the disclosure.

[0297] The above description is merely an exemplary embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. The protection scope of the present disclosure is determined by the appended claims.

Claims

1. A method for obtaining character information in a picture, comprising: Acquire a picture and extract at least one item of character information included in the picture; as well as Verify the at least one character information based on the knowledge graph, Wherein, verifying at least one item of character information based on the knowledge graph includes: Confirming, based on the knowledge graph, that there is erroneous character information in the at least one item of character information; and Correcting the erroneous character information based on the knowledge graph, The at least one item of character information includes multiple items of character information, and the confirming, based on the knowledge graph, that there is an error in the at least one item of character information includes: Obtaining multiple entities according to the multiple pieces of character information of the image, and selecting an entity to be verified from the multiple entities to determine whether the character information corresponding to the entity to be verified has errors; and According to the hierarchical structure of the knowledge graph, confirm whether there is an error in the character information corresponding to the entity to be verified, and if there is an error in the character information corresponding to the entity to be verified, confirm that the character information corresponding to the entity to be verified is the character information with the error, The step of confirming whether the character information corresponding to the entity to be verified has errors according to the hierarchical structure of the knowledge graph includes: Classifying the multiple entities according to the hierarchical structure of the knowledge graph; Determine the level of the entity to be verified in the hierarchical structure of the knowledge graph; Calculating the similarity between the entity to be verified and all entities in the knowledge graph that have the same level and the same relationship with the entity to be verified, so as to obtain multiple entity similarities related to the entity to be verified; and When the maximum value among the multiple entity similarities is less than a predetermined entity similarity threshold, it is determined that the entity to be verified is an entity to be verified with an error and that character information corresponding to the entity to be verified has an error.

2. The acquisition method according to claim 1, wherein: The correcting the erroneous character information based on the knowledge graph includes: Determine the number of all entities in the knowledge graph that have the same level and the same relationship as the entity to be verified to obtain the number of entities; In the case where the number of entities is equal to 1, directly replace the character information corresponding to the entity to be verified that has an error with the character information corresponding to an entity that has the same level and the same relationship as the entity to be verified, or , Calculating the probability that an entity having the same level and the same relationship as the entity to be verified is the entity to be verified with an error to obtain an entity probability, and when the entity probability is greater than a predetermined entity probability threshold, replacing the character information corresponding to the entity to be verified with the character information corresponding to the entity having the same level and the same relationship as the entity to be verified; and When the number of entities is greater than 1, perform the following steps: Determining at least two candidate entities based on the plurality of entity similarities; Calculating the probability that each of the at least two candidate entities is the entity to be verified with an error, so as to obtain a candidate probability for each of the at least two candidate entities; and The character information corresponding to the entity to be verified with errors is replaced with the character information corresponding to the candidate entity with the maximum candidate probability.

3. The acquisition method according to claim 2, wherein: Determining at least two candidate entities based on the multiple entity similarities comprises: Based on the multiple entity similarities, all entities in the knowledge graph that have the same level and the same relationship as the entity to be verified are sorted in descending order, and a predetermined number of entities at the front of the sequence are selected as the at least two candidate entities; or Based on the multiple entity similarities, all entities in the knowledge graph that have the same level and the same relationship as the entity to be verified are sorted in ascending order, and a predetermined number of entities at the end of the sequence are selected as the at least two candidate entities.

4. The acquisition method according to claim 2 or 3, in, The calculating the probability that each of the at least two candidate entities is the entity to be verified with an error comprises: Determine whether the entity to be verified includes a next-level entity to be verified, to obtain a first determination result, wherein the next-level entity to be verified is all next-level entities subordinate to the entity to be verified; Determine whether the entity to be verified corresponds to a related entity to be verified to obtain a second determination result, wherein the related entity to be verified is all entities that are at the same level as the entity to be verified in the hierarchical structure of the knowledge graph, are associated with the entity to be verified, and have different relationships with the upper-level entity to which the entity to be verified belongs; and Selecting a method for calculating the probability that each of the at least two candidate entities is the entity to be verified with an error based on the first determination result and the second determination result; The first determination result is a first determination sub-result when it is determined that the entity to be verified includes the entity to be verified at a next level, or a second determination sub-result when it is determined that the entity to be verified does not include the entity to be verified at a next level; and The second determination result is the third determination sub-result when it is determined that the entity to be verified corresponds to the relevant entity to be verified or the fourth determination sub-result when it is determined that the entity to be verified does not correspond to the relevant entity to be verified.

5. The acquisition method according to claim 4, wherein: The method for selecting, based on the first determination result and the second determination result, a probability for calculating that each of the at least two candidate entities is the entity to be verified with an error comprises: When the first determination result is a first determination sub-result and the second determination result is a third determination sub-result, calculating a candidate probability of each of the at least two candidate entities by using the next-level entity to be verified and / or the related entity to be verified; When the first determination result is the second determination sub-result and the second determination result is the third determination sub-result, calculating the candidate probability of each of the at least two candidate entities by using the relevant entity to be verified; When the first determination result is a first determination sub-result and the second determination result is a fourth determination sub-result, calculating a candidate probability of each of the at least two candidate entities using the next-level entity to be verified; and When the first determination result is the second determination sub-result and the second determination result is the fourth determination sub-result, the entity similarity of each of the at least two candidate entities is used as the probability that each of the at least two candidate entities is the entity to be verified with errors.

6. The acquisition method according to claim 5, wherein: The calculating the candidate probability of each of the at least two candidate entities by using the relevant entity to be checked comprises: Obtaining the related entities to be verified corresponding to the entity to be verified; The following method is performed for each of the entities included in the related entities to be verified: Obtaining a relevant candidate entity, wherein the relevant candidate entity and each of the at least two candidate entities are at the same level in the hierarchical structure of the knowledge graph, are associated with each of the at least two candidate entities, and the relationship between the relevant candidate entity and the upper-level entity to which each of the at least two candidate entities belongs is equal to the relationship between each of the entities included in the relevant entity to be verified and the entity to be verified, Calculating the similarity between each entity included in the relevant entity to be verified and the relevant candidate entity; Acquire the relevant similarity of each of the at least two candidate entities based on the similarity between each of the entities included in the relevant to-be-verified entities and the relevant candidate entity; and A candidate probability of each of the at least two candidate entities is obtained based on the relevant similarity of each of the at least two candidate entities and the entity similarity.

7. The acquisition method according to claim 6, in, The acquiring the relevant similarity of each of the at least two candidate entities based on the similarity between each of the entities included in the relevant entities to be verified and the relevant candidate entity comprises: Determine the number of entities included in the relevant entities to be verified; When the number of entities included in the relevant entities to be verified is equal to 1, the similarity between the entities included in the relevant entities to be verified and the relevant candidate entities is used as the relevant similarity of each of the at least two candidate entities; When the number of entities included in the relevant entities to be verified is greater than 1, a weighted sum of multiple similarities between the entities included in the relevant entities to be verified and the relevant candidate entities is used as the relevant similarity of each of the at least two candidate entities; The acquiring the candidate probability of each of the at least two candidate entities based on the correlation similarity and entity similarity of each of the at least two candidate entities comprises: taking the weighted sum of the correlation similarity and entity similarity of each of the at least two candidate entities as the candidate probability of each of the at least two candidate entities.

8. The acquisition method according to claim 5, wherein: The calculating the candidate probability of each of the at least two candidate entities by using the next level entity to be checked comprises: Obtaining the next level entity to be verified included in the entity to be verified; Dividing the next-level entities to be verified into at least one next-level entity group to be verified based on the relationship between the next-level entities to be verified and each entity in the next-level entities to be verified; The following method is performed for each of the at least one next-level entity groups to be checked: Acquire a next-level candidate entity group, wherein the next-level candidate entity group is all entities that are subordinate to each of the at least two candidate entities and whose relationship with each of the at least two candidate entities is equal to the relationship between each of the entities included in the next-level entity group to be verified and the entity to be verified, Calculating the similarity between each of the at least one next-level entity group to be verified and the next-level candidate entity group to obtain at least one group similarity corresponding to the at least one next-level entity group to be verified; Obtaining a next level similarity of each of the at least two candidate entities based on the at least one group similarity; and A candidate probability of each of the at least two candidate entities is calculated based on the next level similarity and the entity similarity of each of the at least two candidate entities.

9. The acquisition method according to claim 8, in, The obtaining of at least one group similarity corresponding to the at least one next-level entity group to be verified comprises: Obtaining a maximum similarity of each of the multiple entities included in each of the at least one next-level entity groups to be verified, so as to obtain multiple maximum similarities of the multiple entities included in each of the at least one next-level entity groups to be verified, wherein the maximum similarity is the largest similarity among the similarities between each of the multiple entities included in each of the at least one next-level entity groups to be verified and all entities included in the next-level candidate entity group; and Based on the multiple maximum similarities, obtaining at least one group similarity corresponding to the at least one group of next-level to-be-verified entities; Acquiring at least one group similarity corresponding to the at least one next-level entity group to be verified based on the multiple maximum similarities includes: taking a weighted sum of the multiple maximum similarities as the at least one group similarity corresponding to the at least one next-level entity group to be verified.

10. The acquisition method according to claim 9, in, The obtaining of a next level similarity of each of the at least two candidate entities based on the at least one group similarity comprises: Determine the number of next-level entity groups to be verified included in the next-level entity to be verified, When the number of the next-level entity groups to be verified is equal to 1, the group similarity of the next-level entity groups to be verified included in the next-level entity to be verified is used as the next-level similarity of each of the at least two candidate entities, When the number of the next-level entity groups to be verified is greater than 1, a weighted sum of multiple group similarities of the next-level entity groups to be verified included in the next-level entity to be verified is used as the next-level similarity of each of the at least two candidate entities; The calculating of the candidate probability of each of the at least two candidate entities based on the next level similarity of each of the at least two candidate entities and the entity similarity includes: taking the product of the next level similarity based on each of the at least two candidate entities and the entity similarity as the candidate probability of each of the at least two candidate entities.

11. The acquisition method according to claim 1, further comprising constructing a knowledge graph, comprising: Obtain the data structure of the medical knowledge graph; Export corresponding entities from at least the medical picture archiving and communication system and the laboratory information management system; Entity-aligning the entity exported from the medical image archiving and communication system and the entity exported from the laboratory information management system to obtain a plurality of entity-aligned entities; as well as The aligned multiple entities are filled into the medical knowledge graph based on the data structure of the medical knowledge graph.

12. The acquisition method according to claim 11, wherein constructing a knowledge graph further comprises: Obtaining sub-item descriptions of clinical examination sub-items involved in the laboratory information management system, and processing the sub-item descriptions and filling them into the medical knowledge graph; or / and Export corresponding entities from the hospital information system, align the entities exported from the hospital information system with multiple entities after the entity alignment, and fill the entities exported from the hospital information system and aligned with multiple entities after the entity alignment into the medical knowledge graph based on the data structure of the medical knowledge graph.

13. A device for acquiring character information in a picture, configured to acquire character information in a picture by using the acquisition method according to any one of claims 1 to 12.

14. A non-transitory storage medium comprising computer program instructions stored thereon, in, When the computer program instructions are executed by a processor, the computer executes the acquisition method according to any one of claims 1 to 12.

15. A page processing method, comprising: Get the image of the page; as well as Acquire the verified character information of the image of the page by executing the acquisition method as described in any one of claims 1 to 12; as well as The verified character information of the image of the page is structured to obtain a structured data combination.

16. The page processing method according to claim 15, further comprising: Confirming, based on the knowledge graph, that the image on the page includes a medical image; as well as The medical image is added to the structured data combination.

17. The page processing method according to claim 15 or 16, further comprising: Based on the knowledge graph and the multiple entities corresponding to the verified character information, descriptions corresponding to the multiple entities are added to the structured data combination.

Images