Report interpretation method and device based on large model, equipment and medium

Abstract

The invention provides a report interpretation method and device based on a large model, equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and natural language processing. According to the implementation scheme, text information in a target report image is extracted; identifying an index name in the text information and indication information corresponding to the index name; responding to a first index name of which the indication information is not recognized in the text information, and retrieving the first index name in a preset knowledge base to obtain index knowledge corresponding to the first index name; and inputting the text information, the index knowledge and the preset prompt text into a report interpretation model so as to generate report interpretation information of the target report image according to the text information and the index knowledge through the report interpretation model under the guidance of the preset prompt text.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and natural language processing technology, and specifically to a report interpretation method, device, electronic device, computer-readable storage medium and computer program product based on a large model. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] With the rise of artificial intelligence technology, it has demonstrated excellent capabilities in image recognition and text analysis. Related artificial intelligence technology has shown potential to outperform traditional methods in multiple studies and clinical trials.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a report interpretation method, device, electronic device, computer-readable storage medium and computer program product based on a large model.

[0006] According to one aspect of the present disclosure, a large model-based report interpretation method is provided, including: extracting text information from a target report image; identifying indicator names and indication information corresponding to the indicator names in the text information, the indication information being used to indicate whether the indicator corresponding to the indicator name is within a normal range; in response to a first indicator name for which no indication information is identified in the text information, retrieving the first indicator name from a preset knowledge base to obtain indicator knowledge corresponding to the first indicator name, the indicator knowledge including an indicator reference range; and inputting the text information, the indicator knowledge and a preset prompt text into a report interpretation model, so as to generate report interpretation information for the target report image according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text.

[0007] According to another aspect of the present disclosure, a report interpretation device based on a large model is provided, including: a first extraction unit, configured to extract text information in a target report image; a first recognition unit, configured to recognize an indicator name in the text information and indication information corresponding to the indicator name, the indication information being used to indicate whether the indicator corresponding to the indicator name is within a normal range; a first retrieval unit, configured to retrieve the first indicator name in a preset knowledge base in response to the text information containing a first indicator name for which no indication information is recognized, so as to obtain indicator knowledge corresponding to the first indicator name, the indicator knowledge including an indicator reference range; and a first generation unit, configured to input the text information, the indicator knowledge and a preset prompt text into a report interpretation model, so as to generate report interpretation information of the target report image according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to execute the above-mentioned large model-based report interpretation method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned large model-based report interpretation method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-mentioned large model-based report interpretation method when executed by a processor.

[0011] According to one or more embodiments of the present disclosure, the scalability of report interpretation can be improved while ensuring the accuracy of report interpretation.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0015] Figure 2 A flowchart of a report interpretation method based on a large model according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A flowchart of identifying an indicator name and indicator information according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A flowchart of a report interpretation method based on a large model according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 5 A structural block diagram of a report interpretation device based on a large model according to an embodiment of the present disclosure is shown;

[0019] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0023] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0025] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the large model-based report interpretation method of the present disclosure to be performed.

[0026] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0027] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0028] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to upload report images and send report interpretation requests. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0029] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0030] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0031] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0032] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0033] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0034] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0035] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0036] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0037] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0038] According to the embodiments of the present disclosure, Figure 2 As shown, a report interpretation method based on a large model is provided, including: step S201, extracting text information from a target report image; step S202, identifying the indicator name in the text information and the indication information corresponding to the indicator name, the indication information being used to indicate whether the indicator corresponding to the indicator name is within a normal range; step S203, in response to the text information containing a first indicator name for which no indication information is identified, retrieving the first indicator name in a preset knowledge base to obtain indicator knowledge corresponding to the first indicator name, the indicator knowledge including an indicator reference range; and step S204, inputting the text information, the indicator knowledge and the preset prompt text into a report interpretation model, so as to generate report interpretation information of the target report image according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text.

[0039] Thus, by identifying the text information in the target report image, the indicator name and the corresponding indication information are obtained, and the indicator knowledge is retrieved for the first indicator name for which the indication information is not identified, and the text information, indicator knowledge and preset prompt text are input into the report interpretation model, so that the text information and indicator knowledge are analyzed under the guidance of the preset prompt text to generate report interpretation information. Thus, the scalability of report interpretation can be improved while ensuring the accuracy of report interpretation.

[0040] In some embodiments, the target report image may be a picture of a medical test report of the user uploaded by the user through the client.

[0041] In some embodiments, the user can send a report interpretation request to the server while uploading the image. After receiving the report interpretation request and the report image, the server can interpret the report using the above-mentioned large model-based report interpretation method.

[0042] In some embodiments, text information in the target report image may be extracted by optical character recognition (OCR) technology.

[0043] In some embodiments, before extracting text information, the target report image may be pre-processed (eg, image enhancement) to improve the accuracy of text information extraction.

[0044] In some embodiments, text information may be extracted by combining operations such as layout analysis, title level recognition, and table recognition.

[0045] Among them, through layout analysis, the structure of the document is analyzed and understood for genres such as medical test reports. Through document segmentation, area classification, line and character extraction, etc., the content layout is disassembled and text fragments in different areas are identified. Through title hierarchy recognition, the hierarchical structure content such as titles in medical test reports is identified, so that key content such as department information can be accurately identified, which will help improve the accuracy of physical examination reports in the future. Table recognition is performed through table area detection, row and column segmentation, cell segmentation and extraction, table structure analysis, etc., so as to obtain the text fragments corresponding to each table area, which can further optimize the accuracy of indicator extraction results in subsequent medical test reports.

[0046] Subsequently, on this basis, the indicator name in the text information and its corresponding indication information can be identified. In some embodiments, the indication information is used to indicate whether the indicator corresponding to the indicator name is within a normal range. The indication information can be, for example, a reference interval of a normal value of a certain indicator, or an abnormal indicator mark used to identify whether the indicator is abnormal, such as "+", "-", "↑", "↓", etc.

[0047] In some embodiments, the indicator name and indication information in the text information can be identified through natural language processing technology (such as named entity recognition technology), and by judging whether the indicator name and the indication information belong to the same text segment, it is determined whether there is a corresponding relationship between the indicator name and the indication information.

[0048] If the text information includes one or more first indicator names whose corresponding indication information is not identified, relevant indicator knowledge can be retrieved for the first indicator names in a preset knowledge base.

[0049] In some embodiments, the preset knowledge base may be a knowledge graph or a database, which pre-stores information such as indicator names corresponding to multiple indicators, synonyms of the indicator names, reference ranges, units, clinical significance, knowledge reference sources, etc.

[0050] In some embodiments, the retrieval of relevant indicator knowledge for the first indicator name in the preset knowledge base can be based on the first indicator name to perform text matching on the pre-stored indicator names and synonyms of the pre-stored indicator names in the preset knowledge base to obtain the indicator knowledge corresponding to the first indicator name in the preset knowledge base.

[0051] In some embodiments, the retrieval of relevant indicator knowledge for the first indicator name in the preset knowledge base can be based on semantic similarity, matching preset indicator names whose semantic similarity with the first indicator name is greater than the preset similarity in the preset knowledge base to obtain corresponding indicator knowledge.

[0052] In some embodiments, the retrieval of relevant indicator knowledge for the first indicator name in the preset knowledge base can be firstly based on the first indicator name, the text of the pre-stored indicator name and the synonyms of the pre-stored indicator name in the preset knowledge base is accurately matched. If the corresponding indicator is not retrieved through the text matching, then based on the semantic similarity, the preset indicator name with a semantic similarity greater than the preset similarity with the first indicator name is matched in the preset knowledge base, thereby obtaining the corresponding indicator knowledge. In this way, the obtained indicator knowledge can be made more accurate.

[0053] In some embodiments, while identifying the indicator name and indication information, the report name, sample source and other information in the text information can also be further identified through natural language processing technology (such as named entity recognition technology), and the above information can also be used as a query term when performing indicator retrieval. Thus, the indicator knowledge obtained can be made more accurate.

[0054] In some embodiments, the indicator knowledge may include an indicator reference range. In some embodiments, the indicator knowledge may also include information such as the report sheet to which the indicator belongs, clinical significance, and knowledge reference source.

[0055] In some embodiments, after obtaining the indicator knowledge corresponding to the first indicator name of all unrecognized indication information, the text information, indicator knowledge and preset prompt text are input into the report interpretation model together, so that the report interpretation model, under the guidance of the preset prompt text, extracts the indicator name, indicator value and reference range in the text information and indicator knowledge accordingly, and analyzes and interprets the information of each indicator extracted in combination with the relevant indicator knowledge to generate report interpretation information.

[0056] In some embodiments, the report interpretation model may be a general large language model.

[0057] In some embodiments, the report interpretation model may be a large language model obtained through supervised fine-tuning training based on a general large language model. Each sample data may include text information in a sample report, relevant indicator knowledge, and sample report interpretation information.

[0058] In some embodiments, the preset prompt text can be used to prompt the report interpretation model to extract information such as indicator names, indicator values, indication information, reference ranges, etc. in the text information and indicator knowledge, and analyze and interpret the extracted information of each indicator in combination with relevant indicator knowledge to generate report interpretation information.

[0059] In some embodiments, if all the index names extracted in the text information contain corresponding indication information, the text information and the preset prompt text can be directly input into the report interpretation model to obtain the report interpretation information generated by the report interpretation model.

[0060] In some embodiments, Figure 3 As shown, identifying the indicator name in the text information and the indication information corresponding to the indicator name may include: step S301, performing named entity recognition on the text information to obtain the indicator name; step S302, matching the text fragments containing the indicator name in the text information based on a preset regular expression to identify the reference interval information corresponding to the indicator name; and step S303, in response to identifying the reference interval information, determining the reference interval information as indication information.

[0061] In some embodiments, identifying the indicator names in text information and the indication information corresponding to the indicator names can be by first performing named entity recognition on the text information to obtain all the indicator names in the text information, and then determining the text segment where each indicator name is located according to each indicator name (for example, a line of text in the test report corresponding to the indicator name), and matching the text segment based on a preset regular expression to obtain the reference interval information corresponding to the corresponding indicator name.

[0062] In some embodiments, the above regular expression may include the text format of the indicator name and reference interval information, which is used to match the same text format in each text fragment, and determine the text information corresponding to the reference interval information part as the reference interval information corresponding to the corresponding indicator name.

[0063] Therefore, first, the names of various indicators in the text information are obtained through named entity recognition; then, based on the preset regular expression, the text fragments containing each indicator name are matched, so that the reference interval information corresponding to each indicator name can be accurately and efficiently identified.

[0064] In some embodiments, if a certain indicator name is not matched to the reference interval information through a regular expression, a search may be performed in a preset knowledge base for the indicator name to obtain the corresponding indicator knowledge.

[0065] In some embodiments, identifying the indicator name in the text information and the indication information corresponding to the indicator name may also include: in response to not identifying the reference interval information, performing semantic understanding on the text fragment to identify the abnormal indicator mark corresponding to the indicator name; and in response to identifying the abnormal indicator mark, determining the abnormal indicator mark as the indication information.

[0066] In some embodiments, if an indicator name is not matched to the reference interval information through a regular expression, the text segment containing the indicator name can be further semantically understood and analyzed based on semantic understanding to determine whether the text segment contains abnormal indicator marks, such as "+", "-", "↑", "↓", etc.

[0067] Therefore, when the reference interval information is not identified, the text fragment can be further semantically understood to determine whether the text fragment contains the abnormal indicator mark corresponding to the indicator name. In this way, the original indication information of each indicator in the report can be accurately and comprehensively obtained, thereby improving the accuracy of subsequent report interpretation.

[0068] In some embodiments, when a certain indicator name has neither its corresponding reference interval information nor its corresponding abnormal indicator mark identified, a search can be performed in a preset knowledge base for the indicator name to obtain its corresponding indicator knowledge.

[0069] In some embodiments, acquiring a target report image may include: in response to receiving a first image, performing image classification on the first image to determine an image category of the first image; and in response to determining that the image category of the first image is a report category, determining the first image as a target report image.

[0070] The above-mentioned image classification can be implemented based on a trained image classification model. The image classification model can classify the input images into report categories and non-report categories (such as landscape images, handwritten note images, etc.), which can be obtained by training based on multiple sample images with labeled image categories.

[0071] Therefore, in response to receiving the first report image uploaded by the user, it is first determined whether the image is a report-type image, so that interference images can be quickly screened out and resource waste caused by subsequent processing of the interference images can be avoided.

[0072] In some embodiments, in response to determining that the image category of the first image is a report category, determining the first image as a target report image may include: in response to determining that the image category of the first image is a report category, performing text extraction on the first image to obtain a first text in the first image; and in response to determining that the first text does not contain preset keywords, determining the first image as a target report image.

[0073] Among them, some keywords can be preset and the report images containing these keywords can be screened out by performing keyword matching.

[0074] In some embodiments, the preset keywords may be "discharge", "admission", "medical record" and other keywords used to indicate the type of report. Such keywords can quickly filter out redundant reports that do not need to be interpreted.

[0075] In some embodiments, the preset keywords may be specific keywords in reports that are not suitable for machine interpretation, such as electrocardiogram reports, hearing tests, psychological tests, etc. Such keywords can quickly screen out reports that are not suitable for machine interpretation.

[0076] Therefore, by further identifying the text in the report image and further screening the report image through keyword matching, interfering document images or report images that are not suitable for machine interpretation can be screened out, avoiding the waste of resources caused by subsequent processing of the above images, while further improving the accuracy of report interpretation.

[0077] In some embodiments, in response to identifying the above-mentioned non-report type image or identifying the above-mentioned report image containing the above-mentioned keywords, subsequent information extraction and report interpretation operations will no longer be performed on the above-mentioned report image, and corresponding prompt information will be fed back to the user at the same time, for example, "Figure X and Figure X do not seem to be test reports. You can try to change the picture or skip and continue the interpretation."

[0078] In some embodiments, a user can upload multiple first images at one time through the client. If one or more target report images are obtained after identifying the multiple first images, subsequent report interpretation operations can be performed only based on the identified one or more target report images.

[0079] In some embodiments, the preset prompt text may include a first prompt text and a second prompt text. Through a report interpretation model, under the guidance of the preset prompt text, generating report interpretation information of a target report image according to text information and indicator knowledge may include: through the report interpretation model, under the guidance of the first prompt text, obtaining abnormal indicator information according to text information and indicator knowledge; and through the report interpretation model, under the guidance of the second prompt text, generating report interpretation information at least according to the abnormal indicator information.

[0080] In some embodiments, the preset prompt text may include prompt texts corresponding to multiple steps, such as a first prompt text corresponding to the abnormal indicator information extraction step and a second prompt text corresponding to the report interpretation information generation step. For example, the first prompt text may be "Please extract the abnormal indicator in <report information> and the numerical value and reference range information corresponding to the indicator according to <report information> and <medical reference knowledge>"; the second prompt text may be "Provide professional medical interpretation of the abnormal items / parts extracted above". The above-mentioned report information corresponds to the text information in the target report image, and the above-mentioned medical reference knowledge corresponds to the indicator knowledge.

[0081] Therefore, by setting the preset prompt text, the report interpretation model can extract abnormal indicator information according to the steps, and generate report interpretation information based on the abnormal indicator information, so as to further improve the accuracy of report interpretation, reduce the amount of data processing in the generation of report interpretation information, and improve the efficiency of report interpretation information generation.

[0082] In some embodiments, the first prompt text can be further configured to guide the report interpretation model to prioritize the use of text information in the target report image for abnormal indicator identification, and then identify abnormal indicators based on the reference interval information in the indicator knowledge. For example, the first prompt text can be "When <report information> contains a normal reference interval, and the test result of a certain examination item or part is not within this interval, this examination item or part can be marked as 'abnormal item / part'; if <report information> does not provide a normal reference interval, match it with the reference range in <medical reference knowledge>, and mark the examination items or parts that are not within this reference interval as 'abnormal items / parts'". Thus, by giving priority to the use of the original information in the target report image, the accuracy of report interpretation can be improved.

[0083] In some embodiments, the preset prompt text may also include a third prompt text. Through the report interpretation model, under the guidance of the preset prompt text, the report interpretation information of the target report image is generated according to the text information and indicator knowledge. It may also include: through the report interpretation model, under the guidance of the third prompt text, obtaining the user age information in the text information; and wherein, through the report interpretation model, under the guidance of the first prompt text, obtaining abnormal indicator information according to the text information and indicator knowledge may include: through the report interpretation model, under the guidance of the first prompt text, obtaining abnormal indicator information according to the text information, indicator knowledge and user age information.

[0084] In some embodiments, the preset prompt text may include a third prompt text, a first prompt text, and a second prompt text in sequence. The third prompt text may be used to guide the report interpretation model to extract the user age information from the text information. For example, the third prompt text may be "Extract the user age from <report information>".

[0085] In some embodiments, the first prompt text can be further configured to guide the report interpretation model to extract abnormal indicator information based on different information according to the user's age information.

[0086] In some exemplary embodiments, the first prompt text can be further configured as "When the <report information> contains a normal reference interval, and the test result of a certain examination item or part is not within this interval, this examination item or part can be marked as 'abnormal item / part'; if the <report information> does not provide a normal reference interval, determine whether the age of the subject is less than 18 years old. If he is less than 18 years old, end the current step and directly execute the next step; if the age of the subject is not less than 18 years old, match it with the reference range in the <medical reference knowledge>, and mark the examination item or part that is not within this reference interval as 'abnormal item / part'."

[0087] Therefore, by setting the preset prompt text, the report interpretation model can further extract the user's age information, and perform the extraction of abnormal indicator information based on the user's age information, thereby further improving the accuracy of the abnormal indicator information extraction, and then improving the accuracy of the report interpretation.

[0088] In some embodiments, the preset prompt text may further include prompt text for guiding the report interpretation model to extract the report name and user gender information. In addition, the first prompt text may be further configured to obtain the reference interval information of the relevant indicator in the indicator knowledge based on the report name, user age and user gender information when the normal reference interval is not provided in the text information, and determine whether the corresponding indicator is an abnormal indicator.

[0089] In some exemplary embodiments, the first prompt text may be "When <Report Information> contains a normal reference interval, and the test result of a certain examination item or part is not within this interval, this examination item or part can be marked as 'abnormal item / part'; if <Report Information> does not provide a normal reference interval, determine whether the age of the subject is less than 18 years old. If he is less than 18 years old, end the current step and directly execute the next step; if the age of the subject is not less than 18 years old, use the examination item name (report name), age, gender, etc. in the <Report Information> to match the reference range in the <Medical Reference Knowledge>, and mark the examination items or parts that are not within this reference interval as 'abnormal items / parts'."

[0090] In some embodiments, the prompt text for guiding the report interpretation model to extract the report form name may further include a prompt text for guiding the report interpretation model to refine the report form name with reference to the indicator knowledge. For example, it may be "Describe the report form name as detailed as possible, such as blood routine, urine routine, etc., and refer to <medical reference knowledge> for refinement".

[0091] In some embodiments, the preset prompt text may also include a fourth prompt text. Through the report interpretation model, under the guidance of the preset prompt text, the report interpretation information of the target report image is generated according to the text information and indicator knowledge. It may also include: through the report interpretation model, under the guidance of the fourth prompt text, determining whether the text information contains interpretable report information; and wherein, through the report interpretation model, under the guidance of the first prompt text, according to the text information and indicator knowledge, obtaining abnormal indicator information may include: in response to determining that the text information contains report information, through the report interpretation model, under the guidance of the first prompt text, according to the report information and indicator knowledge, obtaining abnormal indicator information.

[0092] In some embodiments, the preset prompt text may include a fourth prompt text, a third prompt text, a first prompt text and a second prompt text in sequence. In which, the fourth prompt text may be used to guide the report interpretation model to determine whether the text information contains interpretable report information.

[0093] In some embodiments, the interpretable report information may include, for example, pathological examination reports, radiological examination results, nuclear magnetic resonance examination results, diagnostic conclusions, etc. of preset examination items, wherein the preset examination items are used to determine the scope of the report sheets suitable for interpretation by the report interpretation model, for example, electrocardiogram reports, hearing tests, psychological tests, etc. do not fall within the above scope. In some exemplary embodiments, the fourth prompt text may include "determine whether <report sheet information> contains medical test data and results of <preset examination items>, and if so, proceed to the next step".

[0094] Therefore, by setting the preset prompt text, the report interpretation model can be used again to determine whether the current target report is interpretable report information before information extraction; in response to the report information containing interpretable information, subsequent information extraction and interpretation are performed, thereby further improving the accuracy of report interpretation.

[0095] In some embodiments, the fourth prompt text may also include "If there is no specific medical test data and results in the <report information>, please respond: 'Sorry, I can only recognize the medical test report. You can try to upload a new medical report', and then do not continue with the following steps", so as to guide the report interpretation model to send a corresponding prompt message to the user when it recognizes that the text information does not contain interpretable report information, and no longer perform subsequent report interpretation steps. In this way, the user experience is further improved while avoiding waste of resources.

[0096] In some embodiments, the preset prompt text may also include prompt text for guiding the report interpretation model to generate health advice and treatment department recommendation information. It is understandable that relevant technicians can set the prompt content in the preset prompt text according to actual needs, and there is no limitation here.

[0097] Figure 4 A flowchart of a report interpretation method based on a large model according to an exemplary embodiment of the present disclosure is shown.

[0098] In some exemplary embodiments, Figure 4 As shown, the report interpretation method based on the large model may include: step S401, judging whether the first input image is a target report image; step S402, in response to the first image not being a target report image, ending the report interpretation, and feeding back corresponding prompt information to the user; step S403, in response to the first image being a target report image, extracting report information from the text information of the target report image, wherein the report information includes the indicator name, indicator indication information, report form name, user age, etc.; step S404, for the first indicator name that does not contain corresponding indication information in the text information of the target report image, retrieving indicator knowledge corresponding to the first indicator name in a preset knowledge base; step S405, inputting the text information of the target report image, the retrieved indicator knowledge and the preset prompt text into the report interpretation model, so as to first judge whether the text information contains interpretable report information through the report interpretation model; step S406, in response to the text information not containing interpretable report information, ending the report interpretation, and feeding back corresponding prompt information to the user; step S407, in response to the text information containing interpretable report information, generating report interpretation information based on the text information and indicator knowledge.

[0099] In some embodiments, Figure 5As shown, a report interpretation device 500 based on a large model is also provided, including: a first extraction unit 510, configured to extract text information from a target report image; a first recognition unit 520, configured to recognize an indicator name in the text information and indication information corresponding to the indicator name, the indication information being used to indicate whether the indicator corresponding to the indicator name is within a normal range; a first retrieval unit 530, configured to retrieve the first indicator name in a preset knowledge base in response to the text information containing a first indicator name for which no indication information is recognized, so as to obtain indicator knowledge corresponding to the first indicator name, the indicator knowledge including an indicator reference interval; and a first generation unit 540, configured to input text information, indicator knowledge and preset prompt text into a report interpretation model, so as to generate report interpretation information of the target report image according to the text information and indicator knowledge through the report interpretation model under the guidance of the preset prompt text.

[0100] in, Figure 5 The various units of the device 500 shown in FIG. 5 can be used in conjunction with the reference Figure 2 The steps in the large model-based report interpretation method described above correspond to each other. Therefore, the operations, features and advantages described above for the large model-based report interpretation method are also applicable to the device 500 and the various units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0101] In some embodiments, the first identification unit may include: a first acquisition subunit, configured to perform named entity recognition on text information to obtain the indicator name; a first identification subunit, configured to match text fragments containing the indicator name in the text information based on a preset regular expression to identify reference interval information corresponding to the indicator name; and a first determination subunit, configured to determine the reference interval information as indication information in response to identifying the reference interval information.

[0102] In some embodiments, the first recognition unit may also include: a second recognition subunit, configured to perform semantic understanding of the text fragment in response to failure to recognize the reference interval information to identify the abnormal indicator mark corresponding to the indicator name; and a second determination subunit, configured to determine the abnormal indicator mark as indication information in response to recognition of the abnormal indicator mark.

[0103] In some embodiments, acquiring a target report image may include: in response to receiving a first image, performing image classification on the first image to determine an image category of the first image; and in response to determining that the image category of the first image is a report category, determining the first image as a target report image.

[0104] In some embodiments, in response to determining that the image category of the first image is a report category, determining the first image as a target report image may include: in response to determining that the image category of the first image is a report category, performing text extraction on the first image to obtain a first text in the first image; and in response to determining that the first text does not contain preset keywords, determining the first image as a target report image.

[0105] In some embodiments, the preset prompt text may include a first prompt text and a second prompt text, and the first generation unit may include: a second acquisition subunit, configured to acquire abnormal indicator information according to text information and indicator knowledge through a report interpretation model under the guidance of the first prompt text; and a first generation subunit, configured to generate report interpretation information at least based on the abnormal indicator information through a report interpretation model under the guidance of the second prompt text.

[0106] In some embodiments, the preset prompt text may also include a third prompt text, and the first generation unit may also include: a third acquisition subunit, configured to obtain user age information in the text information through a report interpretation model under the guidance of the third prompt text; and wherein the second acquisition subunit may be further configured to: obtain abnormal indicator information based on text information, indicator knowledge and user age information through a report interpretation model under the guidance of the first prompt text.

[0107] In some embodiments, the preset prompt text may also include a fourth prompt text, and the first generation unit may also include: a third determination subunit, configured to determine whether the text information contains interpretable report information through a report interpretation model under the guidance of the fourth prompt text; and wherein the second acquisition subunit may be further configured to: in response to determining that the text information contains report information, obtain abnormal indicator information based on the report information and indicator knowledge through a report interpretation model under the guidance of the first prompt text.

[0108] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0110] refer to Figure 6, a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0111] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0112] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600. The input unit 606 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0113] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the above-mentioned report interpretation method based on the large model. For example, in some embodiments, the above-mentioned report interpretation method based on the large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the above-mentioned report interpretation method based on the large model may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the above-mentioned large model-based report interpretation method in any other appropriate manner (for example, by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0119] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0121] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A report interpretation method based on a large model, comprising: Extract text information from target report images; Identify an indicator name in the text information and indication information corresponding to the indicator name, wherein the indication information is used to indicate whether the indicator corresponding to the indicator name is within a normal range; In response to the text information containing a first indicator name that is not recognized as the indication information, searching the first indicator name in a preset knowledge base to obtain indicator knowledge corresponding to the first indicator name, the indicator knowledge including an indicator reference interval; as well as The text information, the indicator knowledge and the preset prompt text are input into a report interpretation model so that the report interpretation information of the target report image is generated according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text.

2. The method according to claim 1, wherein: The identifying of the indicator name in the text information and the indication information corresponding to the indicator name includes: Performing named entity recognition on the text information to obtain the indicator name; Based on a preset regular expression, matching the text segment containing the indicator name in the text information to identify the reference interval information corresponding to the indicator name; and In response to identifying the reference section information, the reference section information is determined as the indication information.

3. The method according to claim 2, wherein: The identifying the indicator name in the text information and the indication information corresponding to the indicator name also includes: In response to failure to identify the reference interval information, performing semantic understanding on the text segment to identify an abnormal indicator mark corresponding to the indicator name; and In response to identifying the abnormal indicator mark, the abnormal indicator mark is determined as the indication information.

4. The method according to any one of claims 1 to 3, wherein: The acquisition of the target report image includes: In response to receiving a first image, performing image classification on the first image to determine an image category of the first image; and In response to determining that the image category of the first image is a report category, the first image is determined as the target report image.

5. The method according to claim 4, wherein: In response to determining that the image category of the first image is a report category, determining the first image as the target report image includes: In response to determining that the image category of the first image is a report category, performing text extraction on the first image to obtain a first text in the first image; and In response to determining that the first text does not include a preset keyword, the first image is determined as the target report image.

6. The method according to any one of claims 1 to 5, wherein: The preset prompt text includes a first prompt text and a second prompt text, and the report interpretation information of the target report image generated by the report interpretation model according to the text information and the indicator knowledge under the guidance of the preset prompt text includes: By means of the report interpretation model, under the guidance of the first prompt text, according to the text information and the indicator knowledge, abnormal indicator information is obtained; and Through the report interpretation model, under the guidance of the second prompt text, the report interpretation information is generated at least according to the abnormal indicator information.

7. The method according to claim 6, wherein: The preset prompt text further includes a third prompt text, and the report interpretation information of the target report image generated according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text also includes: Through the report interpretation model, under the guidance of the third prompt text, the user age information in the text information is obtained; and wherein, The obtaining of abnormal indicator information through the report interpretation model under the guidance of the first prompt text according to the text information and the indicator knowledge includes: Through the report interpretation model, under the guidance of the first prompt text, the abnormal indicator information is obtained according to the text information, the indicator knowledge and the user age information.

8. The method according to claim 6 or 7, wherein: The preset prompt text further includes a fourth prompt text, and the report interpretation information of the target report image generated according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text also includes: Through the report interpretation model, under the guidance of the fourth prompt text, it is determined whether the text information contains interpretable report information; and wherein, The obtaining of abnormal indicator information through the report interpretation model under the guidance of the first prompt text according to the text information and the indicator knowledge includes: In response to determining that the text information contains the report information, the abnormal indicator information is obtained through the report interpretation model, under the guidance of the first prompt text, according to the report information and the indicator knowledge.

9. A report interpretation device based on a large model, comprising: A first extraction unit is configured to extract text information from a target report image; A first identification unit is configured to identify an indicator name in the text information and indication information corresponding to the indicator name, wherein the indication information is used to indicate whether the indicator corresponding to the indicator name is within a normal range; A first retrieval unit is configured to, in response to the text information containing a first indicator name that does not identify the indication information, retrieve the first indicator name in a preset knowledge base to obtain indicator knowledge corresponding to the first indicator name, the indicator knowledge including an indicator reference interval; as well as The first generating unit is configured to input the text information, the indicator knowledge and the preset prompt text into a report interpretation model, so as to generate report interpretation information of the target report image according to the text information and the indicator knowledge through the report interpretation model under the guidance of the preset prompt text.

10. The device according to claim 9, wherein: The first identification unit comprises: A first acquisition subunit is configured to perform named entity recognition on the text information to obtain the indicator name; a first identification subunit configured to match the text segment containing the indicator name in the text information based on a preset regular expression to identify the reference interval information corresponding to the indicator name; and The first determining subunit is configured to, in response to identifying the reference interval information, determine the reference interval information as the indication information.

11. The device according to claim 10, wherein: The first identification unit also includes: a second identification subunit, configured to, in response to failure to identify the reference interval information, perform semantic understanding on the text segment to identify an abnormal indicator mark corresponding to the indicator name; and The second determining subunit is configured to, in response to identifying the abnormal indicator mark, determine the abnormal indicator mark as the indication information.

12. The device according to any one of claims 9 to 11, wherein: The acquisition of the target report image includes: In response to receiving a first image, performing image classification on the first image to determine an image category of the first image; and In response to determining that the image category of the first image is a report category, the first image is determined as the target report image.

13. The device according to claim 12, wherein: In response to determining that the image category of the first image is a report category, determining the first image as the target report image includes: In response to determining that the image category of the first image is a report category, performing text extraction on the first image to obtain a first text in the first image; and In response to determining that the first text does not include a preset keyword, the first image is determined as the target report image.

14. The device according to any one of claims 9 to 13, wherein: The preset prompt text includes a first prompt text and a second prompt text, and the first generating unit includes: a second acquisition subunit configured to acquire abnormal indicator information according to the text information and the indicator knowledge through the report interpretation model and under the guidance of the first prompt text; and The first generating subunit is configured to generate the report interpretation information at least according to the abnormal indicator information through the report interpretation model and under the guidance of the second prompt text.

15. The device according to claim 14, wherein: The preset prompt text also includes a third prompt text, and the first generating unit also includes: The third acquisition subunit is configured to acquire the user age information in the text information through the report interpretation model under the guidance of the third prompt text; and wherein, The second acquisition subunit is further configured as follows: Through the report interpretation model, under the guidance of the first prompt text, the abnormal indicator information is obtained according to the text information, the indicator knowledge and the user age information.

16. The device according to claim 14 or 15, wherein: The preset prompt text further includes a fourth prompt text, and the first generating unit further includes: The third determination subunit is configured to determine whether the text information contains interpretable report information through the report interpretation model under the guidance of the fourth prompt text; and wherein, The second acquisition subunit is further configured as follows: In response to determining that the text information contains the report information, the abnormal indicator information is obtained through the report interpretation model, under the guidance of the first prompt text, according to the report information and the indicator knowledge.

17. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-8.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images