Medical image report structured processing method based on thinking chain and related device

Abstract

The invention discloses a medical image report structured processing method based on a thinking chain and a related device, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining at least one thinking chain, the thinking chain comprises a plurality of nodes with a hierarchical relationship, the instruction prompt ranges of the plurality of nodes are gradually decreased, and the instruction prompt range of the plurality of nodes is gradually decreased for the at least one thinking chain; a to-be-processed medical image report text, structured output of a previous node and instruction prompt of a current node are input into a pre-trained large language model, so that the large language model sequentially performs accurate, standard and complete structured data extraction on the medical image report text according to hierarchical logic of a thinking chain; according to the method, errors and omissions in the data extraction process are avoided, the accuracy, normalization and integrity of the extracted structured data are guaranteed, the whole process is completed automatically, manual participation is not needed, and the structured processing efficiency of the medical image report is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method for structuring medical imaging reports based on thought chains and related devices. Background Art

[0002] Artificial intelligence-assisted medical diagnosis and treatment is an emerging key research and application direction in the field of clinical medicine in recent years. Various types of information in clinical medicine databases provide a rich source of data for the training of artificial intelligence models. Among them, medical imaging reports have become one of the most important data supports for artificial intelligence model learning because of their ability to be correlated with multimodal imaging data.

[0003] However, due to the complexity and variability of clinical scenarios, especially the different writing habits of different radiologists, medical imaging reports are often stored in clinical databases in an irregular, unstructured or semi-structured form, which brings inconvenience to the extraction, analysis, management and subsequent model training of medical imaging data. Therefore, processing medical imaging reports into a standardized and structured format and retaining the accuracy and completeness of the information as much as possible is of great significance for artificial intelligence-assisted medical care.

[0004] Since the structured processing of imaging report text involves professional medical imaging knowledge, it is currently generally done manually, which is inefficient, time-consuming and labor-intensive. Summary of the invention

[0005] In view of the above problems, this application provides a medical imaging report structured processing method and related devices based on thought chain, so as to achieve the purpose of accurate, standardized and complete structured data extraction from medical imaging report text. The specific scheme is as follows:

[0006] The first aspect of the present application provides a method for structured processing of medical imaging reports based on thought chains, comprising:

[0007] Obtain the text of the medical imaging report to be processed;

[0008] Determine at least one thought chain, wherein the thought chain includes a plurality of nodes having a hierarchical relationship, and the instruction prompt ranges of the plurality of nodes decrease gradually;

[0009] For the at least one thought chain, the medical imaging report text, the structured output of the previous node and the instruction prompt of the current node are input into a pre-trained large language model to obtain the structured output of the current node, wherein the large language model is pre-trained using a medical imaging report text sample with annotated structured data;

[0010] According to the structured output of the last node in the at least one thought chain, the structured data of the medical imaging report text is determined.

[0011] In a possible implementation, if the at least one thought chain includes a thought chain, the method for training the large language model includes:

[0012] According to the annotated structured data of the medical imaging report text sample, determine the structured output of each node of the medical imaging report text sample in the thought chain as the output label of each node;

[0013] Inputting the medical imaging report text sample, the structured output of the previous node and the instruction prompt of the current node into the large language model to obtain the structured output of the current node;

[0014] Calculate the autoregressive loss of the structured output and output label of the current node;

[0015] When the autoregressive loss converges, the training of the large language model is stopped.

[0016] In a possible implementation, if the at least one thought chain includes a thought chain, determining the structured data of the medical imaging report text according to the structured output of the last node in the at least one thought chain includes:

[0017] Convert the structured output of the last node in the thought chain into a list of triples;

[0018] The triple list is determined as structured data of the medical imaging report text.

[0019] In a possible implementation, if the at least one thought chain includes a first thought chain and a second thought chain, the method for training the large language model includes:

[0020] According to the structured data of the medical imaging report text sample, determine the structured output of the medical imaging report text sample at each node in the first thinking chain as the first output label of each node in the first thinking chain, and determine the structured output of the medical imaging report text sample at each node in the second thinking chain as the second output label of each node in the second thinking chain;

[0021] Inputting the medical imaging report text sample, the structured output of the previous node in the first thought chain, and the instruction prompt of the current node into the large language model to obtain the structured output of the current node of the first thought chain;

[0022] Calculate the first autoregressive loss of the structured output of the current node of the first thought chain and the first output label;

[0023] Inputting the medical imaging report text sample, the structured output of the previous node in the second thinking chain, and the instruction prompt of the current node into the large language model to obtain the structured output of the current node of the second thinking chain;

[0024] Calculate the second autoregressive loss of the structured output of the current node of the second thinking chain and the second output label;

[0025] Calculate the similarity between the structured output of the last node in the first thought chain and the structured output of the last node in the second thought chain to obtain the consistency loss;

[0026] Performing a weighted summation on the first autoregressive loss, the second autoregressive loss, and the consistency loss to obtain a total loss;

[0027] When the total loss converges, the training of the large language model is stopped.

[0028] In a possible implementation, if the at least one thought chain includes a first thought chain and a second thought chain, determining the structured data of the medical imaging report text according to the structured output of the last node in the at least one thought chain includes:

[0029] Respectively converting the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain into a triple list;

[0030] The intersection or union of the two triple lists is determined as the structured data of the medical imaging report text.

[0031] In a possible implementation, if the current node is the first node in the thinking chain, the instruction prompt of the current node includes system prompts and strategy prompts, the system prompts include role prompts, task prompts and structured output formats, the structured output of the previous node is empty, and the instruction prompts of other nodes in the thinking chain except the first node only include the strategy prompts, and the strategy prompts are used to define the content of the structured output of the corresponding node.

[0032] The second aspect of the present application provides a medical imaging report structured processing device based on thought chain, comprising:

[0033] A report text acquisition unit, used to acquire the medical imaging report text to be processed;

[0034] A thought chain determination unit, used to determine at least one thought chain, wherein the thought chain includes a plurality of nodes having a hierarchical relationship, and the instruction prompt ranges of the plurality of nodes decrease gradually;

[0035] A model processing unit is used for inputting the medical imaging report text, the structured output of the previous node and the instruction prompt of the current node into a pre-trained large language model for the at least one thought chain to obtain the structured output of the current node, wherein the large language model is pre-trained using medical imaging report text samples with annotated structured data;

[0036] The structured data determination unit is used to determine the structured data of the medical imaging report text according to the structured output of the last node in the at least one thought chain.

[0037] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for structuring medical imaging reports based on thought chains according to the first aspect or any implementation method of the first aspect.

[0038] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the medical imaging report structured processing method based on the thought chain mentioned in the first aspect or any implementation method of the first aspect.

[0041] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the method for structuring medical imaging reports based on the thought chain according to the above-mentioned first aspect or any implementation method of the first aspect.

[0042] Through the above technical scheme, the present application provides a method and related devices for structuring medical imaging reports based on a thinking chain. Since the thinking chain includes multiple nodes with a hierarchical relationship, and the instruction prompt ranges of the multiple nodes decrease step by step, for at least one thinking chain, the medical imaging report text to be processed, the structured output of the previous node and the instruction prompt of the current node are input into a pre-trained large language model, so that the large language model extracts accurate, standardized and complete structured data from the medical imaging report text in sequence according to the hierarchical logic of the thinking chain, avoiding errors and omissions in the data extraction process, ensuring the accuracy, standardization and completeness of the extracted structured data, and the entire process is completed automatically without human participation, thereby improving the efficiency of structuring medical imaging reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A schematic diagram of a system architecture provided for this application;

[0045] Figure 2 A schematic diagram of the structure of a terminal 100 provided in this application;

[0046] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0047] Figure 4 A flowchart of a method for structuring medical imaging reports based on thought chains provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a large language model training for a single-path thinking chain provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structured processing of medical imaging reports of a single-path thinking chain provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a large language model training of a dual-path thinking chain provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structured processing of medical imaging reports of the dual-path thinking chain provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of a large language model training with a thought chain of intermediate nodes removed provided in an embodiment of the present application;

[0053] Fig.10 A schematic diagram of a medical imaging report structured processing by removing an intermediate node thought chain provided in an embodiment of the present application;

[0054] Fig.11 A schematic diagram of the structure of a medical image report structured processing device based on thought chain provided in an embodiment of the present application;

[0055] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0057] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0058] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0059] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0060] Among them, an application can be installed on the terminal 100, and the above application can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface, such as the medical imaging report text to be processed, and send the above parameters to the server 200. The server 200 can obtain the processing results based on the received parameters, such as the structured data of the medical imaging report text, and return the processing results to the terminal 100.

[0061] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0062] Next describe Figure 1 The product form of the mid-terminal 100;

[0063] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0064] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0065] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0066] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0067] Among them, the input device 132 can receive input data and the like.

[0068] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display an interface for uploading a medical imaging report text to be processed, a structured processing result of a medical imaging report text, and the like.

[0069] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0070] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0071] Among them, the memory 120 can be used to store software codes related to the medical imaging report structured processing method based on the thought chain, the processor 170 can execute the steps of the medical imaging report structured processing method based on the thought chain, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to realize corresponding functions.

[0072] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0073] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0074] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0075] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0076] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0077] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0078] Next describe Figure 1 The product form of the server 200;

[0079] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0080] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0082] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0083] Among them, the memory 204 can be used to store software codes related to the medical imaging report structured processing method based on the thinking chain, the processor 202 can execute the steps of the medical imaging report structured processing method based on the thinking chain of the chip, and can also schedule other units to realize corresponding functions.

[0084] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0085] In some implementations, the entire medical imaging report text structuring task process is broken down into tasks such as noun recognition, relationship extraction, and terminology normalization, and then independent models are trained for each task. In this type of solution, the various tasks lack organic unity, the technical solutions involved are different, and it is difficult to associate and correspond with each other, which is prone to errors and omissions in information extraction, affecting the accuracy, standardization, and completeness of the structured data extracted from the medical imaging report.

[0086] In order to solve the above problems, the embodiment of the present application provides a method for structuring medical image reports based on thought chain. The method for structuring medical image reports based on thought chain of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0087] Reference Figure 4 , Figure 4 A flowchart of a method for structuring medical imaging reports based on thought chain is provided in an embodiment of the present application, such as Figure 4 As shown, a method for structuring medical image reports based on thought chains provided in an embodiment of the present application may include steps 401 to 404, and these steps are described in detail below.

[0088] 401: Get the text of the medical imaging report to be processed;

[0089] The text of medical imaging reports generally includes the patient's basic information, imaging examination results, conclusions and recommendations. Due to the different writing habits of different radiologists, medical imaging reports are often stored in clinical databases in non-standard unstructured or semi-structured forms.

[0090] Exemplarily, the medical imaging report text to be processed is extracted from the clinical database.

[0091] 402: determining at least one thought chain, the thought chain including a plurality of nodes having a hierarchical relationship, and the instruction prompt ranges of the plurality of nodes decreasing level by level;

[0092] Chain of Thought (CoT) aims to improve the ability of large language models to perform complex reasoning by breaking down a problem into multiple steps, reasoning step by step and finally giving an answer, rather than providing an answer directly.

[0093] In this embodiment, the medical imaging report can be structured based on one chain of thought or two chains of thought.

[0094] Exemplarily, the thought chain can be: positive / abnormal extraction → positive / abnormal + anatomical position extraction → positive / abnormal + anatomical position extraction + auxiliary information, including three nodes: positive / abnormal extraction node, positive / abnormal + anatomical position extraction node, positive / abnormal + anatomical position extraction + auxiliary information node.

[0095] Exemplarily, the thinking chain can also be: anatomical position extraction → anatomical position + positive / abnormal extraction → anatomical position + positive / abnormal extraction + auxiliary information, including three nodes: anatomical position extraction node, anatomical position + positive / abnormal extraction node, anatomical position + positive / abnormal extraction + auxiliary information node.

[0096] Multiple nodes in the thinking chain have a hierarchical relationship.

[0097] The instruction hint of each node is used to define the content of the structured output of the corresponding node, for example:

[0098] The command prompt of the normal / abnormal extraction node is "schema: normal / abnormal", which is used to define the content of the structured output of this node as a normal / abnormal label list;

[0099] The command prompt of the Normal / Abnormal Extraction → Anatomical Location node is “schema: <normal / abnormal, anatomical location>”, which is used to define the structured output of this node as a structured dictionary with normal / abnormal as the key and anatomical location as the value;

[0100] The command prompt of the normal / abnormal + anatomical location extraction + auxiliary information node is "schema: <abnormal, anatomical location, auxiliary information 1, auxiliary information 2, ...>", which is used to define the content of the structured output of this node as a two-level nested dictionary structure, such as "{abnormal 1: {anatomical location 1: [auxiliary information 1, auxiliary information 2, ...]}, abnormal 2: {anatomical location 2: [...]}, ...}".

[0101] Among them, a medical imaging report can extract at least one abnormality, each abnormality corresponds to an anatomical location, and each anatomical location corresponds to auxiliary information.

[0102] The instruction prompt range of multiple nodes in the thinking chain decreases step by step, which is a progressive extraction logic.

[0103] It should also be noted that the instruction prompts of the thinking chain are generated according to the instruction prompt template. The instruction prompt template includes system prompts and strategy prompts. Among them, the system prompts are common to all tasks, and the strategy prompts are used to define the content of the structured output of the corresponding node. The strategy prompts for each node are different.

[0104] 403: For at least one thought chain, input the medical imaging report text, the structured output of the previous node, and the instruction prompt of the current node into the pre-trained large language model to obtain the structured output of the current node;

[0105] It should be noted that if the current node is the first node in the thinking chain, the structured output of the previous node is empty, and the instruction prompts of the current node include system prompts and strategy prompts. The system prompts include role prompts, task prompts, and structured output formats. For example, "You are now a professional radiologist. Please extract key information from the report in accordance with the following schema definition." Among them, "radiologist" is the role prompt, "extract key information from the report" is the task prompt, and "schema definition method" is the structured output format. The instruction prompts of other nodes in the thinking chain except the first node only include strategy prompts.

[0106] Taking the thinking chain of anatomical position extraction → anatomical position + positive / abnormal extraction → anatomical position + positive / abnormal extraction + auxiliary information as an example, the strategy prompts for each node are as follows:

[0107] Positive / abnormal extraction (instruction 1-1): extract the label corresponding to the normal or abnormal description subject from the report. The policy prompt template adopts the form including but not limited to "schema: normal / abnormal", and the output form is a normal / abnormal label list;

[0108] Normal / Abnormal + Anatomical Location Extraction (Instructions 1-2): Extract the labels corresponding to the normal or abnormal description subject and their corresponding anatomical location labels from the report. The strategy prompt template adopts the form including but not limited to "schema: <normal / abnormal, anatomical location>", and the output form is a structured dictionary with normal / abnormal as the key and anatomical location as the value;

[0109] Normal / abnormal + anatomical location + ancillary information extraction (instructions 1-3): Extract the normal / abnormal description main label, the corresponding anatomical location label and the ancillary information description label from the report, where the ancillary information includes the uncertainty, severity, time progression, relative significance, quantitative description and other information of the normal / abnormal occurring at a certain location. The strategy prompt template adopts the form including but not limited to "schema: <abnormal, anatomical location, ancillary information 1, ancillary information 2, ...>", and the output form is a two-level nested dictionary structure, such as "{abnormal 1: {anatomical location 1: [ancillary information 1, ancillary information 2, ...]}, abnormal 2: {anatomical location 2: [...]}, ...}".

[0110] Taking the thinking chain of anatomical position extraction → anatomical position + positive / abnormal extraction → anatomical position + positive / abnormal extraction + auxiliary information as an example, the strategy prompts for each node are as follows:

[0111] Anatomical location extraction (instruction 2-1): extract the label corresponding to the anatomical location description subject from the report. The strategy prompt template adopts the form including but not limited to "schema: anatomical location", and the output form is the label list of the anatomical location;

[0112] Anatomical location + normal / abnormal extraction (instruction 2-2): Extract the labels corresponding to the anatomical location description body and their corresponding normal / abnormal labels from the report. The strategy prompt template adopts the form including but not limited to "schema: <anatomical location, normal / abnormal>", and the output form is a structured dictionary with anatomical location as key and normal / abnormal as value;

[0113] Anatomical location + normal / abnormal + auxiliary information extraction (instructions 2-3): Extract the anatomical location description main label, the corresponding normal / abnormal label and the auxiliary information description label from the report. The strategy prompt template adopts the form including but not limited to "schema: <anatomical location, normal / abnormal, auxiliary information 1, auxiliary information 2, ...>", and the output form is a two-level nested dictionary structure, such as "{anatomical location 1: {abnormal 1: [auxiliary information 1, auxiliary information 2, ...]}, anatomical location 2: {abnormal 2: [...]}, ...}".

[0114] For each thinking chain, the structured output of the previous node is used as the input of the current node. At the same time, the input accepted also includes the medical imaging report text and the instruction prompt of the current node to obtain the structured output of the current node.

[0115] The large language model can be any open source generative large language model that supports instruction fine-tuning and multi-round dialogue. The large language model used in all thought chain steps is the same. The large language model is pre-trained using medical imaging report text samples with annotated structured data. The training process will be described in detail later.

[0116] 404: Determine structured data of the medical imaging report text according to the structured output of the last node in at least one thought chain.

[0117] If at least one thought chain includes a thought chain, the structured output of the last node in the thought chain is converted into a triple list of <normal / abnormal, anatomical location, auxiliary information>, and the triple list is determined as the structured data of the medical imaging report text.

[0118] If at least one thinking chain includes the first thinking chain and the second thinking chain, the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain are respectively converted into a triple list of <normal / abnormal, anatomical position, and auxiliary information>, and then the two triple lists are merged to obtain the final structured data. If the maximum accuracy of information extraction needs to be guaranteed, the merging method is to take the intersection of the two lists; if the maximum recall rate of information extraction needs to be guaranteed, the merging method is to take the union of the two lists.

[0119] The present embodiment discloses a method for structuring medical imaging reports based on a thinking chain. Since the thinking chain includes multiple nodes with a hierarchical relationship, and the instruction prompt ranges of the multiple nodes decrease step by step, for at least one thinking chain, the medical imaging report text to be processed, the structured output of the previous node, and the instruction prompt of the current node are input into a pre-trained large language model, so that the large language model extracts accurate, standardized, and complete structured data from the medical imaging report text in sequence according to the hierarchical logic of the thinking chain, thereby avoiding errors and omissions in the data extraction process, ensuring the accuracy, standardization, and completeness of the extracted structured data, and the entire process is completed automatically without manual participation, thereby improving the efficiency of structuring medical imaging reports.

[0120] In a possible implementation, the medical image report is structured based on a thought chain, that is, the medical image report is structured based on a single-path thought chain. Taking the thought chain of anatomical position extraction → anatomical position + positive / abnormal extraction → anatomical position + positive / abnormal extraction + auxiliary information as an example, the training process of the large model is as follows: Figure 5 As shown, of course, the thought chain can also be anatomical position extraction→anatomical position+positive / abnormal extraction→anatomical position+positive / abnormal extraction+supplementary information, which is not specifically limited in this embodiment.

[0121] The medical imaging report text samples used to train the large language model have been pre-annotated with structured data. The thinking chain annotation parser accepts the complete and precisely annotated structured data as input, extracts different levels of structured information suitable for the target thinking chain task as label output, that is, determines the structured output of the medical imaging report text sample at each node in the thinking chain, and uses it as the output label of each node. Among them, the structured data is the complete structured information manually annotated by medical experts based on the medical imaging report text samples, which should include normal / abnormal, anatomical location and auxiliary information including uncertainty, severity, time progression, relative significance, quantitative description, etc. in the report. The annotation form is a two-level nested dictionary structure, such as "{abnormal 1: {anatomical location 1: [auxiliary information 1, auxiliary information 2, ...]}, abnormal 2: {anatomical location 2: [...]}, ...}".

[0122] In the process of training the large language model using medical imaging report text samples, the medical imaging report text samples, the structured output of the previous node and the instruction prompt of the current node are input into the large language model to obtain the structured output of the current node, and the autoregressive loss of the structured output of the current node and the output label is calculated. The loss function is the token classification cross entropy loss, which constrains the model to predict the token at the current position (i.e., the structured output of the current node) based on the report, instructions, historical dialogues and the output sequence before the current token position (i.e., the structured output of the previous node).

[0123] When the autoregressive loss has not converged, the parameters of the large language model are updated according to stochastic gradient descent, and the large language model is continued to be trained using medical imaging report text samples until the autoregressive loss converges, and the training of the large language model is stopped.

[0124] After the model training is completed, Figure 6 As shown, according to the instruction prompts, the structured data of each level are extracted in the order of the thinking chain, and the structured output of the last node of the thinking chain is converted into a triple list of <normal / abnormal, anatomical location, auxiliary information>, and the triple list is determined as the structured data of the medical imaging report text.

[0125] This embodiment uses the instruction fine-tuning of the pre-trained large language model to utilize the powerful text processing capabilities of the large language model to enable it to quickly and efficiently adapt to the task of structuring image report information. The use of instruction fine-tuning training based on the thinking chain can enhance the large language model's ability to understand different levels of information in the report, and perform structured extraction in sequence according to the information hierarchy logic, which can avoid errors and omissions in the information extraction process.

[0126] In a possible implementation, the medical imaging report is structured based on two thinking chains, that is, the medical imaging report is structured based on a dual-path thinking chain. The training process of the large model is as follows: Figure 7 As shown, a thinking chain annotation parser is also needed, which accepts complete and precisely annotated structured data as input and extracts different levels of structured information suitable for different thinking chain tasks as label output, that is, determines the structured output of the medical imaging report text sample at each node in the first thinking chain, which serves as the first output label of each node in the first thinking chain, and determines the structured output of the medical imaging report text sample at each node in the second thinking chain, which serves as the second output label of each node in the second thinking chain.

[0127] In the process of training the large language model using medical imaging report text samples, the medical imaging report text samples, the structured output of the previous node in the first thinking chain and the instruction prompt of the current node are input into the large language model to obtain the structured output of the current node of the first thinking chain, and the first autoregressive loss of the structured output of the current node of the first thinking chain and the first output label is calculated.

[0128] The medical imaging report text sample, the structured output of the previous node in the second thinking chain, and the instruction prompt of the current node are input into the large language model to obtain the structured output of the current node of the second thinking chain, and the second autoregressive loss of the structured output of the current node of the second thinking chain and the second output label is calculated.

[0129] Different from the single-path thinking chain, it is also necessary to calculate the similarity between the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain to obtain the consistency loss. Specifically, the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain are uniformly converted into a triple list of <normal / abnormal, anatomical position, and auxiliary information>, and then the similarity of the two triple lists is calculated. The specific calculation form of the similarity includes but is not limited to Jaccard similarity, cosine similarity, etc.

[0130] The first autoregressive loss, the second autoregressive loss, and the consistency loss are weighted and summed to obtain the total loss, where the weights are manually selected hyperparameters. When the total loss does not converge, the large language model parameters are updated according to the stochastic gradient descent, and the large language model is continued to be trained using the medical imaging report text samples until the total loss converges, and the training of the large language model is stopped.

[0131] After the model training is completed, Figure 8As shown, according to the instruction prompts, the structured data of each level are extracted in sequence according to the two thinking chains respectively, and the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain are uniformly converted into a triple list of <normal / abnormal, anatomical position, auxiliary information>, and then the two triple lists are merged to obtain the final structured data.

[0132] This embodiment is based on multi-round conversational instruction fine-tuning, which decomposes the complete imaging report structured task into two structured thinking chains. The training model extracts structured information of different levels in the imaging report step by step, extracts information by merging dual-path thinking chains, and imposes consistency constraints during training, which can further improve the accuracy and completeness of information extraction.

[0133] In a possible implementation, the structured large model training of thought chain image report can appropriately remove some nodes in the thought chain, that is, reduce the steps of structured extraction.

[0134] Taking the removal of the intermediate nodes of positive / abnormal + anatomical location extraction and anatomical location + positive / abnormal extraction as an example, the training process of the large language model is as follows Fig. 9 As shown in the figure, the first chain of thought becomes positive / abnormal extraction → positive / abnormal + anatomical location extraction + auxiliary information, and the second chain of thought becomes anatomical location extraction → anatomical location + positive / abnormal extraction + auxiliary information extraction. Figure 7 The training process shown is similar and will not be repeated here.

[0135] After the model training is completed, Fig.10 As shown, according to the instruction prompts, the structured data of each level are extracted in sequence according to the two thinking chains respectively, and the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain are respectively converted into a triple list of <normal / abnormal, anatomical position, auxiliary information>, and then the two triple lists are merged to obtain the final structured data.

[0136] This embodiment removes some nodes in the thought chain, that is, reduces the steps of structured extraction, thereby improving the training speed of the large language model and the speed of structured processing of medical imaging reports.

[0137] The above introduces a method for structuring medical image reports based on thought chains provided in an embodiment of the present application. The following will introduce a device for executing the above method for structuring medical image reports based on thought chains.

[0138] See also Fig.11 , Fig.11 The schematic diagram of the structure of a medical image report structured processing device based on thought chain provided in the embodiment of the present application. Fig.11 As shown, the medical image report structured processing device based on thought chain includes:

[0139] A report text acquisition unit 111 is used to acquire the medical image report text to be processed;

[0140] A thought chain determination unit 112 is used to determine at least one thought chain, wherein the thought chain includes a plurality of nodes having a hierarchical relationship, and the instruction prompt ranges of the plurality of nodes decrease gradually;

[0141] The model processing unit 113 is used for inputting the medical imaging report text, the structured output of the previous node and the instruction prompt of the current node into a pre-trained large language model for the at least one thought chain to obtain the structured output of the current node, wherein the large language model is pre-trained using a medical imaging report text sample with annotated structured data;

[0142] The structured data determining unit 114 is used to determine the structured data of the medical imaging report text according to the structured output of the last node in the at least one thought chain.

[0143] In a possible implementation, if the at least one chain of thought includes a chain of thought, the medical image report structured processing device based on the chain of thought further includes:

[0144] The large language model training unit is used to determine the structured output of each node of the medical imaging report text sample in the thought chain according to the structured data that has been annotated by the medical imaging report text sample, and use it as the output label of each node respectively; input the medical imaging report text sample, the structured output of the previous node and the instruction prompt of the current node into the large language model to obtain the structured output of the current node; calculate the autoregressive loss of the structured output and the output label of the current node; when the autoregressive loss converges, stop training the large language model.

[0145] In a possible implementation, if the at least one thought chain includes a thought chain, the structured data determination unit 114 is specifically used to convert the structured output of the last node in the thought chain into a triple list; and determine the triple list as the structured data of the medical imaging report text.

[0146] In a possible implementation, if the at least one chain of thought includes a first chain of thought and a second chain of thought, the medical image report structured processing device based on the chain of thought further includes:

[0147] A large language model training unit is used to determine the structured output of the medical imaging report text sample at each node in the first thinking chain according to the structured data of the medical imaging report text sample, and use it as the first output label of each node in the first thinking chain, and determine the structured output of the medical imaging report text sample at each node in the second thinking chain, and use it as the second output label of each node in the second thinking chain; input the medical imaging report text sample, the structured output of the previous node in the first thinking chain, and the instruction prompt of the current node into the large language model to obtain the structured output of the current node of the first thinking chain; calculate the structured output of the current node of the first thinking chain The first autoregressive loss with the first output label; the medical imaging report text sample, the structured output of the previous node in the second thinking chain and the instruction prompt of the current node are input into the large language model to obtain the structured output of the current node of the second thinking chain; the second autoregressive loss of the structured output of the current node of the second thinking chain and the second output label is calculated; the similarity between the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain is calculated to obtain the consistency loss; the first autoregressive loss, the second autoregressive loss and the consistency loss are weightedly summed to obtain the total loss; when the total loss converges, the training of the large language model is stopped.

[0148] In a possible implementation, if the at least one thinking chain includes a first thinking chain and a second thinking chain, the structured data determination unit 114 is specifically used to convert the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain into triple lists respectively; and determine the intersection or union of the two triple lists as the structured data of the medical imaging report text.

[0149] In a possible implementation, if the current node is the first node in the thinking chain, the instruction prompt of the current node includes system prompts and strategy prompts, the system prompts include role prompts, task prompts and structured output formats, the structured output of the previous node is empty, and the instruction prompts of other nodes in the thinking chain except the first node only include the strategy prompts, and the strategy prompts are used to define the content of the structured output of the corresponding node.

[0150] The present embodiment discloses a medical imaging report structured processing device based on a thinking chain. Since the thinking chain includes multiple nodes with a hierarchical relationship, and the instruction prompt ranges of the multiple nodes decrease step by step, for at least one thinking chain, the medical imaging report text to be processed, the structured output of the previous node and the instruction prompt of the current node are input into a pre-trained large language model, so that the large language model extracts accurate, standardized and complete structured data of the medical imaging report text in sequence according to the hierarchical logic of the thinking chain, thereby avoiding errors and omissions in the data extraction process, ensuring the accuracy, standardization and completeness of the extracted structured data, and the whole process is completed automatically without human participation, thereby improving the efficiency of structuring processing of medical imaging reports.

[0151] The present application also provides an electronic device in an embodiment. Fig.12 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.12 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] like Fig.12 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 121, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 122 or a program loaded from a storage device 128 to a random access memory (RAM) 123. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 123. The processing device 121, the ROM 122, and the RAM 123 are connected to each other via a bus 124. An input / output (I / O) interface 125 is also connected to the bus 124.

[0153] Typically, the following devices may be connected to the I / O interface 125: an input device 126 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 127 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 128 including, for example, a memory card, a hard disk, etc.; and a communication device 129. The communication device 129 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0154] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the thought chain-based medical imaging report structured processing methods provided in the embodiments of the present application.

[0155] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any one of the thought chain-based medical imaging report structured processing methods provided in the embodiment of the present application.

[0156] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0157] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0158] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0159] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for structured processing of medical imaging reports based on thought chain, characterized in that: include: Obtain the text of the medical imaging report to be processed; Determine at least one thought chain, wherein the thought chain includes a plurality of nodes having a hierarchical relationship, and the instruction prompt ranges of the plurality of nodes decrease gradually; For the at least one thought chain, the medical imaging report text, the structured output of the previous node and the instruction prompt of the current node are input into a pre-trained large language model to obtain the structured output of the current node, wherein the large language model is pre-trained using a medical imaging report text sample with annotated structured data; According to the structured output of the last node in the at least one thought chain, the structured data of the medical imaging report text is determined.

2. The method for structuring medical image reports based on thought chain according to claim 1 is characterized in that: If the at least one thought chain includes a thought chain, the method for training the large language model includes: According to the annotated structured data of the medical imaging report text sample, determine the structured output of each node of the medical imaging report text sample in the thought chain as the output label of each node; Inputting the medical imaging report text sample, the structured output of the previous node and the instruction prompt of the current node into the large language model to obtain the structured output of the current node; Calculate the autoregressive loss of the structured output and output label of the current node; When the autoregressive loss converges, the training of the large language model is stopped.

3. The method for structuring medical image reports based on thought chain according to claim 1 is characterized in that: If the at least one thought chain includes a thought chain, determining the structured data of the medical imaging report text according to the structured output of the last node in the at least one thought chain includes: Convert the structured output of the last node in the thought chain into a list of triples; The triple list is determined as structured data of the medical imaging report text.

4. The method for structuring medical image reports based on thought chain according to claim 1 is characterized in that: If the at least one chain of thought includes a first chain of thought and a second chain of thought, the method for training the large language model includes: According to the structured data of the medical imaging report text sample, determine the structured output of the medical imaging report text sample at each node in the first thinking chain as the first output label of each node in the first thinking chain, and determine the structured output of the medical imaging report text sample at each node in the second thinking chain as the second output label of each node in the second thinking chain; Inputting the medical imaging report text sample, the structured output of the previous node in the first thought chain, and the instruction prompt of the current node into the large language model to obtain the structured output of the current node of the first thought chain; Calculate the first autoregressive loss of the structured output of the current node of the first thought chain and the first output label; Inputting the medical imaging report text sample, the structured output of the previous node in the second thought chain, and the instruction prompt of the current node into the large language model to obtain the structured output of the current node of the second thought chain; Calculate the second autoregressive loss of the structured output of the current node of the second thinking chain and the second output label; Calculate the similarity between the structured output of the last node in the first thought chain and the structured output of the last node in the second thought chain to obtain the consistency loss; Performing a weighted summation on the first autoregressive loss, the second autoregressive loss, and the consistency loss to obtain a total loss; When the total loss converges, the training of the large language model is stopped.

5. The method for structuring medical image reports based on thought chain according to claim 1 is characterized in that: If the at least one chain of thought includes a first chain of thought and a second chain of thought, determining the structured data of the medical imaging report text according to the structured output of the last node in the at least one chain of thought includes: Respectively converting the structured output of the last node in the first thinking chain and the structured output of the last node in the second thinking chain into a triple list; The intersection or union of the two triple lists is determined as the structured data of the medical imaging report text.

6. The method for structuring medical image reports based on thought chain according to any one of claims 1 to 5, characterized in that: If the current node is the first node in the thinking chain, the instruction prompt of the current node includes system prompts and strategy prompts, the system prompts include role prompts, task prompts and structured output formats, the structured output of the previous node is empty, and the instruction prompts of other nodes in the thinking chain except the first node only include the strategy prompts, and the strategy prompts are used to define the content of the structured output of the corresponding node.

7. A medical image report structured processing device based on thought chain, characterized in that: include: A report text acquisition unit, used to acquire the medical imaging report text to be processed; A thought chain determination unit, used to determine at least one thought chain, wherein the thought chain includes a plurality of nodes having a hierarchical relationship, and the instruction prompt ranges of the plurality of nodes decrease gradually; A model processing unit is used for inputting the medical imaging report text, the structured output of the previous node and the instruction prompt of the current node into a pre-trained large language model for the at least one thought chain to obtain the structured output of the current node, wherein the large language model is pre-trained using medical imaging report text samples with annotated structured data; The structured data determination unit is used to determine the structured data of the medical imaging report text according to the structured output of the last node in the at least one thought chain.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the medical imaging report structured processing method based on thought chain as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the medical imaging report structured processing method based on thought chain as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the medical imaging report structured processing method based on thought chain as described in any one of claims 1 to 6.

Images