Medical knowledge enhanced chest radiograph automated diagnosis system

Abstract

The present application relates to the technical field of intelligent systems, and proposes a medical knowledge enhanced chest radiograph automatic diagnosis system. The system comprises a first module configured to provide a knowledge enhanced text encoder φ text , which is constructed according to a medical knowledge base, wherein the text encoder φ text is configured to perform first stage pre-training; a second module configured to provide an image encoder φ image , which is configured to perform contrastive learning on the text encoder φ text after first stage pre-training; and a third module configured to provide a disease query network φ DQN , which is configured to perform disease prediction, wherein the image encoder φ image , the text encoder φ text after first stage pre-training, and the disease query network φ DQN are configured to perform second stage pre-training. The present application models medical terms through contrastive learning technology based on a professional medical knowledge base, and further realizes a knowledge enhanced chest radiograph and diagnosis report based automatic diagnosis system, which has achieved significant effect improvement in various medical scenarios.

Description

Technical Field

[0001] This invention generally relates to the field of intelligent systems technology. Specifically, this invention relates to an automated chest X-ray diagnostic system with enhanced medical knowledge. Background Technology

[0002] In recent years, foundation models have achieved tremendous success in computer vision and natural language understanding, demonstrating their powerful feature transfer and generalization capabilities. However, in the application of foundation models in the medical field, although there is some research based on image-text pre-trained models, the overall development still lags behind. This is because understanding and modeling complex professional concepts in clinical reports, as well as fine-grained long-tail recognition problems, requires models to understand clinical expertise, a point that existing models often fail to consider.

[0003] Specifically, in existing technologies for automated chest X-ray diagnostic systems, Ekin Tiu et al. proposed a self-supervised pre-training model based on image-text contrastive learning (Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning). During the pre-training phase, this model directly performs comparative learning using chest X-rays and diagnostic reports, enabling zero-shot multi-label classification in downstream tasks. While this paper achieved preliminary automated chest X-ray diagnosis through image-text pre-training, it still has significant room for improvement because it does not consider the professional knowledge required for basic medical models.

[0004] The applicant's research revealed that existing automated chest X-ray diagnostic systems suffer from the following problems: First, they lack integration with professional medical knowledge. Understanding complex medical terminology in imaging reports and diagnosing diseases require highly specialized knowledge, and current pre-trained models do not consider incorporating expert knowledge into representation learning. Second, their application is limited. Current methods are trained only on images and diagnostic reports, limiting their application to diseases mentioned in the reports and preventing generalization to other fine-grained disease diagnostic tasks. Summary of the Invention

[0005] To at least partially address the aforementioned problems in the prior art, this invention proposes a medical knowledge-enhanced automated chest X-ray diagnostic system, comprising:

[0006] The first module is configured to provide a knowledge-enhanced text encoder φ text It is constructed based on a medical knowledge base, wherein the text encoder φ text It was configured to perform the first phase of pre-training;

[0007] The second module is configured to provide an image encoder φ image It is configured as the text encoder φ after the first stage of pre-training. text To conduct comparative learning; and

[0008] The third module is configured to provide a disease query network φ DQN It is configured to perform disease prediction, wherein the image encoder φ image The text encoder φ after the first stage of pre-training text and disease query network φ DQN It was configured to perform the second phase of pre-training.

[0009] In one embodiment of the present invention, the medical knowledge-enhanced automated chest X-ray diagnostic system is configured to perform the following actions:

[0010] The chest X-ray image to be diagnosed is input into the image encoder φ, which has been pre-trained in the second stage. image To generate feature codes for the image to be diagnosed;

[0011] Input the name of the disease to be diagnosed into the pre-trained text encoder φ from the second stage. text The code for the name of the disease to be diagnosed is generated in the Chinese language; and

[0012] The feature encoding of the image to be diagnosed and the name encoding of the disease to be diagnosed are input into the disease query network φ after the second stage of pre-training. DQN The technology generates disease prediction and lesion localization.

[0013] In one embodiment of the present invention, a knowledge-enhanced text encoder φ is constructed based on a medical knowledge base. text include:

[0014] Preprocessing of the medical knowledge base includes extracting medical entities from the medical knowledge base, wherein the medical entities are represented by entity names, entity definitions, and relationships between entities;

[0015] Positive and negative sample pairs are constructed based on the expression methods of the medical entities, wherein positive sample pairs are formed based on different expression methods of the same medical entity, and negative sample pairs are formed based on different expression methods of different medical entities; and

[0016] The positive and negative sample pairs are encoded by a text encoder and trained using contrastive learning, wherein a knowledge-enhanced text encoder φ is generated through supervision using contrastive loss. text .

[0017] In one embodiment of the present invention, it is specified that the knowledge-enhanced text encoder φ text The first phase of pre-training includes:

[0018] The text encoder φ for knowledge enhancement is based on the contrastive loss combined with a deep learning gradient descent algorithm. text Conduct the first phase of pre-training.

[0019] In one embodiment of the present invention, it is specified that the image encoder φ image The text encoder φ after the first stage of pre-training text Comparative learning was conducted, and the disease query network φ was used. DQN Disease prediction includes the following steps:

[0020] The report text is transformed into a sequence of medical entities, which includes the medical entities and their classifications. The classifications of medical entities include anatomical location, observation presence, observation absence, and observation uncertainty. Here, the term "anatomical location" refers to an entity of an organ or location (e.g., "lung"), and the term "observation" refers to a disease or symptom (e.g., "pneumonia").

[0021] The observation and classification of statistical medical entities form the query disease set Q;

[0022] With x i Representing an image, using t i Represents the image x i The corresponding sequence of report text;

[0023] Image encoder φ image For image x i Encode it as follows: x i =φ image (x i ), where x i Represents image coding features;

[0024] The text encoder φ, pre-trained in the first stage text For the sequence t of the report text i Encode it as follows: t i =φ text (t i ), where t i Represents text encoding features;

[0025] The text encoder φ, pre-trained in the first stage text The disease set Q is encoded as follows: Q = φ text (Q), where Q represents the coded feature of the disease set being queried;

[0026] The image encoding feature x i Or the text encoding feature t iAnd the disease set encoding feature Q input to the disease query network φ DQN In, and by the disease query network φ DQN The output predicts the diseases in the query disease set Q, expressed as: s i =φ DQN (x i , t i ,Q), where s i This indicates the disease prediction output;

[0027] Encoding features x in an image i and text encoding features t i Perform comparative learning, and use comparative loss L contrastive Supervision was conducted, where positive sample pairs included image-text pairs from the same case, and negative sample pairs included image-text pairs from different cases;

[0028] L is obtained through binary cross-entropy loss BCE Disease prediction outputs s i To supervise; and

[0029] The contrast loss L contrastive and the binary cross-entropy loss L BCE The total monitoring loss L is generated by weighted averaging, expressed as: L = α1L contrastive +α2L BCE , where α1 and α2 represent hyperparameters.

[0030] In one embodiment of the present invention, it is specified that the image encoder φ image The text encoder φ after the first stage of pre-training text and disease query network φ DQN The second phase of pre-training includes:

[0031] Based on the total supervision loss L and the deep learning gradient descent algorithm, the text encoder φ is... text The image encoder φ image and the disease query network φ DQN The second phase of pre-training will be conducted.

[0032] In one embodiment of the present invention, the disease query network φ is specified as follows: DQN Based on the transformer structure, the disease query network φ DQN It includes multiple mutual attention interaction layers, wherein an attention map is generated by averaging the mutual attention weights of the multiple mutual attention interaction layers, and the attention map represents the confidence level regarding where the lesion is located in the image.

[0033] In one embodiment of the present invention, it is specified that, according to the image encoder φimage and the text encoder φ text One of the steps involves initializing the downstream network, training the neural network on the downstream dataset, and then testing its performance.

[0034] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, the computer program running the system when executed by a processor.

[0035] The present invention also proposes a computer system, comprising:

[0036] A processor, configured to execute machine-executable instructions; and

[0037] A memory that stores machine-executable instructions, which, when executed by a processor, perform the following actions:

[0038] Provides a knowledge-enhanced text encoder φ text It is constructed based on a medical knowledge base, wherein the text encoder φ text It was configured to perform the first phase of pre-training;

[0039] Provides image encoder φ image It is configured as the text encoder φ after the first stage of pre-training. text To conduct comparative learning; and

[0040] Provides a disease query network φ DQN It is configured to perform disease prediction, wherein the image encoder φ image The text encoder φ after the first stage of pre-training text and disease query network φ DQN It was configured to perform the second phase of pre-training.

[0041] This invention offers at least the following advantages: Based on a professional medical knowledge base, it models medical terminology using contrastive learning technology, further realizing a knowledge-enhanced automatic diagnostic system based on chest X-rays and diagnostic reports, achieving significant performance improvements in various medical scenarios. This invention achieves superior results in the diagnosis of multiple diseases through medical knowledge-enhanced pre-training technology, maintaining excellent pre-training performance even in scenarios with scarce medical data. Furthermore, due to the integration of medical knowledge, this invention can generalize to any disease diagnosis task by simply inputting the complete disease name. Moreover, compared to existing technologies, this invention can provide a network-predicted attention map for lesion localization, offering superior interpretability. Attached Figure Description

[0042] To further illustrate the advantages and other features of the various embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It is understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0043] Figure 1 A schematic diagram of a computer system implementing the system according to the present invention is shown.

[0044] Figure 2 A schematic diagram of a medical knowledge-enhanced automated chest X-ray diagnostic system is shown in one embodiment of the present invention.

[0045] Figure 3 The diagram illustrates a flowchart of a knowledge-enhanced text encoder pre-training process according to one embodiment of the present invention.

[0046] Figure 4 The diagram illustrates a process flow for image-text pre-training in one embodiment of the present invention. Detailed Implementation

[0047] It should be noted that the components in the various figures may be shown exaggeratedly for illustrative purposes and are not necessarily to scale. In each figure, the same reference numerals are used for components that are identical or have the same function.

[0048] In this invention, unless otherwise specified, "arranged on," "arranged above," and "arranged on" do not exclude the possibility of an intermediate element between them. Furthermore, "arranged on or above" merely indicates the relative positional relationship between two components, and in certain cases, such as when the product orientation is reversed, it can also be converted to "arranged below or under," and vice versa.

[0049] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0050] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0051] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0052] It should also be noted that, within the scope of this invention, the terms "same," "equal," and "equal to" do not imply that the two values ​​are absolutely equal, but rather allow for a certain reasonable margin of error. In other words, the terms also encompass "substantially the same," "substantially equal," and "substantially equal to." Similarly, in this invention, the directional terms "perpendicular to," "parallel to," etc., also encompass the meanings of "substantially perpendicular to" and "substantially parallel to."

[0053] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 A computer system 100 implementing the system according to the invention is shown. Unless otherwise specified, the method and / or system according to the invention can be used in… Figure 1 The invention may be implemented in the computer system 100 shown to achieve the objectives of the invention, or the invention may be implemented in a distributed manner in multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0056] like Figure 1 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a Universal Serial Bus (USB) interface 112. The system bus 101 can be any of several bus architecture types, such as a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. The system bus 101 is used for communication between the various bus devices. In addition to... Figure 1Other bus devices or interfaces besides those shown are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 may, for example, store basic input / output system (BIOS) data for basic routines used to implement information transfer at startup, while RAM 104 provides the system with faster access speed running memory. Computer system 100 also includes hard disk drive 109 for reading and writing to hard disk 110, optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 may, for example, store operating system and application programs. Drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 100. Computer system 100 may also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may also include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. Furthermore, the computer system 100 may also include a network interface 114 for network connectivity, wherein the network interface 114 can be connected to the Internet 116 via a network device such as a router 115, wherein the connection can be wired or wireless. Additionally, the computer system 100 may also include a Universal Serial Bus (USB) interface 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as microphones, cameras, etc.

[0057] When the present invention is Figure 1When implemented on the computer system 100, the medical knowledge-enhanced pre-training technique achieves superior results in the diagnosis of multiple diseases, even in scenarios where medical data is scarce. Furthermore, due to the integration of medical knowledge, only the complete disease name needs to be input, allowing the invention to be generalized to any disease diagnosis task. In addition, compared to existing technologies, the invention can provide a network-predicted attention map for lesion localization, offering superior interpretability. Moreover, the embodiments can be provided as computer program products comprising one or more machine-readable media on which machine-executable instructions are stored. When executed by one or more machines, such as a computer, computer network, or other electronic device, these instructions can cause one or more machines to perform operations according to the embodiments of the invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0058] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0059] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by the corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by the corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by the corresponding output device (such as a display, speaker, etc.), and so on.

[0060] This invention proposes a medical knowledge-enhanced automated chest X-ray diagnostic system, characterized by comprising:

[0061] Knowledge-enhanced text encoder φ text It is constructed based on a medical knowledge base, wherein the text encoder φ text It was configured to perform the first phase of pre-training;

[0062] Image encoder φ image It is configured as the text encoder φ after the first stage of pre-training. text To conduct comparative learning; and

[0063] Disease query network φ DQN It is configured to perform disease prediction;

[0064] Among them, for the image encoder φ image The text encoder φ after the first stage of pre-training text and disease query network φ DQN It was configured to perform the second phase of pre-training;

[0065] Figure 2A schematic diagram of a medical knowledge-enhanced automated chest X-ray diagnostic system is shown in one embodiment of the present invention. Figure 2 As shown, the system includes:

[0066] Module 201 is configured to provide a knowledge-enhanced text encoder φ text It is constructed based on a medical knowledge base, wherein the text encoder φ text It was configured to perform the first phase of pre-training;

[0067] The second module 202 is configured to provide an image encoder φ image It is configured as the text encoder φ after the first stage of pre-training. text To conduct comparative learning; and

[0068] Module 203 is configured to provide a disease query network φ DQN It is configured to perform disease prediction, wherein the image encoder φ image The text encoder φ after the first stage of pre-training text and disease query network φ DQN It was configured to perform the second phase of pre-training.

[0069] This invention includes constructing positive and negative sample pairs based on a knowledge graph, training a text encoder through contrastive learning, performing representation learning on an existing medical knowledge base, and implicitly modeling the relationships between medical entities in the text representation space; knowledge-enhanced graph-text representation learning, utilizing the knowledge-enhanced text encoder pre-trained in the first step, and performing contrastive learning based on images and standardized image reports; and designing a disease query network using Transformer modal interaction, which can obtain a diagnostic result prediction by inputting any disease name, and can locate lesions through attention maps, thereby enhancing the interpretability of the system.

[0070] The present invention will now be described in detail with reference to specific embodiments.

[0071] According to one aspect of the present invention, a knowledge-enhanced automatic diagnosis method for chest X-rays is provided, the method comprising two stages: a first stage of knowledge-enhanced text encoder pre-training and a second stage of image-text pre-training.

[0072] Figure 3 This diagram illustrates a flowchart of a knowledge-enhanced text encoder pre-training process according to one embodiment of the present invention, as shown below. Figure 3 As shown, the first stage may include:

[0073] The medical knowledge base preprocessing involves extracting medical entities, entity definitions, and relationships between entities. Each entity has three representations: entity name, entity definition, and other entities + corresponding relationships.

[0074] Construct positive and negative sample pairs, where different expressions of the same entity form positive sample pairs, such as (entity A name, entity A definition), (entity A name, other entities + corresponding relationship), and different entities form negative sample pairs, such as (entity A name, entity B name), (entity A name, entity B definition), etc.

[0075] The positive and negative sample pairs constructed based on the medical knowledge base are then processed by the text encoder φ. text The code is encoded, trained using contrastive learning, and supervised by the classic "contrastive loss" method to obtain a knowledge-enhanced text encoder.

[0076] Using the supervised loss described above, combined with any mainstream deep learning gradient descent algorithm, the first stage of text encoder pre-training can be completed.

[0077] Figure 4 This diagram illustrates a flowchart of an image-text pre-training process according to one embodiment of the present invention, as shown below. Figure 4 As shown, the second stage may include:

[0078] The report text is processed using medical text entity extraction methods, such as RadGraph, to transform the data pairs into a sequence of entities. For each entity, RadGraph provides a corresponding classification, including: anatomical location, observation-presence, observation-absence, and observation-uncertainty. Here, anatomical location corresponds to entities representing organs or locations, such as 'lung', and observation corresponds to diseases or symptoms, such as 'pneumonia'.

[0079] After the above processing, each report is transformed into a sequence of entities and categories. By counting all observed entities, we select the |Q| most frequently occurring entities to form the query disease set Q, and simultaneously obtain the existence label of each report based on the category given by "RadGraph".

[0080] Consider a specific text-image pair, using x i To refer to a specific image, use t i Substitute and x i The corresponding text pairs, note that since the entity extraction described above has been performed, the report text has been transformed into a sequence of "entity + category".

[0081] First, the image is encoded using an image encoder, such as the common "Residual Network 50", as shown in the following formula: xi =φ image (x i ), where x i Represents image coding features, φ image Indicates an image encoder;

[0082] Using the text encoder φ pre-trained in the first stage text The sequence of "entity + category" is encoded as follows: t i =φ text (t i )

[0083] Using the text encoder φ pre-trained in the first stage text The disease set being queried is encoded as follows: Q = φ text (Q)

[0084] Input image-encoded features or text-encoded features, along with the query disease set, into a disease query network φ based on a Transformer structure. DQN The disease query network contains four mutual attention interaction layers. Of course, other variations and improvements can be adopted without affecting the main idea of ​​this invention.

[0085] The disease query network outputs predictions for all diseases in the queried disease set. i =φ DQN (x i , t i ,Q);

[0086] In addition, by averaging the mutual attention weights in each mutual attention layer, an attention map can be obtained as a measure of the system's confidence in the location of the lesion in the original image;

[0087] The system includes image-text contrast learning and classification of disease query network predictions during training;

[0088] Specifically, for image-text contrast learning, the classic "contrast loss" is used for supervised learning. contrastive Positive sample pairs are image-text pairs from the same case, and negative sample pairs are image-text pairs from different cases. For s i Supervised L using the classic "Binary Cross Entropy (BCE) loss". BCE ;

[0089] The final loss is obtained by weighted averaging of the two losses mentioned above, i.e., L = α1L contrastive +α2L BCE α1 and α2 are two hyperparameters, and this invention recommends that they both be selected as 1.0, but can be specifically selected by those skilled in the art.

[0090] By utilizing the supervised loss described above, and combining it with any mainstream deep learning gradient descent algorithm, the training of the pre-trained model can be completed.

[0091] The system can be used in two modes: zero-sample mode and fine-tuning mode.

[0092] In zero-shot mode, given an image and a disease name, the image feature code and disease name code are obtained by inputting them into the image encoder and text encoder, respectively. These are then input into the disease query network to obtain the corresponding disease prediction and lesion localization.

[0093] In fine-tuning mode, users can initialize the downstream network with any of the encoders mentioned above as needed, then perform classic neural network training on the downstream dataset, and finally conduct targeted usage tests.

[0094] It should be noted that the image encoder "residual network 50" of the present invention can be replaced by any image encoder structure of the same type, such as the "visual Transformer" structure; the disease query network can arbitrarily increase the number of layers and specific detailed parameters; the medical entity extraction algorithm can be replaced by any extraction method that can achieve the same purpose.

[0095] This invention has achieved state-of-the-art performance in simulated zero-sample diagnostic tasks for multiple diseases, including a fine-grained long-tailed disease diagnostic task for 193 chest diseases and a diagnostic task for 14 chest diseases. In fine-tuning mode, this invention also achieves superior results in these tasks. Furthermore, the performance of this invention surpasses the average results of three clinicians in multiple disease diagnostic tasks.

[0096] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A medical knowledge augmented chest radiograph automated diagnosis system, characterized by, include: a first module configured to provide a knowledge-augmented text encoder configured to be constructed from a medical knowledge base, wherein the text encoder is configured to be first-stage pre-trained; a second module configured to provide an image encoder configured to perform contrastive learning with the first stage pre-trained text encoder configured to perform contrastive learning with the first stage pre-trained text encoder as well as The third module is configured to provide a disease query network. It is configured to perform disease prediction, in which the image encoder The text encoder after the first stage of pre-training and disease query network It was configured to perform the second phase of pre-training; The system is used in a zero-shot mode. In zero-shot mode, given an image and a disease name, the image encoder and text encoder are input respectively to obtain the image feature code and the disease name code. These are then input into a disease query network to obtain the corresponding disease prediction and lesion localization. The system is configured to perform the following actions: The chest X-ray image to be diagnosed is input into the image encoder after the second stage of pre-training. To generate feature codes for the image to be diagnosed; Input the name of the disease to be diagnosed into the pre-trained text encoder of the second stage. The code generates the name of the disease to be diagnosed. as well as The feature codes of the image to be diagnosed and the name codes of the disease to be diagnosed are input into the disease query network after the second stage of pre-training. The technology generates disease prediction and lesion localization. The disease query network mentioned above Based on the transformer structure, the disease query network It includes multiple mutual attention interaction layers, wherein an attention map is generated by averaging the mutual attention weights of the multiple mutual attention interaction layers, and the attention map represents the confidence level regarding where the lesion is located in the image.

2. The medical knowledge-enhanced automatic chest X-ray diagnostic system according to claim 1, characterized in that, Construct a knowledge-enhanced text encoder based on a medical knowledge base. include: Preprocessing of the medical knowledge base includes extracting medical entities from the medical knowledge base, wherein the medical entities are represented by entity names, entity definitions, and relationships between entities; Positive and negative sample pairs are constructed based on the expression methods of the medical entities, wherein positive sample pairs are formed based on different expression methods of the same medical entity, and negative sample pairs are formed based on different expression methods of different medical entities; and The positive and negative sample pairs are encoded by a text encoder and trained using contrastive learning, wherein a knowledge-enhanced text encoder is generated through supervision using contrastive loss. .

3. The medical knowledge-enhanced automatic chest X-ray diagnostic system according to claim 2, characterized in that, Knowledge-enhanced text encoder The first phase of pre-training includes: The text encoder for knowledge enhancement is based on the contrastive loss combined with a deep learning gradient descent algorithm. Conduct the first phase of pre-training.

4. The medical knowledge-enhanced automatic chest X-ray diagnostic system according to claim 1, characterized in that, For image encoders The text encoder after the first stage of pre-training Comparative learning was conducted, and the disease query network was used. Disease prediction includes: The report text is converted into a sequence of medical entities, which includes the medical entities and their classifications. The classifications of medical entities include anatomical location, observation presence, observation absence, and observation uncertainty. The observation and classification of statistical medical entities form a query disease set. ; by Representing an image, in Representation and Image The corresponding sequence of report texts; Image encoder For images Encode it as follows: ,in Represents image coding features; The text encoder pre-trained in the first stage The sequence of the report text Encode it as follows: ,in Represents text encoding features; The text encoder pre-trained in the first stage For the query disease set Encode it as follows: ,in This indicates a query for the encoded features of a disease set; Encode the image features Or the text encoding features and the queried disease set encoding features Enter disease query network In, and by the disease query network Output the set of diseases queried The disease in the formula can be predicted as follows: ,in This indicates the disease prediction output; Encoding features for images and text encoding features Conduct comparative learning and compare losses. Supervision was conducted, where positive sample pairs included image-text pairs from the same case, and negative sample pairs included image-text pairs from different cases; Using binary cross-entropy loss Disease prediction output To supervise; and The contrast loss and the binary cross-entropy loss Perform a weighted average to generate the total monitoring loss. It can be expressed as the following formula: ,in as well as This represents hyperparameters.

5. The medical knowledge-enhanced automatic chest X-ray diagnostic system according to claim 4, characterized in that, For image encoders The text encoder after the first stage of pre-training and disease query network The second phase of pre-training includes: Based on the total supervisory loss The text encoder is combined with deep learning gradient descent algorithm The image encoder and the disease query network Conduct the second phase of pre-training.

6. The medical knowledge-enhanced automatic chest X-ray diagnostic system according to claim 1, characterized in that, According to the image encoder and the text encoder One of the steps involves initializing the downstream network, training the neural network on the downstream dataset, and then testing its performance.

7. A computer-readable storage medium having a computer program stored thereon, the computer program executing, when executed by a processor, the system according to any one of claims 1-6.

8. A computer system, comprising: A processor, which is configured to execute machine-executable instructions; as well as A memory that stores machine-executable instructions, which, when executed by a processor, perform the following actions: Provides knowledge-enhanced text encoder It is constructed based on a medical knowledge base, wherein the text encoder is... It was configured to perform the first phase of pre-training; Provides image encoder It is configured as a text encoder after the first stage of pre-training. Conduct comparative learning; as well as Provide disease query network It is configured to perform disease prediction, where the image encoder... The text encoder after the first stage of pre-training and disease query network Configured for the second phase of pre-training; The system is used in a zero-shot mode. In zero-shot mode, given an image and a disease name, the image encoder and text encoder are input respectively to obtain the image feature code and the disease name code. These are then input into a disease query network to obtain the corresponding disease prediction and lesion localization. The machine-executable instructions described therein, when executed by the processor, also perform the following actions: The chest X-ray image to be diagnosed is input into the image encoder after the second stage of pre-training. To generate feature codes for the image to be diagnosed; Input the name of the disease to be diagnosed into the pre-trained text encoder of the second stage. The code generates the name of the disease to be diagnosed. as well as The feature codes of the image to be diagnosed and the name codes of the disease to be diagnosed are input into the disease query network after the second stage of pre-training. The technology generates disease prediction and lesion localization. The disease query network mentioned above Based on the transformer structure, the disease query network It includes multiple mutual attention interaction layers, wherein an attention map is generated by averaging the mutual attention weights of the multiple mutual attention interaction layers, and the attention map represents the confidence level regarding where the lesion is located in the image.

Images