Multi-modal medical information processing method, equipment and medium

Abstract

The invention relates to the technical field of intelligent medical treatment, in particular to a multi-modal medical information processing method and device and a medium, and the method comprises the following steps: S1, constructing a medical information processing model which comprises a self-adaptive adjustment encoder, a multi-scale self-prompt generation module and a feature pyramid network; s2, training the medical information processing model by adopting cross contrast learning; and S3, acquiring medical information of a patient, and inputting the medical information into the trained medical information processing model to obtain a processing result. According to the method, an adaptive adjustment encoder is adopted, and the adaptability of the model to diversified data is enhanced in combination with information of different knowledge domains while image features are captured.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to a multimodal medical information processing method, device and medium. Background Art

[0002] With the rapid development of artificial intelligence and big data technologies, smart healthcare has gradually become an important direction for the medical industry. As one of the core applications of smart healthcare, digital human doctors can simulate the diagnosis and treatment process of human doctors and provide patients with convenient and intelligent health management and diagnosis and treatment recommendations. However, multimodal data has high dimensions, heterogeneity, and complex semantic associations, which poses challenges to the effective fusion and in-depth analysis of data. Each doctor has different diagnosis and treatment styles, decision-making processes, and focus. Digital human doctors need to have the ability to learn and adapt to these unique styles, which places high demands on the learning and generalization capabilities of the model.

[0003] In summary, there is an urgent need for a multimodal medical information processing method, device and medium to solve the problems in the prior art. Summary of the invention

[0004] The present invention aims to provide a multimodal medical information processing method, device and medium. The specific technical solution is as follows: A multimodal medical information processing method comprises the following steps: S1: constructing a medical information processing model, wherein the medical information processing model includes an adaptive adjustment encoder, a multi-scale self-prompt generation module and a feature pyramid network; S2: using cross-contrast learning to train the medical information processing model; S3: Obtain the patient's medical information, input the medical information into the trained medical information processing model, and obtain the processing result; The adaptive adjustment encoder is used to extract visual features of medical images in medical information. The adaptive adjustment encoder adopts the VIT architecture and uses domain bypass to process image embedding after multi-head self-attention operation.

[0005] Optionally, domain bypass is used to process image embedding after multi-head self-attention operation, as follows: Preprocess the image embedding using domain-universal embedding to obtain the first embedding containing universal knowledge; Processing the first embedding with a domain-specific embedding to obtain a second embedding that contains some domain-specific knowledge; Call the multi-layer perceptron to process the second embedding and get the final image embedding.

[0006] Optionally, a multi-scale self-hint generation module is used to generate self-hint tags with multi-scale features, and the process is as follows: Taking the image embeddings of different layers in the adaptively adjusted encoder as input, fusing the image embeddings of different layers to obtain multi-scale image embeddings; The convolutional layers with different step sizes are used to perform foreground segmentation on multi-scale image embeddings to obtain foreground features. Low-confidence tags in foreground features are filtered out to obtain self-prompted tags.

[0007] Optionally, in the multi-scale self-hint generation module, a binary cross-entropy loss is used to optimize the foreground segmentation.

[0008] Optionally, fusing image embeddings of different layers is specifically: calling a feature pyramid network to integrate image embeddings of different layers from top to bottom.

[0009] Optionally, the medical information processing model is trained using cross-comparison learning, and the process is as follows: Medical information is extracted by using block marking to obtain image foreground marking, image background marking, text foreground features and text background features; Using the text foreground features and the text background features as a positive sample pair and a negative sample pair of the image foreground marker, to obtain first granularity information and second granularity information; Using the text background features and the text foreground features as a positive sample pair and a negative sample pair of image background labels to obtain third granularity information and fourth granularity information; Based on the first granularity information, the second granularity information, the third granularity information and the fourth granularity information, cross-modal contrastive learning is performed to complete the model training.

[0010] Optionally, contrastive learning loss is used as the loss for training the medical information processing model.

[0011] In addition, the present invention also includes a computer device, including a memory and a processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the multimodal medical information processing method as described above when executing the computer program.

[0012] In addition, the present invention also includes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal medical information processing method as described above are implemented.

[0013] The application of the technical solution of the present invention has the following beneficial effects: (1) The present invention proposes an adaptive adjustment encoder (AT-encoder) for effectively coordinating the fusion of visual features with domain-general knowledge and domain-specific knowledge. While capturing image features, the AT-encoder combines information from different knowledge domains to enhance the adaptability of the model to diverse data. In addition, the query decoder in the present invention uses domain query features for fusion decoding, and achieves flexible adaptation of different knowledge domains by dynamically adjusting feature selection during the decoding process. This joint design enables the model to demonstrate excellent generalization capabilities in diverse knowledge domains, which helps the model to be widely used in complex scenarios such as multimodal medical care and industrial detection.

[0014] (2) The multi-scale self-prompt generation module in the present invention is used to automatically generate high-quality self-prompts containing multi-scale knowledge to improve the effect of multimodal data fusion. The multi-scale self-prompt generation module enables the model to extract and utilize information at different scales through a multi-scale processing mechanism, and provides effective support for multimodal fusion tasks in the framework of self-supervised learning. By generating self-prompts with multiple levels of details and context, the multi-scale self-prompt generation module not only enhances the model's ability to understand complex tasks, but also improves its performance when multimodal data is scarce, which helps promote the widespread application of multimodal medicine and other fields.

[0015] (3) The present invention proposes a cross-contrast learning method that applies contrastive learning at different granularity levels of the image to achieve efficient alignment of visual features and text embeddings. The present invention performs fine-grained contrast between the foreground and background of the image, and significantly improves the ability to focus on relevant areas by training the model using only target categories and background labels, while suppressing interference from irrelevant objects. The contrastive learning strategy of the present invention can effectively activate key areas, enabling the model to have higher accuracy and robustness when processing complex visual-text fusion tasks.

[0016] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the steps of the multimodal medical information processing method in the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, this embodiment provides a multimodal medical information processing method, comprising the following steps: A multimodal medical information processing method comprises the following steps: S1: constructing a medical information processing model, wherein the medical information processing model includes an adaptive adjustment encoder, a multi-scale self-prompt generation module and a feature pyramid network; S2: using cross-contrast learning to train the medical information processing model; S3: Obtain the patient's medical information, input the medical information into the trained medical information processing model, and obtain the processing result.

[0021] It should be noted that the medical information includes medical images, case text information, genetic data, etc. By applying the multimodal medical information processing method proposed in this embodiment, the patient's multimodal medical information can be integrated to ultimately generate a processing result. The processing result can be a personalized diagnosis and treatment recommendation that helps the patient recover.

[0022] Optionally, an adaptive adjustment encoder is used to extract visual features of medical images in medical information. The adaptive adjustment encoder adopts a VIT architecture (consisting of multiple Transformer layers) and uses domain bypass to process image embedding after multi-head self-attention operation.

[0023] Optionally, domain bypass is used to process image embedding after multi-head self-attention operation, as follows: Using Domain-Universal Embedding Embedding images After preprocessing, we get the first embedding containing general knowledge, which is expressed as follows: ; in, represents the first embedding.

[0024] Using domain-specific embeddings (No. The first embedding is processed by the domain-specific embedding of medical images to obtain the second embedding containing some domain-specific knowledge, which is expressed as follows: ; in, The second embedding is based on the general knowledge and integrates the Domain specific knowledge.

[0025] Calling a multi-layer perceptron Process the second embedding to get the final image embedding: ; in, represents the final image embedding.

[0026] Optionally, a multi-scale self-hint generation module is used to generate self-hint tags with multi-scale features, and the process is as follows: Taking the image embeddings of different layers in the adaptively adjusted encoder as input, fusing the image embeddings of different layers to obtain multi-scale image embeddings; The convolutional layers with different step sizes are used to perform foreground segmentation on multi-scale image embeddings to obtain foreground features. Filter foreground features The low confidence tags in are used to obtain the self-prompt tags, which are expressed as follows: ; in, Indicates a self-prompt tag. represents the activation function sigmoid, Represents the threshold that will be used to determine high-quality foreground markers.

[0027] Optionally, in the multi-scale self-hint generation module, a binary cross entropy loss is used to optimize foreground segmentation. The expression is as follows: ; represents the number of tokens embedded in the image, Indicates A true value label.

[0028] Optionally, fusing image embeddings of different layers is specifically: calling a feature pyramid network to integrate image embeddings of different layers from top to bottom.

[0029] Optionally, the medical information processing model is trained using cross-comparison learning, and the process is as follows: Medical information is extracted by using block marking to obtain image foreground marking, image background marking, text foreground features and text background features; Text foreground features and text background features As image foreground marker Positive sample pairs and negative sample pairs to obtain the first granularity information and second granularity information ; The expression is as follows: ; in, Indicates image-level labels for the classes, Indicates the number of label categories.

[0030] Text background features and text foreground features As image background marker Positive sample pairs and negative sample pairs to obtain the third granularity information and the fourth granularity information ; The expression is as follows: ; Based on the first granularity information , Second granularity information , Third granularity information and the fourth granularity information , perform cross-modal contrastive learning and complete model training.

[0031] Optionally, contrastive learning loss is used as the loss for training the medical information processing model. The expression is as follows: ; in, represents the number of tokens embedded in the image, represents the hyperparameters used for balancing, Represents the temperature factor.

[0032] In addition, this embodiment also discloses a computer device, including a memory and a processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the above-mentioned multimodal medical information processing method when executing the computer program.

[0033] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device.

[0034] The computer device may be a computing device such as a mobile phone, a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory. For example, the computer device may also include an input / output device, a network access device, a bus, etc.

[0035] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0036] The memory can be used to store the computer program and / or module, and the processor implements the computer program by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0037] Wherein, if the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0038] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multimodal medical information processing method are implemented.

[0039] The present invention provides a multimodal medical information processing method, device and medium based on a temporal knowledge graph. The method of the present invention includes an adaptive adjustment encoder (AT-encoder), a multi-scale self-prompt generation module and a feature pyramid network. The AT-encoder combines information from different knowledge domains while capturing image features, thereby enhancing the adaptability of the model to diversified data; the multi-scale self-prompt generation module uses a multi-scale processing mechanism to enable the model to extract and utilize information at different scales, and provide effective support for multimodal fusion tasks in the framework of self-supervised learning. In addition, the present invention adopts cross-contrast learning and applies contrast learning at different granularity levels of the image to achieve efficient alignment of visual features and text embedding.

[0040] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal medical information processing method, characterized in that: The following steps are involved: S1: constructing a medical information processing model, wherein the medical information processing model includes an adaptive adjustment encoder, a multi-scale self-prompt generation module and a feature pyramid network; S2: using cross-contrast learning to train the medical information processing model; S3: Obtain the patient's medical information, input the medical information into the trained medical information processing model, and obtain the processing result; The adaptive adjustment encoder is used to extract visual features of medical images in medical information. The adaptive adjustment encoder adopts the VIT architecture and uses domain bypass to process image embedding after multi-head self-attention operation.

2. The multimodal medical information processing method according to claim 1, characterized in that: The domain bypass is used to process the image embedding after the multi-head self-attention operation. The process is as follows: Preprocess the image embedding using domain-universal embedding to obtain the first embedding containing universal knowledge; Processing the first embedding with a domain-specific embedding to obtain a second embedding that contains some domain-specific knowledge; Call the multi-layer perceptron to process the second embedding and get the final image embedding.

3. The multimodal medical information processing method according to claim 2, characterized in that: The multi-scale self-hint generation module is used to generate self-hint tags with multi-scale features. The process is as follows: Taking the image embeddings of different layers in the adaptively adjusted encoder as input, fusing the image embeddings of different layers to obtain multi-scale image embeddings; The convolutional layers with different step sizes are used to perform foreground segmentation on multi-scale image embeddings to obtain foreground features. Low-confidence tags in foreground features are filtered out to obtain self-prompted tags.

4. The multimodal medical information processing method according to claim 3, characterized in that: In the multi-scale self-hint generation module, a binary cross-entropy loss is adopted to optimize the foreground segmentation.

5. The multimodal medical information processing method according to claim 4, characterized in that: The specific method of fusing image embeddings at different layers is to call the feature pyramid network to integrate image embeddings at different layers from top to bottom.

6. The multimodal medical information processing method according to claim 5, characterized in that: The medical information processing model is trained using cross-contrast learning, and the process is as follows: Medical information is extracted by using block marking to obtain image foreground marking, image background marking, text foreground features and text background features; Using the text foreground features and the text background features as a positive sample pair and a negative sample pair of the image foreground marker, to obtain first granularity information and second granularity information; Using the text background features and the text foreground features as a positive sample pair and a negative sample pair of image background labels to obtain third granularity information and fourth granularity information; Based on the first granularity information, the second granularity information, the third granularity information and the fourth granularity information, cross-modal contrastive learning is performed to complete the model training.

7. The multimodal medical information processing method according to claim 6, characterized in that: Contrastive learning loss is used as the loss for training the medical information processing model.

8. A computer device, characterized in that: including memory and processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the multimodal medical information processing method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal medical information processing method according to any one of claims 1 to 7 are implemented.

Images