Multi-image fusion diagnosis report generation method, model training method and equipment

Abstract

The present invention discloses a diagnostic report generation method, a model training method and a device for multi-image fusion, wherein the model training method comprises the following steps: collecting confocal microendoscopic images of different parts and corresponding white light endoscopic images and diagnostic reports, and establishing a data set; constructing a diagnostic report generation model including an image decoding module and a text generation module; training the diagnostic report generation model based on the data set until a preset convergence condition is reached; the image decoding module is used to generate a fusion result of a local feature vector and an overall feature vector, and the text generation module is used to generate a first topic vector corresponding to the fusion result, and the first topic vector and the second topic vector corresponding to the prior knowledge are sentence-merged to generate a diagnostic report. The present invention not only reduces the burden on doctors and improves the speed of diagnosis, but also avoids omissions, misjudgments and the like, and improves the accuracy of diagnostic reports.

Description

Technical Field

[0001] The present invention relates to the field of automatic generation of diagnostic reports, and in particular to a multi-image fusion diagnostic report generation method, a model training method and a device. Background Art

[0002] Confocal microendoscopy is a medical device that can be inserted into the human body through channels such as gastroscopes and colonoscopes to obtain local histological images at the cellular and subcellular levels, thereby accurately diagnosing diseases such as tiny lesions, gastrointestinal lesions, and early gastrointestinal cancer. It is often used in combination with white light endoscopy in clinical examinations.

[0003] After performing a confocal endomicroscopy examination, the doctor needs to personally analyze the confocal endomicroscopy images and the corresponding white light endoscopic images of the examination process and write a diagnostic report, which is then manually entered into a dedicated EIS (Endoscopy Center Information System). This is not only time-consuming and labor-intensive, it greatly reduces the efficiency of the examination, and depending on the doctor's experience, omissions and misjudgments may occur. Summary of the invention

[0004] The present invention provides a multi-image fusion diagnosis report generation method, a model training method and a device, which solve the above-mentioned technical problems.

[0005] A first aspect of an embodiment of the present invention provides a multi-image fusion diagnosis report generation model training method, comprising the following steps:

[0006] Step 1, collecting confocal endomicroscopic images of different parts and corresponding white light endoscopic images and diagnostic reports to establish a data set;

[0007] Step 2, constructing a diagnosis report generation model, the diagnosis report generation model includes an image decoding module and a text generation module, the image decoding module is used to extract the local feature vector of the confocal microendoscopy image and the overall feature vector of the white light endoscopic image, and generate a fusion result of the local feature vector and the overall feature vector; the text generation module is used to generate a first topic vector corresponding to the fusion result, and merge the first topic vector and the second topic vector corresponding to the prior knowledge into sentences to generate a diagnosis report;

[0008] Step 3: training the diagnostic report generation model based on the data set and optimizing model parameters until a preset convergence condition is reached.

[0009] A second aspect of an embodiment of the present invention provides a method for generating a multi-image fusion diagnosis report, comprising the following steps:

[0010] Step 401, collecting medical image pictures to be fused, wherein the medical image pictures include a target confocal microendoscopic image and a target white-light endoscopic image;

[0011] Step 402, identifying the target confocal microendoscopic image by using a preset lesion diagnosis model to generate target disease diagnosis information;

[0012] Step 403, identifying the target white light endoscopic image by using a preset part recognition model to generate target part information;

[0013] Step 404: input the medical image and the corresponding target disease diagnosis information and target site information into the diagnosis report generation model to generate a graphic diagnosis report including the medical image.

[0014] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the processor executes the steps of the above-mentioned training method or the above-mentioned diagnostic report generating method.

[0015] A fourth aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] A storage device is used to store one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the training method or the diagnostic report generating method.

[0018] The beneficial effects of the present invention are as follows: the present invention provides a multi-image fusion diagnostic report generation method, model training method and equipment, which can identify and fuse the image features of confocal microendoscopy images and related white light endoscopic images, and automatically output the corresponding graphic diagnostic report in combination with disease diagnosis information and location information, which not only reduces the burden on doctors and improves the diagnosis speed, but also avoids omissions, misjudgments, etc., and improves the accuracy of the diagnostic report.

[0019] In order to make the above-mentioned objects, features and advantages of the invention more obvious and easy to understand, the preferred embodiments of the present invention are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a flowchart of the multi-image fusion diagnosis report generation model training method provided in Example 1;

[0022] Figure 2 is a schematic diagram of some confocal microendoscopic images and diagnostic reports in the database in the model training method provided in Example 1;

[0023] Figure 3 is a schematic diagram of associated files in the model training method provided in Example 1;

[0024] Figure 4 is a flowchart of a method for generating a diagnosis report by multi-image fusion provided in Example 2;

[0025] Figure 5 It is a structural schematic diagram of a diagnostic report generation model training device for multi-image fusion provided in Example 3;

[0026] Figure 6 It is a schematic diagram of the structure of the multi-image fusion diagnosis report generating device provided in Example 4. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the scope of protection of the present invention. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in the present invention do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0029] Figure 1FIG. 1 is a flow chart of a multi-image fusion diagnosis report generation model training method provided in Example 1. Figure 1 As shown, the following steps are included:

[0030] Step 1, collecting confocal endomicroscopic images of different parts and corresponding white light endoscopic images and diagnostic reports to establish a data set;

[0031] Step 2, constructing a diagnosis report generation model, the diagnosis report generation model includes an image decoding module and a text generation module, the image decoding module is used to extract the local feature vector of the confocal microendoscopy image and the overall feature vector of the white light endoscopic image, and generate a fusion result of the local feature vector and the overall feature vector; the text generation module is used to generate a first topic vector corresponding to the fusion result, and merge the first topic vector and the second topic vector corresponding to the prior knowledge into sentences to generate a diagnosis report;

[0032] Step 3: training the diagnostic report generation model based on the data set and optimizing model parameters until a preset convergence condition is reached.

[0033] The above embodiment provides a method for training a multi-image fusion diagnosis report generation model. The diagnosis report generation model can identify and fuse the image features of confocal microendoscopy images and related white light endoscopic images, and automatically output the corresponding graphic diagnosis report in combination with disease diagnosis information and site information. The model has a simple structure and fast automatic parsing speed, which not only reduces the burden on doctors and improves the diagnosis speed, but also avoids omissions, misjudgments, etc., and improves the accuracy of the diagnosis report.

[0034] Each step of the above method is described in detail below using specific embodiments.

[0035] In a specific embodiment, the step 1 specifically includes:

[0036] S101, collecting confocal endoscopic microscopy images of different parts and corresponding white light endoscopic images and diagnostic reports. Figure 2 As shown, one embodiment collected 474 confocal microendoscopic images of 90 patients ( Figure 2 (The white light endoscopy images at the same time are omitted in the figure). Corresponding to each image, there is a corresponding diagnosis report from the doctor.

[0037] S102, extracting text from the diagnosis report to form a target diagnosis text result;

[0038] S103, using the coco dataset format to associate the confocal microendoscopic image, the white light endoscopic image and the corresponding target diagnosis text results, and save them as an associated file in json format in pairs of images and text results, such as Figure 3 As shown. It is well known to those skilled in the art that the full name of the coco dataset is Common Objects in Context. It is a large image dataset that contains rich annotation information. It not only annotates the bounding box of the object, but also provides the segmentation mask of the object and image description, etc., so it is widely used in computer vision fields such as object recognition, segmentation and image description, and also has very good recognition effect in the confocal microendoscopy image recognition and white light endoscopic image recognition of the present invention.

[0039] In a preferred embodiment, the diagnostic report generation model includes an image decoding module and a text generation module. It is understandable that the confocal microendoscopic image focuses on the local detail features of the target area, while the white light endoscopic image is aimed at the overall features of the target. In order to fuse multi-scale features, this embodiment uses two feature networks of the image decoding module to extract the local feature vector of the confocal microendoscopic image and the overall feature vector of the white light endoscopic image, respectively, and fuses them based on the bilinear self-attention feature fusion method to generate a fusion result; then the first topic vector corresponding to the fusion result is generated through the text generation module, and the first topic vector and the second topic vector based on prior knowledge are sentence-merged to finally generate a diagnostic report.

[0040] Specifically, in a preferred embodiment, the fusion result of the local feature vector and the overall feature vector is generated as follows:

[0041] S201, using a first feature extraction network, such as a ResNet34 network, to extract features from the confocal microendoscopy image to generate a local feature vector F1, and using a second feature extraction network, such as a ResNet101 network, to extract features from the associated white light endoscopic image to generate an overall feature vector F2. The above ResNet34 network and ResNet101 network are both composed of a series of residual structures (Residual Blocks). The ResNet34 network is shallower and can focus on local features faster and capture specific details of the image faster, while the ResNet101 network is deeper and can capture higher-level features and overall structures. The two differ in network structure, and the specific feature extraction method is not described in detail here.

[0042] Then, S202 is executed to perform a linear transformation on the local feature vector F1 using the pre-trained attention weight matrix W, and the linear transformation result is normalized using the softmax function to generate an attention weight vector w. The specific formula is as follows:

[0043] w = soft max(W·F1);

[0044] S203, performing weighted fusion on the overall feature vector F2 using the attention weight vector w to generate a fusion result The formula used is as follows:

[0045] where represents matrix dot product, i.e. element-by-element multiplication.

[0046] S204, calculating the fusion result The outer product of the local feature vector F1 generates a fused feature vector F3. The specific formula is as follows:

[0047]

[0048] Finally, S205 is executed to perform global average pooling on the fused feature vector F3 to generate the bilinear attention feature F of the confocal microendoscopy image and the white light endoscopic image. The above process uses a bilinear self-attention feature fusion method to fuse the overall features of the white light endoscopic image with the local features of the confocal microendoscopy image, and calculates the attention weights between different features during the fusion process, that is, dynamically adjusts the importance of different features through the relationship between different features, and obtains a more accurate image feature fusion result.

[0049] Then, based on the text generation module, a diagnosis report is generated according to the image feature fusion result and prior knowledge. In a preferred embodiment, the prior knowledge includes disease diagnosis information and site information. Specifically, the disease diagnosis information is generated by identifying the confocal microendoscopic image through a preset lesion diagnosis model; and the site information is generated by identifying the associated white light endoscopy image through a preset site recognition model, thereby encoding and fusion of the two prior knowledge through a bidirectional long short-term memory network, and finally generating a second topic vector.

[0050] Specifically, in a preferred embodiment, generating a diagnostic report by the text generation module specifically includes the following steps:

[0051] S206, receiving the bilinear attention feature F at each time step, and outputting a first topic vector v corresponding to the bilinear attention feature F through a preset first BiLSTM bidirectional long short-term memory network i and the cutoff probability r i, until the cutoff probability r i If the probability is greater than a preset threshold, such as 0.5, then stop generating words and stop outputting topic vectors. The specific formula used is as follows:

[0052]

[0053] v i =tanh(W v h t sent +b v )

[0054]

[0055] where b v ,b r is the bias after training, W v ,W r is the trained weight matrix, h t sent is the hidden state at time step t, is the hidden state of the previous time step t-1, v i is the first topic vector of the output, r i is the cut-off probability, used to determine whether it is the last sentence of the paragraph. Formula h t sent represents the hidden state h at time step t t sent It is calculated by the BiLSTM network, which depends on the hidden state of the previous time step t-1 and the input F at the current time step.

[0056] Then, S207 is executed to encode and fuse the prior knowledge, including the disease diagnosis information and the location information, to generate a second topic vector based on the prior knowledge.

[0057] S208, merging the first topic vector, the cutoff probability and the corresponding characters of the second topic vector into a diagnosis report by presetting a second BiLSTM bidirectional long short-term memory network.

[0058] It can be understood that, in a preferred embodiment, the disease diagnosis information and the location information are encoded and fused to generate a second topic vector based on prior knowledge, specifically:

[0059] S2071, converting the disease diagnosis information and the part information into disease diagnosis information word vectors and part information word vectors respectively through a preset word embedding model, such as a pre-trained Word2Vec network.

[0060] Specifically, disease diagnosis information is recorded as in k is 1, 2, ..., M, which refers to a character in the disease diagnosis information, and M is the length of all characters in the disease diagnosis information. The part information is recorded as in k is 1, 2, ... N refers to a character in the part information, and N is the length of all characters in the part information. The part information and disease diagnosis information are input into the corresponding trained word embedding model Word2Vec network to extract the corresponding disease diagnosis information word vector Word vector with position information

[0061] S2072, encoding the disease diagnosis information word vector by presetting a third BiLSTM bidirectional long short-term memory network as follows:

[0062]

[0063] in For the input at each time step t, is the output for each time step t, including M time steps.

[0064] S2073, clustering and fusing the part information word vector and the encoded disease diagnosis information word vector using a multi-layer perceptual attention mechanism, and normalizing the fusion result using a softmax function, the fusion method is as follows:

[0065]

[0066] Where W h ,W k is the weight matrix after training, r T is the trained weight vector, and σ is the sigmoid function.

[0067] And the normalization method is as follows; x t Represents the normalized result.

[0068] S2074, merging the normalized result, the disease diagnosis information word vector, and the part information word vector to generate the second topic vector based on prior knowledge, using the following formula:

[0069]

[0070] Wherein, Z represents the second topic vector, and T represents all time steps.

[0071] The above method can be used to generate a second topic vector based on the prior knowledge, thereby combining the first topic vector, the cutoff probability and the corresponding characters of the second topic vector into a diagnosis report.

[0072] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0073] Figure 4 FIG. 2 is a flow chart of a method for generating a multi-image fusion diagnosis report provided in Example 2. Figure 4 As shown, the following steps are included:

[0074] Step 401, collecting medical image pictures to be fused, wherein the medical image pictures include a target confocal microendoscopic image and a target white-light endoscopic image;

[0075] Step 402, identifying the target confocal microendoscopic image by using a preset lesion diagnosis model to generate target disease diagnosis information;

[0076] Step 403, identifying the target white light endoscopic image by using a preset part recognition model to generate target part information;

[0077] Step 404: input the medical image and the corresponding target disease diagnosis information and target site information into the diagnosis report generation model to generate a graphic diagnosis report including the medical image.

[0078] The above-mentioned embodiment provides a method for generating a diagnosis report by fusion of multiple images, which uses a pre-trained diagnosis report generation model to extract features of the target medical image and combines the prior knowledge identified by a preset lesion diagnosis model and a preset part recognition model to automatically generate a diagnosis report. This not only reduces the burden on doctors and improves the diagnosis speed, but also avoids omissions, misjudgments, etc., thereby improving the accuracy of the diagnosis report.

[0079] In a preferred embodiment, the diagnostic report generation method further includes a data association step, specifically:

[0080] Acquire the white light endoscopic image after the quality control monitoring step, and establish a correlation relationship between the white light endoscopic image and the confocal microendoscopic image at the same time, as well as the corresponding target disease diagnosis information and target site information;

[0081] The associated white light endoscopic image, the confocal microendoscopic image, the target disease diagnosis information, the target site information and the association relationship are stored in the same storage location according to a preset coding method.

[0082] In the above preferred embodiment, the quality control monitoring step mainly refers to calculating the similarity between the current image and any image in the adjacent time period by a deep learning method for the input white light endoscopic image, calculating the speed information of the mirror entry and mirror withdrawal according to the similarity and frame rate, and judging the quality control monitoring result of the white light endoscopic image, and judging whether to delete or save the white light endoscopic image based on the quality control monitoring result. At the same time, in this embodiment, multiple data at the same time are saved together, and an association relationship is established between them according to time, and the association relationship is recorded when saving. Not only can the disease diagnosis result and the associated part information be obtained through the pre-trained lesion diagnosis model and part recognition model, but also the white light endoscopic image and the confocal microendoscopic image at the same time can be input into the EIS system (endoscopic center information system) together, which has high reliability and convenient operation, and is convenient for doctors to view the white light endoscopic image and the confocal microendoscopic image at the same time when editing or checking the automatically generated diagnosis report, thereby improving the modification efficiency.

[0083] In a preferred embodiment, a local server is selected for model training, and the training environment is shown in Table 1 below:

[0084] Table 1 Training environment table

[0085] Graphics RTX4090 24GB Running the system Ubuntu 18.04 frame PyTorch 1.10.0 Python Version 3.9 CUDA Version 11.1

[0086] The constructed dataset is input into the model for training. The training set, validation set, and test set are randomly divided according to 8:1:1. The training set is used to train the model, the validation set is used to verify the model performance, and the test set is used to test the model. The training method selects the simple cross-validation method, the Epoch is set to 150, the batch_size is set to 4, the initial learning rate is 0.0001, the optimizer is selected as adam, the cutoff threshold is selected as 0.5, the maximum text generation length is set to 65, the first feature extraction network selects ResNet34, and the second feature extraction network selects ResNet101. The text generation module is built using the BiLSTM layer. The parameters of the three submodules are set as follows: the hidden state and word embedding of the BiLSTM layer of the first topic vector generation module are 512, the hidden state and word embedding of the BiLSTM layer of the second topic vector generation module are 300, and the hidden state and word embedding of the BiLSTM layer of the sentence generation module are 512.

[0087] Finally, the common indicators BELU, Rouge-L, and CIDEr in natural language generation tasks are used to evaluate the diagnostic report generation model proposed in the present invention. The performance of this model on the test set is shown in Table 2 below, and it has a relatively excellent report generation effect.

[0088] Table 2 Test data table

[0089] BLEU-1 BLEU-2 Rouge-L CIDEr This system 73.65 64.82 75.12 5.861

[0090] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned multi-image fusion diagnosis report generation model training method or multi-image fusion diagnosis report generation method is implemented.

[0091] Figure 5 3 is a schematic diagram of the structure of a diagnostic report generation model training device for multi-image fusion provided in Example 3. Figure 5 As shown, it includes a data acquisition module 100, a model building module 200 and a training module 300.

[0092] The data acquisition module 100 is used to acquire confocal endomicroscopic images of different parts and corresponding white light endoscopic images and diagnostic reports to establish a data set;

[0093] The model building module 200 is used to build a diagnosis report generation model, and the diagnosis report generation model includes an image decoding module 201 and a text generation module 202. The image decoding module is used to extract the local feature vector of the confocal microendoscopy image and the overall feature vector of the white light endoscopic image, and generate a fusion result of the local feature vector and the overall feature vector; the text generation module is used to generate a first topic vector corresponding to the fusion result, and merge the first topic vector with the second topic vector corresponding to the prior knowledge to generate a diagnosis report;

[0094] The training module 300 is used to train the diagnosis report generation model based on the data set and optimize the parameters of the diagnosis report generation model until a preset convergence condition is reached.

[0095] The above embodiment provides a multi-image fusion diagnosis report generation model training device, through which the diagnosis report generation model can identify and fuse the image features of confocal microendoscopy images and related white light endoscopic images, and automatically output the corresponding graphic diagnosis report in combination with disease diagnosis information and site information. The model structure is simple and the automatic parsing speed is fast, which not only reduces the burden on doctors and improves the diagnosis speed, but also avoids omissions, misjudgments, etc., and improves the accuracy of the diagnosis report.

[0096] In a preferred embodiment, the data acquisition module 100 specifically includes:

[0097] An acquisition unit, used to acquire confocal endomicroscopic images of different parts and corresponding white light endoscopic images and diagnostic reports;

[0098] A text processing unit, used for extracting text from the diagnosis report to form a target diagnosis text result;

[0099] A saving unit is used to associate the confocal endomicroscopy image, the white light endomicroscopy image and the corresponding target diagnosis text results in the coco data set format, and save them as an associated file in the json format.

[0100] In a preferred embodiment, the image decoding module 201 specifically includes:

[0101] A first feature extraction network is used to extract features from the confocal microendoscopy image using a ResNet34 network to generate a local feature vector;

[0102] The second feature extraction network is used to extract features of the associated white light endoscopic image using a ResNet101 network to generate an overall feature vector;

[0103] A normalization unit, used to perform a linear transformation on the local feature vector using a pre-trained attention weight matrix, and normalize the linear transformation result using a softmax function to generate an attention weight vector;

[0104] A first computing unit, configured to perform weighted fusion on the overall feature vector by using the attention weight vector to generate a fusion result;

[0105] A second calculation unit, used for calculating the outer product of the fusion result and the local feature vector to generate a fusion feature vector;

[0106] A pooling unit is used to perform global average pooling on the fused feature vector to generate bilinear attention features of the confocal microendoscopy image and the white light endoscopic image.

[0107] In a preferred embodiment, the text generation module 202 specifically includes:

[0108] A first topic vector generation module is used to receive the bilinear attention feature at each time step, and output a first topic vector and a cutoff probability corresponding to the bilinear attention feature through a preset first BiLSTM bidirectional long short-term memory network until the cutoff probability is greater than a preset probability threshold;

[0109] A second topic vector generating module, used for encoding and fusing the disease diagnosis information and the part information to generate a second topic vector based on prior knowledge;

[0110] A sentence generation module is used to merge the first topic vector, the cutoff probability and the corresponding characters of the second topic vector into a diagnosis report by presetting a second BiLSTM bidirectional long short-term memory network.

[0111] In a preferred embodiment, the second topic vector generating module specifically includes:

[0112] A word embedding unit, used to convert the disease diagnosis information and the part information into a disease diagnosis information word vector and a part information word vector respectively through a preset word embedding model;

[0113] An encoding unit, used for encoding the disease diagnosis information word vector by using a preset third BiLSTM bidirectional long short-term memory network;

[0114] A fusion unit, used for clustering and fusing the part information word vector and the encoded disease diagnosis information word vector using a multi-layer perceptual attention mechanism, and normalizing the fusion result by a softmax function;

[0115] The topic vector generating unit is used to merge the normalization result, the disease diagnosis information word vector and the part information word vector to generate the second topic vector based on prior knowledge.

[0116] Figure 6 4 is a schematic diagram of the structure of a diagnostic report generating device for multi-image fusion provided in Example 4. Figure 6 As shown, it includes an image acquisition module 401, a first recognition module 402, a second recognition module 403 and a report generation module 404.

[0117] The image acquisition module 401 is used to collect medical images to be fused, wherein the medical images include a target confocal microendoscopic image and a target white light endoscopic image;

[0118] The first recognition module 402 is used to recognize the target confocal microendoscopic image through a preset lesion diagnosis model to generate target disease diagnosis information;

[0119] The second recognition module 403 is used to recognize the target white light endoscopy image by using a preset part recognition model to generate target part information;

[0120] The report generation module 404 is used to input the medical image and the corresponding target disease diagnosis information and target site information into the diagnosis report generation model to generate a graphic diagnosis report including the medical image.

[0121] In a preferred embodiment, the diagnostic report generating device also includes a data association module, which is used to collect white light endoscopic images that have undergone quality control monitoring steps, establish an association relationship between the white light endoscopic image and the confocal microendoscopic image at the same time and the corresponding target disease diagnosis information and target site information; and is used to save the associated white light endoscopic image, the confocal microendoscopic image, the target disease diagnosis information, the target site information and the association relationship in the same storage location according to a preset coding method.

[0122] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the processor is caused to execute the training method or the diagnostic report generating method.

[0123] An embodiment of the present invention also provides an electronic device, including a computer-readable storage medium and a processor, wherein when the processor executes the computer program on the computer-readable storage medium, it implements the steps of the above-mentioned multi-image fusion diagnosis report generation model training method or the steps of the multi-image fusion diagnosis report generation method.

[0124] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0127] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.

Claims

1. A multi-image fusion diagnosis report generation model training method, characterized in that: The following steps are involved: Step 1, collecting confocal endomicroscopic images of different parts and corresponding white light endoscopic images and diagnostic reports to establish a data set; Step 2, constructing a diagnosis report generation model, the diagnosis report generation model includes an image decoding module and a text generation module, the image decoding module is used to extract the local feature vector of the confocal microendoscopy image and the overall feature vector of the white light endoscopic image, and generate a fusion result of the local feature vector and the overall feature vector; the text generation module is used to generate a first topic vector corresponding to the fusion result, and merge the first topic vector and the second topic vector corresponding to the prior knowledge into sentences to generate a diagnosis report; Step 3: training the diagnostic report generation model based on the data set and optimizing model parameters until a preset convergence condition is reached.

2. The diagnostic report generation model training method of multi-image fusion according to claim 1 is characterized in that: The step 1 is specifically as follows: Collect confocal endomicroscopic images of different parts and corresponding white light endoscopic images and diagnostic reports; Extracting text from the diagnostic report to form a target diagnostic text result; The confocal endomicroscopy image, the white light endomicroscopy image and the corresponding target diagnosis text results are associated using the coco dataset format and saved as an associated file in json format.

3. The diagnostic report generation model training method of multi-image fusion according to claim 1 or 2 is characterized in that: The fusion result of the local feature vector and the overall feature vector is generated by the image decoding module, specifically: Using a first feature extraction network to extract features from the confocal microendoscopy image to generate a local feature vector; A second feature extraction network is used to extract features from the associated white light endoscopy image to generate an overall feature vector; Performing a linear transformation on the local feature vector using a pre-trained attention weight matrix, and normalizing the linear transformation result using a softmax function to generate an attention weight vector; Performing weighted fusion on the overall feature vector by using the attention weight vector to generate a fusion result; Calculating the outer product of the fusion result and the local feature vector to generate a fusion feature vector; Global average pooling is performed on the fused feature vector to generate bilinear attention features of the confocal endomicroscopy image and the white light endomicroscopy image.

4. The diagnostic report generation model training method of multi-image fusion according to claim 3 is characterized in that: The prior knowledge includes disease diagnosis information and location information. The confocal microendoscopic image is identified by a preset lesion diagnosis model to generate the disease diagnosis information; and the associated white light endoscopy image is identified by a preset part recognition model to generate the part information.

5. The diagnostic report generation model training method of multi-image fusion according to claim 4 is characterized in that: The diagnostic report is generated by the text generation module, specifically: Receiving the bilinear attention feature at each time step, and outputting a first topic vector and a cutoff probability corresponding to the bilinear attention feature through a preset first BiLSTM bidirectional long short-term memory network until the cutoff probability is greater than a preset probability threshold; Encoding and fusing the disease diagnosis information and the location information to generate a second topic vector based on prior knowledge; The first topic vector, the cutoff probability and the corresponding characters of the second topic vector are combined into a diagnosis report by presetting a second BiLSTM bidirectional long short-term memory network.

6. The multi-image fusion diagnosis report generation model training method according to claim 5, characterized in that: The disease diagnosis information and the location information are coded and fused, specifically: The disease diagnosis information and the part information are converted into a disease diagnosis information word vector and a part information word vector respectively through a preset word embedding model; Encoding the disease diagnosis information word vector by presetting a third BiLSTM bidirectional long short-term memory network; Using a multi-layer perceptual attention mechanism to cluster and fuse the part information word vector and the encoded disease diagnosis information word vector, and normalizing the fusion result through a softmax function; The normalized result, the disease diagnosis information word vector and the part information word vector are merged to generate the second topic vector based on prior knowledge.

7. A method for generating a diagnosis report by multi-image fusion, characterized in that: The following steps are involved: Step 401, collecting medical image pictures to be fused, wherein the medical image pictures include a target confocal microendoscopic image and a target white-light endoscopic image; Step 402, identifying the target confocal microendoscopic image by using a preset lesion diagnosis model to generate target disease diagnosis information; Step 403, identifying the target white light endoscopic image by using a preset part recognition model to generate target part information; Step 404, input the medical image and the corresponding target disease diagnosis information and target site information into the diagnosis report generation model trained by the method according to any one of claims 1-6, and generate a graphic diagnosis report including the medical image.

8. The method for generating a diagnosis report by multi-image fusion according to claim 7, characterized in that: It also includes data association steps, specifically: Acquire the white light endoscopic image after the quality control monitoring step, and establish a correlation relationship between the white light endoscopic image and the confocal microendoscopic image at the same time, as well as the corresponding target disease diagnosis information and target site information; The associated white light endoscopic image, the confocal microendoscopic image, the target disease diagnosis information, the target site information and the association relationship are name-coded according to a preset coding method and stored in the same storage location.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the training method as described in any one of claims 1 to 6 or the diagnostic report generation method as described in claim 7 or 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the processor is caused to execute the training method described in any one of claims 1 to 6 or the diagnostic report generating method described in claim 7 or 8.

Images