Medical report generation method and device based on multi-branch feature fusion

Abstract

The present application relates to the technical field of medical report generation, and provides a medical report generation method, device and equipment based on multi-branch feature fusion, which acquires a medical image, inputs the medical image into a trained generation model, and acquires a target medical report. The generation model comprises a multi-branch feature extraction network, a multi-head attention fusion network and a report generation network. The multi-branch feature extraction network is composed of two improved EfficientNet subnetworks and two improved EfficientDet subnetworks. The subnetworks extract different scale features of the medical image, acquire rich feature information of the medical image, the multi-head attention fusion network comprises a channel attention fusion module and a spatial attention fusion module, the fusion module fuses three scale features in the multiple scale features, realizes feature complementation of different scale features, fully utilizes feature information of each scale feature, and finally decodes target fusion features through the report generation network to acquire a high-quality target medical report.

Description

Technical Field

[0001] This invention relates to the field of medical report generation technology, and in particular to a method, apparatus and equipment for generating medical reports based on multi-branch feature fusion. Background Technology

[0002] In the medical field, managing and utilizing clinical data is crucial for improving the quality and efficiency of healthcare. Currently, user basic information, complex imaging data, and other data are constantly being generated, providing data support for medical decision-making. In recent years, with the rapid development of artificial intelligence and big data technologies, these technologies have been widely applied in the medical field, especially medical report generation technology, which is gradually becoming an important component of intelligent assisted treatment.

[0003] In existing technologies, deep learning-based methods can be used to automatically analyze medical data and generate medical reports. For example, a holistic solution framework based on cue learning can be used to generate medical reports. Specifically, this involves combining human cue learning incorporating domain expert knowledge with automatic cue learning using a weighted random sampling strategy, leveraging the powerful sample learning and reasoning capabilities of the Chat GPT model to generate medical reports. Another approach is a feature alignment model based on expert networks and prototype sentences. Specifically, a prototype sentence strategy is used to extract sentence features from the report text and align them with image features. The expert network guides the model to select image features with high semantic similarity to text features, strengthening the model's semantic understanding of image features, thereby generating medical reports. There is also a feature fusion model based on global and local anatomical regions. Specifically, cue learning identifies and labels sentence-level anatomical regions in the report text description, and image anatomical region feature recognition and segmentation techniques are introduced. An attention mechanism is used to fuse global and local image and text features to generate medical reports.

[0004] However, in existing technologies, deep learning-based medical report generation models only perform single feature extraction on medical images, resulting in a single feature extraction dimension and thus failing to generate high-quality medical reports. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, and device for generating medical reports based on multi-branch feature fusion to address the aforementioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a medical report generation method based on multi-branch feature fusion, the method comprising:

[0007] Acquire medical imaging images;

[0008] The medical image is input into the trained generative model to obtain the target medical report of the medical image;

[0009] The generative model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network is used to extract features at different scales. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. Each fusion module is used to fuse three scale features from multiple scale features to obtain target fusion features. The report generation network is used to decode multiple target fusion features to obtain the target medical report corresponding to the medical image.

[0010] In one embodiment, before inputting the medical image into the trained generative model to obtain the target medical report of the medical image, the method further includes:

[0011] Obtain a sample training set, wherein the sample training set includes: multiple sets of training samples, each set of training samples including training medical image, preset medical report text corresponding to the training medical image, and location labels of target objects contained in the training medical image;

[0012] The sample training set is input into the initial generative model, and the weight parameters of the model are adjusted according to the loss function until the model converges, thus obtaining the trained generative model.

[0013] In one embodiment, inputting the medical image into a trained generative model to obtain a target medical report of the medical image includes:

[0014] The medical image is input into the multi-branch feature extraction network, and features are extracted from the medical image through each of the improved EfficientNet sub-networks and each of the improved EfficientDet sub-networks to obtain features at different scales. The improved EfficientNet sub-network is determined by improving upon the first output head of the EfficientNet network, and the improved EfficientDet sub-network is determined by improving upon the second output head of the EfficientDet network.

[0015] Multiple scale features are input into the multi-head attention fusion network, and three scale features from the multiple scale features are fused through each fusion module to obtain the target fusion feature corresponding to each fusion module;

[0016] Multiple target fusion features are input into the report generation network, and the target medical report is obtained through the alignment and decoding of the multiple target fusion features by the report generation network.

[0017] In one embodiment, the step of inputting the medical image into the multi-branch feature extraction network, and extracting features from the medical image through various improved EfficientNet subnetworks and various improved EfficientDet subnetworks to obtain features at different scales includes:

[0018] The medical image is input into the first improved EfficientNet subnetwork for category feature extraction to obtain the first category scale feature;

[0019] The medical image is input into the first improved EfficientDet sub-network for anchor point feature extraction to obtain the first anchor point scale feature.

[0020] The medical image is input into the second improved EfficientNet subnetwork for category feature extraction to obtain the second category scale feature;

[0021] The medical image is input into the second improved EfficientDet subnetwork for anchor point feature extraction to obtain the second anchor point scale feature.

[0022] In one embodiment, the first output head includes: a convolutional layer, a pooling layer, and a fully connected layer. The improved EfficientNet subnetwork is determined based on the first output head of the EfficientNet network and includes:

[0023] The improved EfficientNet subnetwork is obtained by replacing the pooling layer in the first output head with a fully connected layer.

[0024] The second output head includes a first convolutional module, a second convolutional module, and a fully connected layer connected in parallel. The second convolutional module includes a first convolutional layer and a second convolutional layer. The improved EfficientDet subnetwork is determined by improving the second output head of the EfficientDet network, and includes:

[0025] The improved EfficientDet subnetwork is obtained by replacing the second convolutional layer with four fully connected layers.

[0026] In one embodiment, the step of inputting multiple scale features into the multi-head attention fusion network, and fusing three scale features from the multiple scale features through various fusion modules to obtain the target fusion feature corresponding to each fusion module includes:

[0027] The first category scale feature, the first anchor point scale feature, and the second category scale feature are input into the channel attention fusion module for fusion to obtain the first target fusion feature corresponding to the channel attention fusion module.

[0028] The first anchor point scale feature, the second category scale feature, and the second anchor point scale feature are input into the spatial attention fusion module for fusion to obtain the second target fusion feature corresponding to the spatial attention fusion module.

[0029] In one embodiment, the channel attention fusion module includes: a multi-head attention mechanism unit and an affine transformation unit;

[0030] The spatial attention fusion module includes a multi-head attention mechanism unit and an affine transformation unit.

[0031] In one embodiment, the report generation network includes an alignment module, multiple deep feature extraction modules, and a decoding module. The step of inputting multiple target fusion features into the report generation network, and obtaining the target medical report through alignment and decoding of the multiple target fusion features by the report generation network, includes:

[0032] The first target fusion feature and the second target fusion feature are input into the alignment module for alignment processing to obtain the target alignment fusion feature;

[0033] The target alignment and fusion features are input into multiple deep feature extraction modules to extract deep features and obtain the target's deep features.

[0034] The target depth features are input into the decoding module for decoding processing to obtain the target medical report.

[0035] Secondly, embodiments of the present invention provide a medical report generation device based on multi-branch feature fusion, comprising:

[0036] The acquisition module is used to acquire medical imaging images;

[0037] The generation module is used to input the medical image into the trained generation model to obtain the target medical report of the medical image;

[0038] The generative model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network is used to extract features at different scales. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. Each fusion module is used to fuse three scale features from multiple scale features to obtain target fusion features. The report generation network is used to decode multiple target fusion features to obtain the target medical report corresponding to the medical image.

[0039] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the medical report generation method based on multi-branch feature fusion described in the first aspect.

[0040] The technical solution provided by the embodiments of the present invention has the following advantages compared with the prior art:

[0041] This invention provides a medical report generation method based on multi-branch feature fusion. By acquiring medical images and inputting them into a trained generation model, a target medical report is obtained. The generation model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network extracts different scale features from the medical image to obtain rich feature information of the target object contained in the medical image. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. The fusion module fuses three scale features from multiple scale features to achieve feature complementarity at different scales, fully utilizing the feature information of each scale feature. Finally, the report generation network decodes multiple target fusion features to obtain the target medical report. This method avoids the problem in existing technologies where only a single feature extraction is performed on the medical image, failing to fully obtain the feature information of the medical image, thus obtaining a high-quality target medical report. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a medical report generation method based on multi-branch feature fusion provided in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the structure of a generative model provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of a first output head provided in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of another first output head provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of a second output head provided in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of another second output head provided in an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the structure of a fusion module provided in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of a medical report generation device based on multi-branch feature fusion, provided as an embodiment of the present invention. Detailed Implementation

[0052] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0054] In recent years, with the rapid development of artificial intelligence and big data technologies, these technologies have been widely applied in the medical field, especially medical report generation technology, which has gradually become an important component of intelligent assisted treatment. Current technologies can automatically analyze medical data and generate medical reports using deep learning-based methods. However, existing deep learning-based medical report generation models only perform single-step feature extraction from medical images, resulting in a limited range of feature dimensions and thus failing to generate high-quality medical reports.

[0055] Therefore, this invention provides a medical report generation method based on multi-branch feature fusion. By acquiring medical images and inputting them into a trained generation model, a target medical report is obtained. The generation model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network extracts different scale features from the medical image to obtain rich feature information of the target object contained in the medical image. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. The fusion module fuses three scale features from multiple scale features to achieve feature complementarity at different scales, fully utilizing the feature information of each scale feature. Finally, the report generation network decodes multiple target fusion features to obtain the target medical report. Based on this, it avoids the problem in existing technologies where only a single feature extraction is performed on the medical image, failing to fully obtain the feature information of the medical image, thus obtaining a high-quality target medical report.

[0056] In one embodiment, such as Figure 1 As shown, Figure 1 A flowchart illustrating a medical report generation method based on multi-branch feature fusion, provided in this embodiment of the invention, specifically includes the following steps:

[0057] S10: Acquire medical imaging images.

[0058] Among them, medical imaging images refer to images that can observe the internal structure of the user's body. These images can be grayscale images formed by using density differences through X-rays, or real-time dynamic images generated by sound wave reflection, but are not limited to these. This invention does not impose specific limitations, and those skilled in the art can set them according to the actual situation.

[0059] S11: Input the medical image into the trained generative model to obtain the target medical report of the medical image.

[0060] Figure 2 This is a schematic diagram of the structure of a generative model provided in an embodiment of the present invention, with reference to... Figure 2 As shown, the generative model includes: a multi-branch feature extraction network 21, a multi-head attention fusion network 22, and a report generation network 23. The multi-branch feature extraction network 21 consists of two improved EfficientNet subnetworks and two improved EfficientDet subnetworks interacting with each other, with each subnetwork used to extract features at different scales.

[0061] Optionally, based on the above embodiments, in some embodiments of the present invention, since the EfficientNet subnetwork is a classification network used to classify target objects, in order to obtain scale features of medical images by improving the EfficientNet subnetwork and reducing the computational complexity of the model, the first output head of the EfficientNet network is improved to determine the improved EfficientNet subnetwork. Figure 3 This is a schematic diagram of a first output head provided in an embodiment of the present invention. The first output head includes a convolutional layer (conv), a pooling layer (Pooling), and a fully connected layer (FC). (Refer to...) Figure 4 As shown, one way to improve the EfficientNet subnetwork is to replace the pooling layer in the first output head with a fully connected layer (FC) to obtain the improved EfficientNet subnetwork.

[0062] Optionally, based on the above embodiments, in some embodiments of the present invention, since the EfficientDet network is used to detect target objects, in order to obtain the scale features of medical images by improving the EfficientDet sub-network and reduce the computational complexity of the model, the second output head of the EfficientDet network is improved to determine the improved EfficientDet sub-network. Figure 5 This is a schematic diagram of a second output head provided in an embodiment of the present invention. The second output head includes a first convolutional module 51, a second convolutional module 52, and a fully connected layer 53 connected in parallel. The second convolutional module includes a first convolutional layer (conv) and a second convolutional layer (conv). (Refer to...) Figure 6 As shown, one way to implement the improved EfficientDet subnetwork is to replace the second convolutional layer with four fully connected layers to obtain the improved EfficientDet subnetwork.

[0063] It should be noted that the improved EfficientNet subnetwork has a lightweight architecture and can extract shallow and deep semantic features of medical images by using compound scaling technology, thereby accurately obtaining the texture structure information of target objects contained in medical images. The improved EfficientDet subnetwork can accurately capture and locate abnormal regions in medical images based on bidirectional feature pyramids and weighted bidirectional feature fusion mechanisms, achieving high sensitivity detection of abnormal target objects, thereby accurately obtaining more rich feature information.

[0064] The aforementioned multi-head attention fusion network 22 includes a channel attention fusion module and a spatial attention fusion module. Each fusion module is used to fuse three scale features from multiple scale features to obtain the target fusion feature.

[0065] The aforementioned report generation network 23 is used to decode multiple target fusion features to obtain the target medical report corresponding to the medical image.

[0066] Specifically, the system acquires the user's medical images, inputs the acquired medical images into a trained generative model, and obtains the target medical report from the medical images.

[0067] Optionally, based on the above embodiments, in some embodiments of the present invention, reference continues to be made to... Figure 2 As shown, one possible implementation of S11 is:

[0068] S111: Input the medical image into the multi-branch feature extraction network, and extract features from the medical image through various improved EfficientNet sub-networks and various improved EfficientDet sub-networks to obtain features at different scales.

[0069] Specifically, after obtaining the medical image, the image is input into a multi-branch feature extraction network. The improved EfficientNet subnetwork and the improved EfficientDet subnetwork, which are interconnected by the multi-branch feature extraction network, perform feature extraction on the medical image to extract different scale features corresponding to the medical image.

[0070] S112: Input multiple scale features into a multi-head attention fusion network, and fuse three scale features from the multiple scale features through various fusion modules to obtain the target fusion features corresponding to each fusion module.

[0071] Specifically, after obtaining multiple scale features, the multiple scale features are input into a multi-head attention fusion network. The multi-head attention fusion network is used to fuse three scale features from the multiple scale features to obtain the target fusion features corresponding to each fusion module.

[0072] S113: Input multiple target fusion features into the report generation network, and obtain the target medical report by aligning and decoding the multiple target fusion features through the report generation network.

[0073] In generative model applications, alignment refers to the information concatenation process performed on the fusion features of multiple input targets. During training, it refers to the information concatenation process performed on the fusion features of multiple input targets and the textual features of the pre-defined medical report text corresponding to the training medical image.

[0074] Specifically, after obtaining multiple target fusion features, the obtained multiple target fusion features are input into the report generation network. The report generation network performs alignment and decoding processing on the multiple target fusion features to obtain the target medical report corresponding to the medical image.

[0075] Thus, the medical report generation method based on multi-branch feature fusion provided in this embodiment acquires medical images and inputs them into a trained generation model to obtain the target medical report. The generation model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network extracts different scale features from the medical image to obtain rich feature information of the target object contained in the medical image. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. The fusion module fuses three scale features from multiple scale features to achieve feature complementarity at different scales, fully utilizing the feature information of each scale feature. Finally, the report generation network decodes multiple target fusion features to obtain the target medical report. Based on this, it avoids the problem in existing technologies where only a single feature extraction is performed on the medical image, failing to fully obtain the feature information of the medical image, thereby obtaining a high-quality target medical report.

[0076] Optionally, based on the above embodiments, in some embodiments of the present invention, the method further includes the following before performing S11:

[0077] S01: Obtain the sample training set.

[0078] The training set includes multiple sets of training samples. Each set includes a training medical image, a corresponding preset medical report text, and location labels for target objects within the training medical image. Specifically, the location labels for target objects refer to the bounding box position parameters of the target objects within the training medical image. For example, this training set may be a publicly available dataset such as the MIMIC-CXR dataset. However, it is not limited to this; the present invention is not specifically limited, and those skilled in the art can set it according to actual circumstances.

[0079] S02: Input the sample training set into the initial generative model, and adjust the model's weight parameters according to the loss function until the model converges, thus obtaining the trained generative model.

[0080] The loss function can be an existing loss function such as the cross-entropy loss function or the mean squared error loss function, but it is not limited to these. This invention does not impose specific limitations, and those skilled in the art can set it according to the actual situation.

[0081] Specifically, a training set is obtained. The training set includes multiple sets of training samples. Each set includes training medical images, corresponding pre-defined medical report texts, and location labels of target objects within the training medical images. For the initial generative model, the training set is used to train it. The training set is input into the initial generative model, and during training, the model's weight parameters are adjusted according to the loss function until the model converges, thus obtaining the trained generative model.

[0082] It should be noted that during the initial generative model training process, different scale features are extracted through two improved EfficientNet sub-networks and two improved EfficientDet sub-networks, respectively. Three scale features from multiple scale features are fused through the channel attention fusion module and the spatial attention fusion module to obtain multiple target fusion features. The report generation network aligns the multiple target fusion features and the preset medical report text. The predicted medical report is obtained based on the aligned fusion features after alignment. The weight parameters of the model are adjusted according to the loss function until the model converges, thus obtaining the trained generative model.

[0083] Optionally, during the training of the initial generative model, a preset number of iterations can be set, and training can end when the preset number of iterations is reached, thereby obtaining the trained generative model.

[0084] Optionally, based on the above embodiments, in some embodiments of the present invention, reference is made to... Figure 2 As shown, one possible implementation of S111 is:

[0085] S1111: Input the medical image into the first improved EfficientNet subnetwork to extract category features and obtain the first category scale features.

[0086] Among them, the category feature is used to describe the characteristics of the abnormal attribute category of the target object contained in the medical image. For example, if the target object contained in the medical image is the lung, the abnormal attribute category of the lung can be, for example, shadow, but is not limited thereto. The present invention does not specifically limit it, and those skilled in the art can set it according to the actual situation.

[0087] Specifically, the obtained medical image is input into the first improved EfficientNet sub-network, and the first improved EfficientNet sub-network performs category feature extraction processing on the medical image to obtain the first category scale feature corresponding to the medical image.

[0088] S1112: Input the medical image into the first improved EfficientDet sub-network to extract anchor point features and obtain the first anchor point scale features.

[0089] Among them, anchor point features are used to describe the features of the bounding box position of the target object contained in the medical image.

[0090] Specifically, the obtained medical image is input into the first improved EfficientDet sub-network, and the anchor point feature extraction processing is performed on the medical image through the first improved EfficientDet sub-network to obtain the first anchor point scale feature corresponding to the medical image.

[0091] S1113: Input the medical image into the second improved EfficientNet subnetwork to extract category features and obtain the second category scale features.

[0092] Specifically, the obtained medical image is input into the second improved EfficientNet sub-network, and the second improved EfficientNet sub-network performs category feature extraction processing on the medical image to obtain the second category scale feature corresponding to the medical image.

[0093] S1114: Input the medical image into the second improved EfficientDet subnetwork to extract anchor point features and obtain the second anchor point scale features.

[0094] Specifically, the obtained medical image is input into the second improved EfficientDet sub-network, and the anchor point feature extraction process is performed on the medical image through the second improved EfficientDet sub-network to obtain the second anchor point scale feature corresponding to the medical image.

[0095] Thus, this embodiment, by improving the EfficientNet subnetwork, can extract both shallow and deep semantic features from medical images, accurately obtaining the texture structure information of target objects contained within the medical images. By improving the EfficientDet subnetwork, it can precisely capture and locate lesion regions in medical images, achieving high-sensitivity detection of abnormal target objects and accurately obtaining richer feature information. Furthermore, it reduces the computational complexity of the generative model and improves the efficiency of obtaining target medical reports.

[0096] Optionally, based on the above embodiments, in some embodiments of the present invention, reference continues to be made to... Figure 2 As shown, one possible implementation of S112 is:

[0097] S1121: Input the first category scale feature, the first anchor point scale feature and the second category scale feature into the channel attention fusion module for fusion to obtain the first target fusion feature corresponding to the channel attention fusion module.

[0098] Specifically, the obtained first category scale features, first anchor point scale features, and second category scale features are input into the channel attention fusion module. The channel attention fusion module performs fusion processing on the first category scale features, first anchor point scale features, and second category scale features to obtain the first target fusion feature corresponding to the channel attention fusion module.

[0099] S1122: Input the first anchor point scale feature, the second category scale feature, and the second anchor point scale feature into the spatial attention fusion module for fusion to obtain the second target fusion feature corresponding to the spatial attention fusion module.

[0100] Specifically, the obtained first anchor point scale features, second category scale features, and second anchor point scale features are input into the spatial attention fusion module. The spatial attention fusion module performs fusion processing on the first anchor point scale features, second category scale features, and second anchor point scale features to obtain the second target fusion feature corresponding to the spatial attention fusion module.

[0101] It should be noted that the channel attention fusion module and the spatial attention fusion module have the same network structure. Based on this, refer to Figure 7 As shown, the channel attention fusion module includes a multi-head attention mechanism unit 71 and an affine transformation unit 72. The spatial attention fusion module includes a multi-head attention mechanism unit 71 and an affine transformation unit 72.

[0102] Thus, this embodiment can fuse the first category scale features, the first anchor point scale features, and the second category scale features through the channel attention fusion module to obtain the first target fusion features corresponding to the channels, thereby focusing more on the feature information between the channels. The spatial attention fusion module can fuse the first anchor point scale features, the second category scale features, and the second anchor point scale features to obtain the second target fusion features corresponding to the space, thereby focusing more on the feature information in the space, thereby improving the quality of the target medical report obtained.

[0103] Optionally, based on the above embodiments, in some embodiments of the present invention, reference continues to be made to... Figure 2 As shown, the report generation network 23 includes an alignment module, multiple deep feature extraction modules, and a decoding module. Based on this, one implementation of S113 could be:

[0104] S1131: Input the first target fusion feature and the second target fusion feature into the alignment module for alignment processing to obtain the target alignment fusion feature.

[0105] S1132: Input the target alignment and fusion features into multiple deep feature extraction modules to extract deep features and obtain target deep features.

[0106] S1133: Input the target depth features into the decoding module for decoding processing to obtain the target medical report.

[0107] Specifically, after obtaining the first target fusion feature and the second target fusion feature, the first target fusion feature and the second target fusion feature are input into the alignment module. The alignment module performs alignment processing on the first target fusion feature and the second target fusion feature to obtain the target alignment fusion feature. The target alignment fusion feature is then input into multiple deep feature extraction modules. The multiple deep feature extraction modules perform deep feature extraction on the target alignment fusion feature to obtain the target deep feature. Finally, the target deep feature is input into the decoding module. The decoding module performs decoding processing on the target deep feature to obtain the target medical report.

[0108] Thus, this embodiment can utilize feature information fused from multiple different scales to obtain the target medical report through a report generation network, thereby improving the quality of the obtained target medical report.

[0109] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0110] In one embodiment, such as Figure 8 As shown, a medical report generation device based on multi-branch feature fusion is provided, including: an acquisition module 10 and a generation module 11.

[0111] The acquisition module 10 is used to acquire medical imaging images.

[0112] The generation module 11 is used to input medical images into the trained generation model to obtain the target medical report of the medical images.

[0113] The generative model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network is used to extract features at different scales. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. Each fusion module is used to fuse three scale features from multiple scale features to obtain the target fusion feature. The report generation network is used to decode multiple target fusion features to obtain the target medical report corresponding to the medical image.

[0114] In the above embodiments, a medical image is acquired by an acquisition module, and a generation module inputs the medical image into a trained generation model to obtain the target medical report. The generation model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network extracts different scale features from the medical image to obtain rich feature information of the target object contained in the medical image. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. The fusion module fuses three scale features from multiple scale features to achieve feature complementarity at different scales, fully utilizing the feature information of each scale feature. Finally, the report generation network decodes multiple target fusion features to obtain the target medical report. Based on this, the problem of existing technologies that only perform single feature extraction on the medical image and cannot fully obtain the feature information of the medical image is avoided, thus obtaining a high-quality target medical report.

[0115] Specific limitations regarding the medical report generation device based on multi-branch feature fusion can be found in the limitations of the medical report generation method based on multi-branch feature fusion described above, and will not be repeated here. Each module in the aforementioned server can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0116] This invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the medical report generation method based on multi-branch feature fusion provided in this invention. For example, when the processor executes the computer program, it can implement... Figures 1 to 7 The technical solutions of any of the method embodiments shown are similar in implementation principle and technical effect, and will not be described again here.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM), etc.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A medical report generation method based on multi-branch feature fusion, characterized in that, include: Acquire medical imaging images; The medical image is input into the trained generative model to obtain the target medical report of the medical image; The generative model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network is used to extract features at different scales. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. Each fusion module is used to fuse three scale features from multiple scale features to obtain target fusion features. The report generation network is used to decode multiple target fusion features to obtain the target medical report corresponding to the medical image. The step of inputting the medical image into the trained generative model to obtain the target medical report of the medical image includes: The medical image is input into the multi-branch feature extraction network. Features are extracted from the medical image through various improved EfficientNet sub-networks and various improved EfficientDet sub-networks to obtain features at different scales. The improved EfficientNet sub-network is determined by improving upon the first output head of the EfficientNet network. The first output head includes convolutional layers, pooling layers, and fully connected layers. Improving the EfficientNet sub-network based on the first output head of the EfficientNet network includes replacing the pooling layers in the first output head with fully connected layers. The improved EfficientDet sub-network is determined by improving upon the second output head of the EfficientDet network. The second output head includes a first convolutional module, a second convolutional module, and fully connected layers connected in parallel. The second convolutional module includes a first convolutional layer and a second convolutional layer. Improving the EfficientDet sub-network based on the second output head of the EfficientDet network includes replacing the second convolutional layer with four fully connected layers. Multiple scale features are input into the multi-head attention fusion network, and three scale features from the multiple scale features are fused through each fusion module to obtain the target fusion feature corresponding to each fusion module; Multiple target fusion features are input into the report generation network, and the target medical report is obtained through the alignment and decoding of the multiple target fusion features by the report generation network. The process involves inputting the medical image into the multi-branch feature extraction network, and extracting features from the medical image through various improved EfficientNet sub-networks and various improved EfficientDet sub-networks to obtain features at different scales, including: The medical image is input into the first improved EfficientNet subnetwork for category feature extraction to obtain the first category scale feature; The medical image is input into the first improved EfficientDet sub-network for anchor point feature extraction to obtain the first anchor point scale feature. The medical image is input into the second improved EfficientNet subnetwork for category feature extraction to obtain the second category scale feature; The medical image is input into the second improved EfficientDet subnetwork for anchor point feature extraction to obtain the second anchor point scale feature.

2. The method of claim 1, wherein, Before inputting the medical image into the trained generative model to obtain the target medical report of the medical image, the method further includes: Obtain a sample training set, wherein the sample training set includes: multiple sets of training samples, each set of training samples including training medical image, preset medical report text corresponding to the training medical image, and location labels of target objects contained in the training medical image; The sample training set is input into the initial generative model, and the weight parameters of the model are adjusted according to the loss function until the model converges, thus obtaining the trained generative model.

3. The method of claim 2, wherein, The process involves inputting multiple scale features into the multi-head attention fusion network, fusing three scale features from the multiple scale features through various fusion modules, and obtaining the target fusion feature corresponding to each fusion module, including: The first category scale feature, the first anchor point scale feature, and the second category scale feature are input into the channel attention fusion module for fusion to obtain the first target fusion feature corresponding to the channel attention fusion module. The first anchor point scale feature, the second category scale feature, and the second anchor point scale feature are input into the spatial attention fusion module for fusion to obtain the second target fusion feature corresponding to the spatial attention fusion module.

4. The method of claim 3, wherein, The channel attention fusion module includes: a multi-head attention mechanism unit and an affine transformation unit; The spatial attention fusion module includes a multi-head attention mechanism unit and an affine transformation unit.

5. The method of claim 3, wherein, The report generation network includes an alignment module, multiple deep feature extraction modules, and a decoding module. The process of inputting multiple target fusion features into the report generation network, and obtaining the target medical report through alignment and decoding of the multiple target fusion features by the report generation network, includes: The first target fusion feature and the second target fusion feature are input into the alignment module for alignment processing to obtain the target alignment fusion feature; The target alignment and fusion features are input into multiple deep feature extraction modules to extract deep features and obtain the target's deep features. The target depth features are input into the decoding module for decoding processing to obtain the target medical report. 6.A medical report generation device based on multi-branch feature fusion, characterized in that, include: The acquisition module is used to acquire medical imaging images; The generation module is used to input the medical image into the trained generation model to obtain the target medical report of the medical image; The generative model includes a multi-branch feature extraction network, a multi-head attention fusion network, and a report generation network. The multi-branch feature extraction network consists of two improved EfficientNet sub-networks and two improved EfficientDet sub-networks interacting with each other. Each sub-network is used to extract features at different scales. The multi-head attention fusion network includes a channel attention fusion module and a spatial attention fusion module. Each fusion module is used to fuse three scale features from multiple scale features to obtain target fusion features. The report generation network is used to decode multiple target fusion features to obtain the target medical report corresponding to the medical image. The generation module is specifically used to input the medical image into the multi-branch feature extraction network, and to extract features from the medical image through various improved EfficientNet sub-networks and various improved EfficientDet sub-networks to obtain features at different scales. The improved EfficientNet sub-networks are determined by improving upon the first output head of the EfficientNet network. The first output head includes convolutional layers, pooling layers, and fully connected layers. The improved EfficientNet sub-networks are determined by improving upon the first output head of the EfficientNet network, including using fully connected layers... The improved EfficientNet subnetwork is obtained by replacing the pooling layer in the first output head with a layer; the improved EfficientDet subnetwork is determined by improving the second output head of the EfficientDet network; the second output head includes a first convolutional module, a second convolutional module, and a fully connected layer connected in parallel; the second convolutional module includes a first convolutional layer and a second convolutional layer; the improved EfficientDet subnetwork is determined by improving the second output head of the EfficientDet network, including replacing the second convolutional layer with four fully connected layers. Multiple scale features are input into the multi-head attention fusion network, and three scale features from the multiple scale features are fused through each fusion module to obtain the target fusion feature corresponding to each fusion module; Multiple target fusion features are input into the report generation network, and the target medical report is obtained through the alignment and decoding of the multiple target fusion features by the report generation network. The generation module is further used to input the medical image into the first improved EfficientNet sub-network for category feature extraction and to obtain the first category scale feature; The medical image is input into the first improved EfficientDet sub-network for anchor point feature extraction to obtain the first anchor point scale feature. The medical image is input into the second improved EfficientNet subnetwork for category feature extraction to obtain the second category scale feature; The medical image is input into the second improved EfficientDet subnetwork for anchor point feature extraction to obtain the second anchor point scale feature. 7.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, When the processor executes the computer program, it implements the steps of the medical report generation method based on multi-branch feature fusion as described in any one of claims 1 to 5.

Images