Multi-branch feature fusion-based intelligent generation method and system for endoscopic examination reports

Abstract

The present invention provides a method and system for intelligently generating an endoscopic inspection report with multi-branch feature fusion, and relates to the field of image processing technology. The present invention first acquires and preprocesses the endoscopic data for which a report is to be generated, and acquires an endoscopic report dictionary; then extracts the scene-level features, category-level features, and region-level features of the endoscopic data from three branches: scene-level, category-level, and region-level; finally, fuses the scene-level features, category-level features, and region-level features, and combines them with the endoscopic report dictionary to generate an endoscopic inspection report. In an embodiment of the present invention, the endoscopic inspection report generation task is divided into three branches, and corresponding artificial intelligence methods are used from different dimensions of the endoscopic data to realize feature extraction, and then feature fusion is performed to realize endoscopic inspection report generation. Compared with existing methods, the endoscopic inspection report generated by the present invention is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for intelligently generating an endoscopic inspection report by fusion of multiple branches of features. Background Art

[0002] Endoscopy allows direct observation of the true condition of the examined area and further clarifies the diagnosis by performing biopsy and cytological examinations on suspected lesions. It is the preferred examination method for upper gastrointestinal tract lesions. Currently, intelligent medical image analysis has been introduced to reduce the workload of medical staff and improve the diagnostic and treatment capabilities of imaging departments at primary healthcare institutions.

[0003] For intelligent analysis of endoscopic images, the existing method generally preprocesses the endoscopic inspection image to obtain the visual features and part feature vectors of the endoscopic inspection image; fuses the visual features and part feature vectors to obtain the overall feature vector of the endoscopic inspection image; inputs the overall feature vector into the encoder of the Transformer model to obtain the hidden layer state value after encoding the overall feature vector; inputs the hidden layer state value into the decoder of the Transformer model to obtain the target text sequence vector corresponding to the endoscopic inspection image, and the decoder introduces a position pattern memory, which records the key pattern information corresponding to the part feature vector; obtains the endoscopic inspection report based on the target text sequence vector and a pre-constructed report dictionary.

[0004] However, this method only generates reports based on the extraction of the overall feature vector of the image, resulting in low accuracy of the generated reports. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a method and system for intelligently generating endoscopic inspection reports with multi-branch feature fusion, which solves the technical problem of low accuracy of reports generated by existing intelligent analysis of endoscopic images.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention provides a method for intelligently generating an endoscopic inspection report using multi-branch feature fusion, the method comprising:

[0010] S1. Acquire and pre-process endoscopic data for generating a report, and obtain an endoscopic report dictionary;

[0011] S2. Extracting scene-level features, category-level features, and region-level features of the endoscope data from three branches: scene-level, category-level, and region-level;

[0012] S3. Fusion of the scene-level features, category-level features, and region-level features, combined with the endoscopy report dictionary, to generate an endoscopy inspection report.

[0013] Preferably, the extracting scene-level features, category-level features and region-level features of the endoscope data from the three branches of scene level, category level and region level includes:

[0014] S201, inputting the pre-processed endoscope data into a pre-trained scene-level model to obtain scene-level features;

[0015] S202, inputting the pre-processed endoscope data into a pre-trained category-level model to obtain category-level features;

[0016] S203: Input the pre-processed endoscope data into a pre-trained region-level model to obtain region-level features.

[0017] Preferably, the process of constructing the pre-trained scene-level model includes:

[0018] The endoscope data in the ImageNet dataset is used to train the ResNet101 network to obtain a scene-level model.

[0019] Preferably, the process of constructing the pre-trained category-level model includes:

[0020] ResNext101, DenseNet101 and Vgg16 were used to perform classification training on endoscopic image data to obtain category-level models of endoscopic parts.

[0021] Preferably, the process of constructing the pre-trained regional level model includes:

[0022] The Faster-RCNN target detection network is used to perform target detection training on the EDD2020 dataset to obtain a region-level model.

[0023] Preferably, the pre-processed endoscope data is input into a pre-trained region-level model to obtain region-level features, including:

[0024] The pre-processed endoscope data is input into the pre-trained regional level model to obtain regional level features. Among them, k represents the number of selected key regions, and the final region-level feature R is obtained through splicing and linear operations.

[0025] Preferably, the fusing of the scene-level features, category-level features and region-level features, combined with the endoscopy report dictionary, to generate an endoscopy report includes:

[0026] S301. By fusing scene-level features, category-level features, and region-level features of different branches, an image vector representation that fuses overall scene-level features, category-level features, and key region-level features is obtained.

[0027] S302. Input the features obtained by the fusion in the previous stage into the Transformer network structure, encode the image features into vectors through the Encoder encoding layer, combine them with the endoscopy report dictionary, decode the encoded features into text output through the decoding layer, linear layer and Softmax function, use the text output last time as the input of the next stage, and repeat the process of outputting one text each time to obtain the endoscopy inspection report.

[0028] In a second aspect, the present invention provides an intelligent generation system for endoscopic inspection reports with multi-branch feature fusion, the system comprising:

[0029] a data acquisition module for acquiring and preprocessing endoscopic data for generating reports, and for acquiring an endoscopic report dictionary;

[0030] a feature extraction module, configured to extract scene-level features, category-level features, and region-level features of the endoscope data from three branches: scene-level, category-level, and region-level;

[0031] A report generation module is used to fuse the scene-level features, category-level features and region-level features, and combine them with the endoscope report dictionary to generate an endoscope inspection report.

[0032] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program for intelligently generating an endoscopic inspection report with multi-branch feature fusion, wherein the computer program enables a computer to execute the method for intelligently generating an endoscopic inspection report with multi-branch feature fusion as described above.

[0033] In a third aspect, the present invention provides an electronic device, comprising:

[0034] one or more processors;

[0035] Memory; and

[0036] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a method for intelligently generating an endoscopic inspection report for performing multi-branch feature fusion as described above.

[0037] (3) Beneficial effects

[0038] The present invention provides a method and system for intelligently generating endoscopic inspection reports using multi-branch feature fusion. Compared with existing technologies, it has the following advantages:

[0039] The present invention first obtains and preprocesses the endoscopic data for which a report is to be generated, and obtains an endoscopic report dictionary; then, scene-level features, category-level features, and region-level features of the endoscopic data are extracted from three branches: scene-level, category-level, and region-level; finally, the scene-level features, category-level features, and region-level features are fused, and combined with the endoscopic report dictionary to generate an endoscopic inspection report. The endoscopic inspection report generation task of the embodiment of the present invention is divided into three branches, and corresponding artificial intelligence methods are used from different dimensions of the endoscopic data to realize feature extraction, and then feature fusion is performed to realize endoscopic inspection report generation. Compared with existing methods, the endoscopic inspection report generated by the present invention is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a block diagram of a method for intelligently generating an endoscopic inspection report using multi-branch feature fusion according to an embodiment of the present invention;

[0042] Figure 2 A structural network diagram between various models in an embodiment of the present invention;

[0043] Figure 3 Renderings of endoscopic inspection reports generated by the embodiments of the present invention and the comparative method. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0045] The embodiments of the present application solve the technical problem of low accuracy of diagnostic results provided by existing intelligent analysis of endoscopic images by providing a method and system for intelligent generation of endoscopic examination reports with multi-branch feature fusion, and realize the extraction of endoscopic features from features of different dimensions to generate multi-dimensional and accurate endoscopic examination reports.

[0046] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0047] Compared with natural images, endoscopic images are highly similar to each other. Doctors are very likely to ignore some lesion information during the diagnosis and treatment decision-making process, resulting in missed diagnosis or misdiagnosis. At the same time, the key area information in the image is an important part that cannot be ignored in the report writing process of doctors. Therefore, considering the high complexity of endoscopic images themselves, the embodiment of the present invention starts from the characteristics of endoscopic data itself and designs a multi-branch feature fusion intelligent generation method for endoscopic examination reports to improve the accuracy of endoscopic examination reports, meet the data characteristics and actual diagnosis and treatment needs of clinical endoscopic image report generation, and automatically generate endoscopic examination reports. This can reduce the workload of endoscopists in writing endoscopic examination reports while improving the safety of the diagnosis and treatment process, alleviate the problem of lack of experience of endoscopists in remote areas under tiered diagnosis and treatment, and assist doctors in clinical diagnosis.

[0048] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0049] The embodiment of the present invention provides a multi-branch feature fusion endoscopic inspection report intelligent generation method, such as Figure 1 Shown, including:

[0050] S1. Acquire and pre-process endoscopic data for generating a report, and obtain an endoscopic report dictionary;

[0051] S2. Extracting scene-level features, category-level features, and region-level features of the endoscope data from three branches: scene-level, category-level, and region-level;

[0052] S3. Fusion of the scene-level features, category-level features, and region-level features, combined with the endoscopy report dictionary, to generate an endoscopy inspection report.

[0053] The endoscopic inspection report generation task of the embodiment of the present invention is divided into three branches. Corresponding artificial intelligence methods are used from different dimensions of endoscopic data to realize feature extraction, and then feature fusion is performed to realize endoscopic inspection report generation. Compared with the existing methods, the endoscopic inspection report generated by the embodiment of the present invention is more accurate.

[0054] The following is a detailed description of each step:

[0055] like Figure 2 It is a structural network diagram between various models in the embodiment of the present invention.

[0056] In step S1, the endoscopic data for which a report is to be generated is acquired and preprocessed to obtain an endoscopic report dictionary. The specific implementation process is as follows:

[0057] Endoscopic data includes endoscopic images and text data pairs, and preprocessing includes data cleaning and data classification.

[0058] Based on existing endoscopy clinical work, a dictionary that meets the norms of the word expression domain used by doctors in writing reports will be constructed. This dictionary includes verbs, positional nouns, positional adjectives, and lesion area descriptors. After screening, an endoscopy report dictionary containing 448 words was obtained.

[0059] In step S2, scene-level features, category-level features, and region-level features of the endoscope data are extracted from the three branches of scene-level, category-level, and region-level. The specific implementation process is as follows:

[0060] It should be noted that in the embodiment of the present invention, the above scene-level features, category-level features and region-level features are obtained through pre-trained scene-level models, category-level models and region-level models respectively. The training process of the above three models is as follows:

[0061] We obtained 8,713 endoscopic image and text data pairs from the gastroenterology department of a large tertiary hospital. After data cleaning, we obtained 7,644 image-text pairs. We then classified the images into six categories: cardia, esophagus, gastric angle, gastric fundus and body, antrum and pylorus, and duodenum. We also obtained 386 endoscopic data sets from the Endoscopy Disease Detection Challenge 2020 and used data augmentation algorithms such as random rotation to expand the dataset to 1,158 images. Key region labels in this dataset fall into the following five categories: Barrett's esophagus, suspicious area, high-grade dysplasia, cancer, and polyp.

[0062] The endoscope data in the ImageNet dataset is used to train the ResNet101 network to obtain a scene-level model.

[0063] Specifically, it includes: using the public dataset ImageNet to train the ResNet101 classification network to implement 20 classification tasks of natural images and obtain the corresponding scene-level model.

[0064] ResNext101, DenseNet101, and Vgg16 were used to conduct classification experiments on 7644 endoscopic images, and a category-level model of endoscopic parts was obtained. Specifically,

[0065] Using the 7,644 collected image-text pairs, we trained three classification networks, ResNext101, DenseNet101, and Vgg16, on a six-category classification task, generating category-level models highly relevant to endoscopic sites. Using different networks improves the reliability and flexibility of this method, enabling effective results with varying network models.

[0066] We use the Faster-RCNN object detection network to train object detection on the EDD2020 dataset and obtain a region-level model. This includes:

[0067] The public dataset EDD2020 is used to train the Faster-RCNN positioning network. The example of the present invention takes into account the situation where the number of datasets is too small. Therefore, 386 endoscopic data and 1158 endoscopic data are pre-trained respectively to obtain the corresponding regional-level models, which can flexibly and effectively solve the problem of too little pre-training data.

[0068] S201: Input the pre-processed endoscope data into the pre-trained scene-level model ResNet101 to obtain scene-level features. Specifically, the process includes:

[0069] The endoscopic image data is input into the above-trained scene-level model ResNet101 to obtain a scene-level feature, which is recorded as in Represents block features, and feature S is composed of 7×7 block features of uniform size sorted horizontally.

[0070] S202: Input the pre-processed endoscope data into a pre-trained category-level model to obtain category-level features. Specifically, the process includes:

[0071] The pre-processed endoscope data is input into the pre-trained category-level model to obtain category-level features C is composed of 7×7 uniformly sized block features that are arranged horizontally.

[0072] S203: Input the pre-processed endoscope data into a pre-trained regional model to obtain regional features. Specifically, the process includes:

[0073] The pre-processed endoscope data is input into the pre-trained regional level model to obtain regional level features. Wherein k represents the number of key regions selected by the embodiment of the present invention, and can take values of 1, 2, 3, etc. Finally, the final region-level feature R is obtained through splicing and linear operations.

[0074] In step S3, the scene-level features, category-level features, and region-level features are integrated with the endoscopy report dictionary to generate an endoscopy report. The specific implementation process is as follows:

[0075] S301. By fusing the features S, C and R of different branches, an image vector representation that integrates the overall scene-level features, category-level features and key area-level features is obtained.

[0076] The beginning of the endoscopic inspection report in the embodiment of the present invention must be the location corresponding to the image. Therefore, the category-level feature is used as the core. First, the category-level feature C is multiplied with the scene-level feature S to obtain a fused feature F that fuses the scene level and category level. Then, the category-level feature C is multiplied with the key area-level feature R to obtain a fused feature M that fuses the category level and key area level. Finally, the two types of features F and M obtained above are multiplied to obtain the final endoscopic image feature representation N that fuses the three types of features. This method effectively reduces the amount of computation in the network and speeds up report generation.

[0077] F=S×C

[0078] M=R×C

[0079] N=F×M

[0080] S302. Input the features obtained by the fusion in the previous stage into the Transformer network structure, encode the image features into vectors through the Encoder encoding layer, combine them with the endoscopic report dictionary mentioned above, and then decode the encoded features into text output through the decoding layer, linear layer and Softmax function. The text output last time is used as the input of the next stage. The cycle is repeated by outputting one text each time to finally obtain the endoscopic examination report.

[0081] The Transformer network structure can fully learn the internal attention relationships between each vector, while also addressing the difficulty of LSTM and RNN in effectively learning long texts. Because endoscopy reports often contain large amounts of text and complex relationships between words, the Transformer structure is required to fully learn the relationships between vector features and improve adaptability to long texts.

[0082] In order to verify the effectiveness of the embodiment of the present invention, the R2GenCMN method in the prior art (this method comes from the 2021 top computer vision conference ACL (Annual Meeting of the Association for Computational Linguistics) paper Cross-modal Memory Networks for Radiology Report Generation) is compared with the embodiment of the present invention. The generated endoscopic examination report is as follows: Figure 3 As shown, from the content of the report, it can be seen that the report generated by the embodiment of the present invention is closer to the implementation situation.

[0083] An embodiment of the present invention provides an intelligent endoscopic inspection report generation system with multi-branch feature fusion, comprising:

[0084] a data acquisition module for acquiring and preprocessing endoscopic data for generating reports, and for acquiring an endoscopic report dictionary;

[0085] a feature extraction module, configured to extract scene-level features, category-level features, and region-level features of the endoscope data from three branches: scene-level, category-level, and region-level;

[0086] A report generation module is used to fuse the scene-level features, category-level features and region-level features, and combine them with the endoscope report dictionary to generate an endoscope inspection report.

[0087] It can be understood that the intelligent generation system for endoscopic inspection reports with multi-branch feature fusion provided in the embodiment of the present invention corresponds to the above-mentioned intelligent generation method for endoscopic inspection reports with multi-branch feature fusion. The explanation, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the intelligent generation method for endoscopic inspection reports with multi-branch feature fusion, and will not be repeated here.

[0088] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for intelligently generating endoscopic inspection reports with multi-branch feature fusion, wherein the computer program enables a computer to execute the method for intelligently generating endoscopic inspection reports with multi-branch feature fusion as described above.

[0089] An embodiment of the present invention further provides an electronic device, including:

[0090] one or more processors;

[0091] Memory; and

[0092] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a method for intelligently generating an endoscopic inspection report for performing multi-branch feature fusion as described above.

[0093] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0094] 1. The endoscopic examination report generation task of the embodiment of the present invention is divided into three branches. Corresponding artificial intelligence methods are used from different dimensions of endoscopic data to realize feature extraction, and then feature fusion is performed to realize endoscopic examination report generation. Compared with the existing methods, the endoscopic examination report generated by the embodiment of the present invention is more accurate. The embodiment of the present invention can meet the data characteristics and actual diagnosis and treatment needs of clinical endoscopic imaging report generation, automatically generate endoscopic examination reports, and reduce the workload of endoscopists in writing endoscopic examination reports while improving the safety of the diagnosis and treatment process. It can also alleviate the problem of lack of experience of endoscopists in remote areas under hierarchical diagnosis and treatment, and assist doctors in clinical diagnosis.

[0095] 2. This embodiment of the present invention uses the Faster-RCNN object detection network to effectively extract multiple key regions in endoscopic images and uses these key regions as information for report generation, increasing the probability of key regions appearing in the generated report. Furthermore, key region sorting can be customized. For images with a large number of key regions, multiple key region features can be selected, while for images with a small number of key regions, relatively few key region features can be selected. This approach flexibly and effectively avoids invalid and redundant feature input.

[0096] 3. The present invention does not require high-quality data annotation. The endoscope data used in the examples of the present invention has no additional annotation requirements beyond report text annotation (required) and category annotation, which reduces the workload of data annotation. When using this method to conduct experiments in other fields, it is only necessary to obtain other similar datasets for pre-training and then apply them to the proposed model, which improves the universality of the method.

[0097] 4. The embodiment of the present invention uses the Transformer structure for report generation. This structure can effectively learn the attention relationship between vectors and effectively improve the recognition of the relationship between words in the generated report.

[0098] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-branch feature fusion intelligent generation method for endoscopic inspection reports, characterized in that: The method comprises: S1. Acquire and pre-process endoscopic data for generating a report, and obtain an endoscopic report dictionary; S2. Extracting scene-level features, category-level features, and region-level features of the endoscopic data from three branches: scene-level, category-level, and region-level; wherein categories include cardia, esophagus, gastric angle, gastric fundus and body, gastric antrum and pylorus, and duodenum; and regions include Barrett's esophagus, suspicious area, high-grade dysplasia, cancer, and polyp; S3. Fusion of the scene-level features, category-level features, and region-level features, combined with the endoscopy report dictionary, to generate an endoscopy report, including: S301, by fusing scene-level features, category-level features, and region-level features of different branches, an image vector representation is obtained that integrates the overall scene-level features, category-level features, and key region-level features, including: Taking the category-level feature as the core, the category-level feature C is first multiplied with the scene-level feature S to obtain the fused feature F of the fused scene-level and category-level features. Then, the category-level feature C is multiplied with the key region-level feature R to obtain the fused feature M of the fused category-level and key region-level features. Finally, the two types of features F and M are multiplied to obtain the final endoscopic image feature representation N that integrates the three types of features. S302. Input the features obtained by the fusion in the previous stage into the Transformer network structure, encode the image features into vectors through the Encoder encoding layer, combine them with the endoscopy report dictionary, decode the encoded features into text output through the decoding layer, linear layer and Softmax function, use the text output last time as the input of the next stage, and repeat the process of outputting one text each time to obtain the endoscopy inspection report.

2. The method for intelligently generating an endoscopic inspection report by multi-branch feature fusion according to claim 1, characterized in that: The extracting of scene-level features, category-level features, and region-level features of the endoscope data from the three branches of scene level, category level, and region level includes: S201, inputting the pre-processed endoscope data into a pre-trained scene-level model to obtain scene-level features; S202, inputting the pre-processed endoscope data into a pre-trained category-level model to obtain category-level features; S203: Input the pre-processed endoscope data into a pre-trained region-level model to obtain region-level features.

3. The method for intelligently generating an endoscopic inspection report by multi-branch feature fusion according to claim 2, characterized in that: The construction process of the pre-trained scene-level model includes: The endoscope data in the ImageNet dataset is used to train the ResNet101 network to obtain a scene-level model.

4. The method for intelligently generating an endoscopic inspection report by multi-branch feature fusion according to claim 2, characterized in that: The construction process of the pre-trained category-level model includes: ResNext101, DenseNet101 and Vgg16 were used to perform classification training on endoscopic image data to obtain category-level models of endoscopic parts.

5. The method for intelligently generating an endoscopic inspection report by multi-branch feature fusion according to claim 2, characterized in that: The construction process of the pre-trained regional level model includes: The Faster-RCNN target detection network is used to perform target detection training on the EDD2020 dataset to obtain a region-level model.

6. The method for intelligently generating an endoscopic inspection report by multi-branch feature fusion according to claim 5, characterized in that: The pre-processed endoscope data is input into a pre-trained region-level model to obtain region-level features, including: The pre-processed endoscope data is input into the pre-trained regional level model to obtain regional level features. Among them, k represents the number of selected key regions, and the final region-level feature R is obtained through splicing and linear operations.

7. An intelligent generation system for endoscopic inspection reports based on multi-branch feature fusion, characterized in that: The system comprises: a data acquisition module for acquiring and preprocessing endoscopic data for generating reports, and for acquiring an endoscopic report dictionary; a feature extraction module for extracting scene-level features, category-level features, and region-level features of the endoscopic data from three branches: scene-level, category-level, and region-level; wherein the categories include the cardia, esophagus, gastric angle, gastric fundus and body, gastric antrum and pylorus, and duodenum; and the regions include Barrett's esophagus, suspicious area, high-grade dysplasia, cancer, and polyp; A report generation module is used to fuse the scene-level features, category-level features, and region-level features and combine them with the endoscope report dictionary to generate an endoscope inspection report, including: S301, by fusing scene-level features, category-level features, and region-level features of different branches, an image vector representation is obtained that integrates the overall scene-level features, category-level features, and key region-level features, including: Taking the category-level feature as the core, the category-level feature C is first multiplied with the scene-level feature S to obtain the fused feature F of the fused scene-level and category-level features. Then, the category-level feature C is multiplied with the key region-level feature R to obtain the fused feature M of the fused category-level and key region-level features. Finally, the two types of features F and M are multiplied to obtain the final endoscopic image feature representation N that integrates the three types of features. S302. Input the features obtained by the fusion in the previous stage into the Transformer network structure, encode the image features into vectors through the Encoder encoding layer, combine them with the endoscopy report dictionary, decode the encoded features into text output through the decoding layer, linear layer and Softmax function, use the text output last time as the input of the next stage, and repeat the process of outputting one text each time to obtain the endoscopy inspection report.

8. A computer-readable storage medium, characterized in that It stores a computer program for intelligent generation of endoscopic inspection reports using multi-branch feature fusion, wherein the computer program enables a computer to execute the method for intelligent generation of endoscopic inspection reports using multi-branch feature fusion as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for intelligently generating an endoscopic inspection report by performing multi-branch feature fusion as described in any one of claims 1 to 6.

Images