Method and system for automatic generation of medical reports based on capsule endoscopy

Abstract

The application discloses a medical report automatic generation method and system based on a capsule endoscope, comprising: acquiring a capsule endoscope video, and removing redundant frames of the capsule endoscope video to obtain a plurality of representative frames; inputting each representative frame into a preset detection model to determine an initial key frame and a classification result, and using the classification result to finely shrink the initial key frame to obtain a plurality of target key frames; inputting each target key frame into a preset multi-modal model to extract a corresponding first visual feature, and splicing the first visual feature with a text feature to obtain an initial text description; iteratively predicting the initial text description until a first iteration termination condition is met, and outputting a target text description; inputting the target text description into a large language model, using a self-recurrence method to gradually predict a first word segmentation sequence until a second iteration termination condition is met, and generating a corresponding structured medical report based on the second word segmentation sequence. The application can automatically generate a diagnosis report.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of report generation, in particular to a medical report automatic generation method and system based on a capsule endoscope. BACKGROUND

[0002] A capsule endoscope is an advanced medical device used for early diagnosis of gastrointestinal diseases. Patients only need to swallow a capsule with a camera, and they can obtain comprehensive images from the esophagus to the colon. Although capsule endoscopes have many advantages in disease diagnosis, recording the entire digestive tract requires continuous shooting of a large number of normal tissue images, which makes doctors spend a lot of time and effort to filter key information such as anatomical landmarks or lesion sites when analyzing, which not only increases the workload of doctors, but also may lead to the omission of some important information.

[0003] Currently, although machine learning has been used to analyze endoscopic videos, it cannot effectively detect potential lesion sites in the gastrointestinal tract, and it cannot provide detailed information such as size, location, and shape of the lesion. In addition, after the analysis is completed, it still needs to rely on doctors to manually integrate the analysis results, which not only consumes time and effort, but also may lead to incomplete or inaccurate information, thereby affecting the efficiency and quality of diagnosis. SUMMARY

[0004] The present application provides a medical report automatic generation method and system based on a capsule endoscope to automatically generate a diagnosis report and improve the efficiency and quality of diagnosis.

[0005] In a first aspect, the present application provides a medical report automatic generation method based on a capsule endoscope, comprising:

[0006] Obtaining a capsule endoscope video and removing redundant frames of the capsule endoscope video to obtain a plurality of representative frames;

[0007] Inputting each of the representative frames into a preset detection model to determine initial key frames and classification results, and using the classification results to refine the initial key frames to obtain a plurality of target key frames;

[0008] Inputting each of the target key frames into a preset multi-modal model to extract corresponding first visual features, and concatenating the first visual features with text features to obtain an initial text description, and iteratively predicting the initial text description until a first iteration termination condition is met, and outputting a target text description, wherein the text features are obtained by mapping the segmented preset prompt text to the multi-modal model.

[0009] The target text description is input into a large language model, and a first segmentation sequence is predicted step by step by using a self-recurrence method until a second iteration termination condition is met, and a corresponding structured medical report is generated based on the second segmentation sequence.

[0010] By eliminating the redundant frames of the capsule endoscopy video, the key information in the video can be retained, while avoiding repeated calculations on duplicate or meaningless frames, improving the speed and accuracy of subsequent processing. By detecting the initial key frames that may contain important information, the relevant frames can be obtained initially, facilitating subsequent analysis. The classification results can help distinguish different types of anatomical structures or lesions, ensuring that only the most relevant and representative key frames are retained. By refining the initial key frames, the number of key frames can be further reduced, minimizing the amount of data processed in subsequent steps while retaining the most important information. By concatenating the first visual features with the text features, visual and textual information can be combined to generate more comprehensive and accurate descriptions, enhancing the clinical value of the report. By iteratively predicting detailed text descriptions, the visual content of each key frame can be accurately translated into language information, avoiding human intervention and automatically generating high-quality text descriptions, improving the speed and consistency of report generation. By using a self-recurrence method to gradually build the report, the text descriptions of all key frames can be integrated into a coherent and structured medical report, ensuring that the report content is complete and logically clear. Doctors can directly use the generated report for diagnosis without the need for manual sorting and writing, saving time and effort. Compared with existing technologies, the present application can automatically generate diagnostic reports, improving the efficiency and quality of diagnosis.

[0011] Further, the redundant frames of the capsule endoscopy video are eliminated to obtain a plurality of representative frames, specifically:

[0012] The capsule endoscopy video is input into a preset analysis model to obtain second visual features corresponding to each frame of the capsule endoscopy video.

[0013] The similarity of adjacent frame second visual features is calculated, and a plurality of clusters are determined based on the comparison result of the similarity and a preset similarity threshold, and representative frames are determined based on each cluster.

[0014] By eliminating the redundant frames of the capsule endoscopy video, the key information in the video can be retained, while avoiding repeated calculations on duplicate or meaningless frames, improving the speed and accuracy of subsequent processing.

[0015] Further, the representative frames are input into a preset detection model to determine initial key frames and classification results, specifically:

[0016] input each of the representative frames into a preset detection model to determine a first probability and a second probability;

[0017] compare the first probability with a preset probability threshold, and if the first probability is greater than the preset probability threshold, determine an initial key frame;

[0018] determine a classification result based on the second probability.

[0019] In this way, the initial key frame that may contain important information is identified by the detection model, and the frame of relevance can be obtained preliminarily, which facilitates subsequent analysis. The classification result can help to distinguish different types of anatomical structures or lesions, and thus ensure that only the most relevant and representative key frame is retained.

[0020] Further, the inputting each of the representative frames into a preset detection model to determine a first probability and a second probability specifically comprises:

[0021] input each of the representative frames into a preset detection model to extract features of each of the representative frames, to obtain corresponding semantic features, wherein the detection model comprises a spatio-temporal feature extraction network, a key frame detection head, and a structure classification head;

[0022] fuse the semantic features and position encoding results to obtain fused features, and input the fused features into the spatio-temporal feature extraction network to obtain deep features, wherein the position encoding results are obtained by position encoding each of the representative frames;

[0023] input the deep features into the key frame detection head and the structure classification head respectively to obtain the first probability and the second probability.

[0024] In this way, the initial key frame that may contain important information and the classification result can be identified subsequently by determining the first probability and the second probability, and thus subsequent analysis is facilitated.

[0025] Further, the training process of the detection model specifically comprises:

[0026] extract features corresponding to the representative frames by using a preset feature encoder, and perform normalization processing on the features to obtain frame features;

[0027] calculate cosine similarity between frame features corresponding to each of the representative frames to obtain a similarity matrix, and based on the similarity matrix and a preset video segmentation ground truth, expand a pre-determined reference key frame to obtain an expanded key frame;

[0028] perform supervised training on the key frame detection network by using the expanded key frame and a preset structure segmentation label, until a loss function is minimized, to determine the detection model.

[0029] In this way, by training the detection model, the initial key frame and the classification result that may contain important information can be accurately identified, and subsequent analysis is facilitated.

[0030] Further, the classification result is used to refine the initial key frame to obtain a plurality of target key frames, specifically:

[0031] In each of the classification results, the initial key frame with the highest first probability is iteratively selected as a candidate key frame;

[0032] The feature similarity between the initial key frame with the same classification result as the candidate key frame and the candidate key frame is calculated, and the initial key frame with a feature similarity greater than a preset similarity threshold is removed. If the number of candidate key frames corresponding to each classification result reaches a preset number threshold or the first probability corresponding to each initial key frame is less than a preset probability threshold, all selected candidate key frames are determined as final target key frames.

[0033] In this way, by refining the initial key frame, the number of key frames can be further reduced to minimize the amount of data processed in subsequent steps while retaining the most important information.

[0034] Further, the initial text description is iteratively predicted until a first iteration termination condition is met, and a target text description is output, specifically:

[0035] Based on the initial text description, a probability distribution value of word segmentation is generated by a self-recurrence method, and a third word segmentation sequence is determined based on the probability distribution value.

[0036] Based on the second word segmentation sequence and the initial text description, a word segmentation sequence is continuously updated until the length of the fourth word segmentation sequence is greater than a preset length threshold, and then a target text description is determined based on the fourth word segmentation sequence.

[0037] In this way, by iteratively predicting to generate a detailed text description, it can be ensured that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention and automatically generating high-quality text descriptions.

[0038] Further, the target text description is input into a large language model, specifically:

[0039] The time point corresponding to each target key frame is obtained, and the time point and the target text description form a description pair.

[0040] The description pair is spliced with a preset text prompt to obtain a splicing result, and the splicing result is input into a large language model.

[0041] In this way, by inputting the target text description into the large language model, the text descriptions of all key frames can be accurately integrated into a coherent and structured medical report, ensuring that the report content is complete and logically clear. Doctors can directly use the generated report for diagnosis without the need for manual sorting and writing, saving time and effort.

[0042] In a second aspect, the application further provides a medical report automatic generation system based on a capsule endoscope, comprising an acquisition module, a processing module, a first prediction module and a second prediction module.

[0043] The acquisition module is configured to acquire a capsule endoscope video and eliminate redundant frames of the capsule endoscope video to obtain a plurality of representative frames.

[0044] The processing module is configured to input each of the representative frames into a preset detection model, determine initial key frames and classification results, and use the classification results to refine the initial key frames to obtain a plurality of target key frames.

[0045] The first prediction module is configured to input each of the target key frames into a preset multi-modal model, extract corresponding first visual features, and splice the first visual features with text features to obtain initial text descriptions. The initial text descriptions are iteratively predicted until a first iteration termination condition is met, and a target text description is output. The text features are obtained by mapping a preset prompt text after tokenization to the multi-modal model.

[0046] The second prediction module is configured to input the target text description into a large language model, use a self-recurrent method to gradually predict a first token sequence until a second iteration termination condition is met, and generate a corresponding structured medical report based on a second token sequence.

[0047] This application embodiment, by removing redundant frames from the capsule endoscopy video, can retain key information in the video while avoiding repeated calculations of duplicate or meaningless frames, thus improving the speed and accuracy of subsequent processing. By identifying initial keyframes that may contain important information through a detection model, relevant frames can be preliminarily obtained, facilitating subsequent analysis. Classification results help distinguish different types of anatomical structures or lesions, ensuring that only the most relevant and representative keyframes are retained. By refining the initial keyframes, the number of keyframes can be further reduced, ensuring that the amount of data processed in subsequent steps is minimized while retaining the most important information. By concatenating visual and textual features, a richer and more accurate description can be generated by combining visual and textual information, enhancing the clinical value of the report. Iterative prediction to generate detailed textual descriptions ensures that the visual content of each keyframe is accurately converted into linguistic information, thus avoiding human intervention and automatically generating high-quality textual descriptions, improving the speed and consistency of report generation. Through a stepwise autoregressive approach to build the report, the textual descriptions of all keyframes can be integrated into a coherent and structured medical report, ensuring completeness and logical clarity. Doctors can directly use the generated report for diagnosis without manual compilation and writing, saving time and effort. Compared with existing technologies, this application can automatically generate diagnostic reports, improving the efficiency and quality of diagnosis.

[0048] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic medical report generation method based on capsule endoscopy as described in this application. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating an embodiment of the automatic medical report generation method based on capsule endoscopy provided in this application;

[0050] Figure 2 This is a flowchart illustrating the process of removing redundant frames provided in this application;

[0051] Figure 3 This is a schematic diagram of the keyframe augmentation algorithm provided in this application;

[0052] Figure 4 This is a schematic diagram of the detection model provided in this application;

[0053] Figure 5 This is a flowchart illustrating the keyframe reduction process provided in this application;

[0054] Figure 6 This is a schematic diagram of an embodiment of the automatic medical report generation system based on capsule endoscopy provided in this application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0056] It should be understood that the step numbers used herein are only for the convenience of description, and are not limited to the execution sequence of the steps.

[0057] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0058] The terms "comprise" and "include" indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0059] The term "and / or" means any combination of one or more of the associated listed terms and all possible combinations thereof, and includes these combinations.

[0060] Capsule endoscopy is an advanced tool for diagnosing gastrointestinal diseases. It achieves full visualization of the digestive tract by swallowing a camera capsule. Its advantages are non-invasive and convenient, but there are two major bottlenecks: first, the full shooting produces a large amount of redundant images, and doctors need to spend time screening key anatomical landmarks or lesions, which may miss important information; second, existing machine learning analysis systems can only assist in identifying abnormalities, but cannot accurately provide clinical key parameters such as lesion morphology and positioning, and rely on manual integration of results to form reports, resulting in low diagnostic efficiency and limited accuracy.

[0061] Next, the terms involved in the present application are analyzed:

[0062] EndoViT is a Vision Transformer (ViT) pre-trained by Masked Autoencoder (MAE) on hundreds of thousands of endoscopic images. Although EndoViT is not trained on WCE images, resulting in its inability to better understand the semantic information of WCE images, its strong visual representation capability can help effectively distinguish the visual differences between different WCE images, thereby achieving simple redundancy reduction.

[0063] L2 normalization is a technique to scale a vector in a vector space so that the L2 norm (i.e. Euclidean norm) of the normalized vector is equal to 1. The L2 norm is the square root of the sum of the squares of the elements of the vector, and the process of normalization is to divide each element of the vector by the L2 norm of the vector.

[0064] Llama-v1.5-7b is a multimodal model that combines vision and language processing capabilities suitable for generating image descriptions. It consists of a vision encoder (based on CLIP, specifically CLIP-ViT-L / 14) and a language model (based on Vicuna, derived from Llama).

[0065] Based on this, the embodiment of the application provides a medical report automatic generation method and system based on a capsule endoscope, which can automatically generate a diagnosis report and improve the efficiency and quality of diagnosis.

[0066] The medical report automatic generation method and system based on a capsule endoscope provided by the embodiment of the application are specifically explained by the following embodiments. First, the medical report automatic generation method based on a capsule endoscope in the embodiment of the application is described.

[0067] The medical report automatic generation method based on a capsule endoscope provided by the embodiment of the application relates to the field of report generation. The medical report automatic generation method based on a capsule endoscope provided by the embodiment of the application can be applied to a terminal, can be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms; and the software can be an application that implements the medical report automatic generation method based on a capsule endoscope, but is not limited to the above forms.

[0068] The detailed description is described using specific language and examples. The description is not intended to limit the scope or application of the application in that the application is susceptible to modifications and variations. Specific exemplary methods are described in sections of this disclosure. Alternative examples can be adapted by one of ordinary skill in the art in light of the overall disclosure. The same reference numerals in different drawings denote the same or similar elements.

[0069] Embodiment One

[0070] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the medical report automatic generation method based on a capsule endoscope provided by the present application, comprising steps S101 to S103.

[0071] Step S101, acquiring a capsule endoscope video, and eliminating redundant frames of the capsule endoscope video to obtain a plurality of representative frames.

[0072] In some embodiments, a patient swallows a capsule with a camera to obtain a comprehensive capsule endoscope video from the esophagus to the colon. Since the capsule endoscope video contains a large number of redundant frames (such as normal tissue images), and the key information (such as anatomical landmarks or lesions) is often scattered in tens of thousands of frames, at the same time, the redundant frames are highly similar in visual content, and have limited contribution to diagnosis, so it is necessary to eliminate the redundant frames to reduce the scale of video frame number, while retaining the diagnostic related content.

[0073] In some embodiments, the eliminating the redundant frames of the capsule endoscope video to obtain a plurality of representative frames comprises: inputting the capsule endoscope video into a preset analysis model to obtain a second visual feature corresponding to each frame of image in the capsule endoscope video; calculating the similarity of the second visual features of adjacent frames, determining a plurality of clusters according to the comparison result of the similarity and a preset similarity threshold, and determining the representative frames based on each cluster. Specifically, first, the capsule endoscope video is input into a preset analysis model to extract the second visual feature of each frame of image. inputting the capsule endoscope video into a preset analysis model to extract the second visual feature of each frame of image , ; then, calculating the second visual feature of the current frame Second visual features of the previous frame The cosine similarity between them is calculated using the following formula: In the formula, For the first Frame and the Frame cosine similarity, and The first Frame and the The second visual feature of the frame; then, the similarity is compared with a preset similarity threshold. Compare the results; if the similarity is... If two frames are considered similar, they are grouped into the same cluster. If the similarity is... If the current frame forms a new cluster, then by classifying all frames in this way, several clusters can be obtained. Finally, since the frames within each cluster are highly similar, a representative frame is randomly selected from each cluster, resulting in the final redundancy-free frame. This step significantly reduces the number of video frames, thereby greatly reducing the computational load of subsequent processing.

[0074] It should be noted that the preset similarity threshold can be automatically adjusted according to the complexity or feature distribution of the video content, and this application does not impose any restrictions on it.

[0075] It should be noted that the analytical model mentioned above... It can be, but is not limited to, EndoViT or convolutional neural networks. When using EndoViT, the flowchart for removing redundant frames is shown below. Figure 2 As shown.

[0076] By removing redundant frames from the capsule endoscopy video, key information in the video can be preserved, while avoiding repeated calculations on duplicate or meaningless frames, thus improving the speed and accuracy of subsequent processing.

[0077] Step S102: Input each of the representative frames into a preset detection model to determine the initial keyframes and classification results, and use the classification results to refine the initial keyframes to obtain several target keyframes.

[0078] In some embodiments, the training process of the detection model includes: extracting features corresponding to the representative frames using a preset feature encoder, and normalizing the features to obtain frame features; calculating the cosine similarity between the frame features corresponding to each representative frame to obtain a similarity matrix, and augmenting predetermined reference keyframes based on the similarity matrix and preset video segment ground truths to obtain augmented keyframes; and performing supervised training on the keyframe detection network using the augmented keyframes and preset structural segment labels until the loss function is minimized, thus determining the detection model. First, the trained feature encoder is used... Features are extracted from each representative frame and L2 normalized to obtain frame features. , wherein the frame features The calculation formula is: Secondly, calculate the frame features corresponding to each representative frame. The cosine similarity between the frame features and those of other representative frames is used to obtain a... Inter-frame cosine similarity matrix The calculation formula is: In the formula, The inter-frame cosine similarity matrix , and The first The first representative frame and the first Each represents a frame feature corresponding to a given frame. Then, when the similarity matrix is ​​obtained... Then, based on the similarity matrix And video segment truth value ( (where 0 represents other, 1 represents esophagus, 2 represents stomach, 3 represents small intestine, and 4 represents large intestine), for each pre-determined reference keyframe. Amplification is performed to obtain amplified keyframes. The amplification condition is between non-keyframes and reference keyframes. Feature similarity The similarity must be greater than a preset similarity threshold (e.g., 2), and the non-keyframe must be similar to the reference keyframe. If they belong to the same structure and meet the conditions, the non-keyframe is considered an augmented keyframe. Finally, when the augmented keyframe is obtained... Then, using these tags and pre-defined structural segmentation labels (such as "other," "esophagus," "stomach," "small intestine," and "large intestine"), the keyframe detection network is trained in a supervised manner until the loss function is minimized, at which point the detection model is determined. A schematic diagram of the keyframe augmentation algorithm is shown below. Figure 3 As shown.

[0079] It should be noted that the calculation formula of the loss function is:

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] In the formula, is the total loss; is the key frame detection loss, including the reference key frame guided loss and the augmented key frame guided loss ; is the structure classification loss, which adopts a multi-class cross-entropy loss, and is the structure segmentation label supervised loss; is the cross-entropy loss, and the loss is only calculated for frames, indicating that we only care about the probability that the model predicts 1 at the key frame position; is the binary cross-entropy loss; is the key frame probability vector predicted by the model; is the structure classification probability matrix predicted by the model; is the number of structure categories.

[0086] In this way, by training the detection model, the initial key frame and the classification result that may contain important information can be accurately identified, and subsequent analysis is facilitated.

[0087] In some embodiments, the step of inputting each of the representative frames into a preset detection model to determine an initial key frame and a classification result comprises: inputting each of the representative frames into a preset detection model to determine a first probability and a second probability; comparing the first probability with a preset probability threshold, and if the first probability is greater than the preset probability threshold, determining the initial key frame; and determining the classification result based on the second probability. Specifically, first, input the representative frame into the preset detection model, and the model can output two probability values (the first probability and the second probability) at the same time, wherein the first probability is the probability that the model outputs each frame as a key frame, and the second probability is the probability that the model outputs each frame as a different anatomical structure category; then, compare the first probability with a preset probability threshold, and if the first probability is greater than the threshold, determine the frame as an initial key frame; at the same time, for the prediction of each frame, select the category with the highest second probability as the final classification result, and let the second probability of the i-th frame be ​ , then the classification result is: , where 0-4 correspond to "other, esophagus, stomach, small intestine, large intestine" respectively, is the second probability vector.

[0088] It should be noted that the detection model includes the feature encoder, the position encoder, the spatio-temporal feature extraction network, the key frame detection head, and the structure classification head, wherein a structural diagram of the detection model is as shown in Figure 4 .

[0089] In this way, the initial key frame that may contain important information is identified by the detection model, and the relevant frames can be obtained preliminarily, which facilitates subsequent analysis, and the classification result can help to distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained.

[0090] In some embodiments, the inputting each of the representative frames into a preset detection model to determine the first probability and the second probability comprises: inputting each of the representative frames into a preset detection model to perform feature extraction on each of the representative frames to obtain corresponding semantic features, wherein the detection model includes a spatio-temporal feature extraction network, a key frame detection head, and a structure classification head; fusing the semantic features with position encoding results to obtain fused features, and inputting the fused features into the spatio-temporal feature extraction network to obtain deep features, wherein the position encoding results are obtained by position encoding each of the representative frames; inputting the deep features into the key frame detection head and the structure classification head respectively to obtain the first probability and the second probability. Specifically, first, input each of the representative frames into the feature encoder of the preset detection model to perform feature extraction, and obtain semantic features of each representative frame; second, obtain position encoding results by position encoding each of the representative frames through a position encoder, and the relevant formula is: , wherein, is the position encoding result, is a learnable parameter matrix of the position encoder, is a current frame timestamp, is the total duration of the video, is the dimension of the position encoding; then, input the semantic features and the position encoding results of each frame into the spatio-temporal feature extraction network ViT for fusion to obtain fused features , wherein the calculation formula of the fusion is: , wherein, is the fused feature, is the semantic feature, For the position encoding result, For semantic feature dimension, This represents vector concatenation; finally, the deep features are... Input the keyframe detection head respectively and the structure classification head Among them, the keyframe detection head Output length is probability vector (That is, the first probability), structural classification head Output size is probability vector (That is, the second probability).

[0091] In some embodiments, the first probability The calculation formula is: In the formula, As the first probability, It is the sigmoid activation function. and It is a keyframe detection head The learnable parameter matrix for prediction of each frame. If the Sigmoid activation value is greater than 0.5, then its predicted class is... A value of 1 corresponds to a keyframe; otherwise, a value of 0 corresponds to a non-keyframe.

[0092] In some embodiments, the second probability The calculation formula is: In the formula, The second probability, This represents the hidden layer output of the model. and It is a structural classification head The learnable parameter matrix, the softmax function computed along the category dimension, and the second probability. That is, the five values ​​of each frame represent the probability that the structure of that frame belongs to one of the five categories: other, esophagus, stomach, small intestine, and large intestine.

[0093] It should be noted that the feature encoder a training process, specifically: combining a public dataset KvasirCapsule and a private dataset to construct a WCE classification dataset containing 18 categories (including 14 types of lesions: normal, hyperemia, erosion, ulcer, polyp, hemorrhage, vascular anomaly, elevated lesion, scar, parasite, lymphatic vessel dilation, lymphoid follicle hyperplasia, foreign body, and mucosal field reduction, and 4 anatomical landmarks: cardia, pylorus, duodenal bulb, ileocecal valve), then training a semantic classifier SEResNet50 dedicated to WCE images on the dataset, and removing the last fully connected layer and fixing the parameters to obtain a feature encoder The encoder can extract semantic features of the image, providing a basis for subsequent key frame augmentation and detection.

[0094] In this way, by determining the first probability and the second probability, the initial key frame and the classification result that may contain important information can be conveniently identified for subsequent analysis.

[0095] In some embodiments, the classification result is used to refine the initial key frame to obtain a plurality of target key frames, including: in each classification result, iteratively selecting the initial key frame with the highest first probability as a candidate key frame; calculating the feature similarity between the initial key frame with the same classification result as the candidate key frame and the candidate key frame, and removing the initial key frame with a feature similarity greater than a preset similarity threshold, if the number of candidate key frames corresponding to each classification result reaches a preset number threshold or the first probability corresponding to each initial key frame is less than a preset probability threshold, all selected candidate key frames are determined as final target key frames. Specifically, when the prediction is completed, the structure classification result and a large number of initial key frames However, in a real clinical scenario, such a large number of initial key frames is not needed Therefore, it is necessary to refine the initial key frame. The process of key frame refinement is shown in FIG. 5. Since the classification result indicates the structure to which each frame belongs, in each classification result, the initial key frame with the highest first probability is iteratively selected as a candidate key frame (a set of refined key frames). Then, the feature similarity between the initial key frame with the same classification result as the candidate key frame and the candidate key frame is calculated, and the initial key frame with a feature similarity greater than a preset similarity threshold is removed. If the iteration termination condition is met, all candidate key frames extracted by the structure classification (i.e., the set of refined key frames) are the target key frames after refinement wherein the iteration termination condition is that when the number of the candidate key frames extracted by each structure classification reaches a preset number threshold (the upper limit of the number of key frames for each category is set in advance according to the information richness of different anatomical structures) or the first probability of the remaining initial key frames is less than a preset probability threshold (such as 4), wherein a flowchart of the key frame refinement is as shown in Figure 5 .

[0096] In this way, by refining the initial key frames, the number of key frames can be further reduced, ensuring that the amount of data processed in the subsequent steps is minimized, while the most important information is retained.

[0097] In step S103, each of the target key frames is input into a preset multi-modal model to extract corresponding first visual features, and the first visual features are spliced with text features to obtain an initial text description. The initial text description is iteratively predicted until a first iteration termination condition is met, and a target text description is output, wherein the text features are obtained by mapping the segmented preset prompt text to the multi-modal model.

[0098] In some embodiments, first, when the target key frames are obtained Then, each of the target key frames is input into a preset multi-modal model (such as Llava-v1.5-7b) to extract first visual features of each target key frame by a CLIP encoder; second, the preset prompt text is segmented into a token sequence , which is then mapped to a text feature by a Vicuna word embedding layer; third, the first visual features are mapped to a language model space by a trainable projection layer to obtain visual embeddings , and the visual embeddings are spliced with the text features to form a joint input sequence ; finally, the model takes the input sequence Input as an initial context to generate a first probability distribution, and the relevant formula is: , wherein is the hidden layer representation of the Input by the Vicuna decoder, and W is an output word table projection matrix. The model selects the token with the maximum probability according to the probability distribution, i.e., determines the initial text description.

[0099] For example, the prompt text could be formatted as "This image appears in X% of the video. Render a clear and concise summary of the image."

[0100] In some embodiments, iteratively predicting the initial text description until a first iteration termination condition is met, and outputting the target text description, includes: generating a probability distribution value for word segmentation based on the initial text description using an autoregressive method, and determining a third word segmentation sequence based on the probability distribution value; continuously updating the word segmentation sequence based on the second word segmentation sequence and the initial text description until the length of the fourth word segmentation sequence is greater than a preset length threshold, and then determining the target text description based on the fourth word segmentation sequence. Specifically, after the initial text description is input into the multimodal model, the multimodal model generates a probability distribution value for word segmentation tokens using an autoregressive method, and selects the word segmentation token with the highest probability based on the probability distribution value to form the third word segmentation sequence; subsequently, for each subsequent position t, based on the historically generated token sequence... Generate the next token based on the initial text description. Based on the probability distribution, select the token with the highest probability, and gradually construct a complete word segmentation sequence. When generating... <eos>The iteration stops when the length of the generated fourth segmented word sequence exceeds a preset length threshold (either due to the end-of-strike character). Finally, the Vicuna word segmenter's decoding function converts the generated fourth segmented word sequence into natural text, forming the final descriptive caption, i.e., the target text description. It should be noted that each iteration selects the token with the highest probability to gradually construct a complete token sequence.

[0101] In some embodiments, the formula for continuously updating the word segmentation sequence is: ; In the formula, This is the hidden layer representation when generating the t-th token. The model generates the current token. The probability distribution, and For word segmentation sequence tokens, For the initial text description, It is the output vocabulary projection matrix of the model.

[0102] It should be noted that multimodal models can be fine-tuned using the LoRA (Low-Rank Adaptation) technique. LoRA uses Vicuna's attention weight matrix... Add a low-rank update matrix ΔW, and freeze the original weight matrix during each weight update. Only ΔW is updated, thus achieving lightweight model tuning, significantly reducing the number of training parameters and computational resource requirements while maintaining the model's generalization ability. The goal of multimodal model training is to minimize the captioning loss, using the standard cross-entropy loss function to measure the difference between the sequence of tokens describing the caption generated by the model and the sequence of tokens describing the caption. In the formula, This represents the number of samples in the training batch. For the first The number of tokens in the real caption of each sample. For the first The first true caption of the sample A token. For the model in a given graph and before In the case of the nth token, predict the nth... Each token is The probability of the caption is calculated. By minimizing this loss function, the model learns to generate descriptions that are as similar as possible to the true caption, thereby improving the accuracy and clinical relevance of the caption.

[0103] In this way, a detailed text description is generated by iterative prediction, which can ensure that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention and automatically generating high-quality text descriptions.

[0104] In step S104, the target text description is input into a large language model, and a first token sequence is predicted step by step by using a self-recurrent method until a second iteration termination condition is met, and a corresponding structured medical report is generated based on a second token sequence.

[0105] In some embodiments, the inputting of the target text description into the large language model includes: obtaining a time point corresponding to each of the target key frames, and forming a description pair by combining the time point and the target text description; and splicing the description pair with a preset text prompt to obtain a spliced result, and inputting the spliced result into the large language model. Specifically, the time point corresponding to each of the target key frames is obtained , and the key frame time point and the corresponding target text description form a time-description pair , where m is the number of extracted refined key frames; then the time-description pair is spliced with the text prompt to form a complete input string for input into the large language model. For example, when the text prompt is "generate a medical report according to the following key frame description:", the complete input string is "generate a medical report according to the key frame description: ".

[0106] In some embodiments, the first token sequence is predicted step by step by using a self-recurrent method, specifically, the complete input string is converted into a token sequence by using a tokenizer of the large language model, and the token sequence is input into a word embedding layer to obtain a corresponding embedding vector representation , where the word embedding layer maps each token to a fixed-dimensional vector space, for example, each token is mapped to a 768-dimensional vector, so that is a matrix, the number of rows of which is equal to the length of the token sequence, and the number of columns of which is equal to the dimension of the embedding vector, then the large language model iteratively predicts the next token according to the input token sequence, for each position t, the model calculates the probability distribution of the next token based on the previously generated tokens , the input text prompt, and the time-description pair , and selects the token with the highest probability as the output token sequence .

[0107] ​In some embodiments, the second iteration termination condition is satisfied until a second iteration termination condition is satisfied, in particular, when generating <eos>The iteration stops when the end symbol (EOS) is reached or the length of the first segmented sequence reaches a maximum length, wherein the maximum length can be preset in advance, which is not limited in the present application.

[0108] In some embodiments, a corresponding structured medical report is generated based on the second segmented sequence, specifically, the generated token sequence, i.e., the second segmented sequence, is converted into natural text by the decoder of the large language model to form the final structured medical report, ensuring that the report content is accurate, coherent and meets clinical needs.

[0109] It should be noted that the Llama-7B model is used in this stage to realize the conversion of the text caption to the structured medical report. The fine-tuning process is as follows: the real key frame description of each patient's key frame time is collected as the training input sample, and the corresponding structured report is collected as the training output sample. When training the Llama-7B model, the goal is to minimize the cross-entropy loss function to maximize the probability of the output sequence, thereby obtaining the large language model. The loss function is defined as: , wherein is the total number of tokens of the summary report, is the token of the real report, is the embedding vector representation of the complete input text.

[0110] It should be noted that in order to improve the robustness of the model in real scenarios, the training data is enhanced by the following strategies to simulate the redundant captions and time deviations that may occur during testing: 1. Increase the number of captions: increase the number of input captions by duplicating existing captions or generating similar descriptions. 2. Randomly adjust the time: randomly slide the time of each key frame forward and backward to generate new time points. Through these strategies, the model can better handle uncertainties and redundant information in real scenarios, ensuring that the generated report is accurate and coherent.

[0111] The embodiments of the present application can retain key information in the video by eliminating redundant frames of the capsule endoscopy video, avoid repeated calculation on repeated or meaningless frames, improve subsequent processing speed and accuracy; the initial key frame that may contain important information can be identified by the detection model, the relevant frames can be obtained initially, which facilitates subsequent analysis, and the classification result can help to distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained; by refining the initial key frames, the number of key frames can be further reduced to minimize the data amount processed in subsequent steps while retaining the most important information; by splicing the first visual feature and the text feature, visual and text information can be combined to generate more rich and accurate descriptions, thereby improving the clinical value of the report; by iteratively predicting to generate detailed text descriptions, the visual content of each key frame can be accurately converted into language information, thereby avoiding human intervention, automatically generating high-quality text descriptions, and improving the speed and consistency of report generation; by gradually constructing the report through the autoregressive method, the text descriptions of all key frames can be integrated into a coherent and structured medical report, ensuring that the report content is complete and logically clear, and doctors can directly use the generated report for diagnosis without manually organizing and writing, thereby saving time and effort. Compared with the prior art, the present application can automatically generate a diagnostic report to improve the efficiency and quality of diagnosis.

[0112] Embodiment two

[0113] Please refer to Figure 6 , Figure 6 The medical report automatic generation system based on a capsule endoscope provided by the present application comprises an acquisition module 100, a processing module 200, a first prediction module 300 and a second prediction module 400.

[0114] The acquisition module 100 is configured to acquire a capsule endoscopy video, eliminate redundant frames of the capsule endoscopy video, and obtain a plurality of representative frames.

[0115] The processing module 200 is configured to input each of the representative frames into a preset detection model, determine initial key frames and classification results, and refine the initial key frames using the classification results to obtain a plurality of target key frames.

[0116] The first prediction module 300 is configured to input each of the target key frames into a preset multi-modal model, extract corresponding first visual features, splice the first visual features and text features to obtain initial text descriptions, iteratively predict the initial text descriptions until a first iteration termination condition is met, and output target text descriptions, wherein the text features are obtained by mapping the segmented preset prompt text to the multi-modal model.

[0117] The second prediction module 400 is configured to input the target text description into a large language model, predict a first segmentation sequence step by step by using a self-recurrence method until a second iteration termination condition is met, and generate a corresponding structured medical report based on the second segmentation sequence.

[0118] The information interaction and execution process between the modules in the above-described capsule endoscopy-based medical report automatic generation system are based on the same concept as the embodiments of the capsule endoscopy-based medical report automatic generation method of the first aspect of the present application, and achieve basically the same technical effects. For specific content, refer to the description in the first embodiment of the method of the present application, which will not be repeated here.

[0119] The device embodiments described above are merely illustrative, and the modules described as separate components may or may not be physically separated, i.e., they may be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the method of the present embodiment.

[0120] Those skilled in the art can understand that all or part of the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0121] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above-described embodiments are only specific embodiments of the present application and do not limit the protection scope of the present application.

[0122] It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.< / eos> < / eos>

Claims

1. A capsule endoscope-based medical report automatic generation method, characterized by, The method comprises the following steps: acquiring a capsule endoscope video, and removing redundant frames of the capsule endoscope video to obtain a plurality of representative frames; inputting each of the representative frames into a preset detection model to determine initial key frames and classification results, and performing fine reduction on the initial key frames by using the classification results to obtain a plurality of target key frames; inputting each of the target key frames into a preset multi-modal model to extract corresponding first visual features, splicing the first visual features with text features to obtain an initial text description, and iteratively predicting the initial text description until a first iteration termination condition is met, and outputting a target text description, wherein the text features are obtained by mapping a segmented preset prompt text to the multi-modal model; inputting the target text description into a large language model, and using a self-recurrent method to gradually predict a first segmented sequence until a second iteration termination condition is met, and generating a corresponding structured medical report based on a second segmented sequence; wherein the removing of the redundant frames of the capsule endoscope video to obtain a plurality of representative frames comprises: inputting the capsule endoscope video into a preset analysis model to obtain second visual features corresponding to each frame of image in the capsule endoscope video; calculating the similarity of adjacent frame second visual features, determining a plurality of clusters according to the comparison result of the similarity and a preset similarity threshold, and determining representative frames based on each of the clusters; wherein the inputting of each of the representative frames into a preset detection model to determine initial key frames and classification results comprises: inputting each of the representative frames into a preset detection model to determine a first probability and a second probability; comparing the first probability with a preset probability threshold, and if the first probability is greater than the preset probability threshold, determining an initial key frame; and determining a classification result based on the second probability; wherein the training process of the detection model comprises: extracting features corresponding to the representative frames by using a preset feature encoder, and performing normalization processing on the features to obtain frame features; calculating the cosine similarity between frame features corresponding to each of the representative frames to obtain a similarity matrix, and based on the similarity matrix and a preset video segmentation true value, expanding a pre-determined reference key frame to obtain an expanded key frame; using the expanded key frame and a preset structure segmentation label to perform supervised training on a key frame detection network until the loss function is minimized, and determining the detection model; wherein the iteratively predicting of the initial text description until a first iteration termination condition is met, and outputting a target text description comprises: generating a probability distribution value of segmentation based on the initial text description by using a self-recurrent method, and determining a third segmented sequence based on the probability distribution value; constantly updating a segmented sequence based on the second segmented sequence and the initial text description until the length of a fourth segmented sequence is greater than a preset length threshold, and then determining a target text description based on the fourth segmented sequence.

2. The capsule endoscopy-based medical report automatic generation method of claim 1, wherein, The inputting of each of the representative frames into a preset detection model to determine a first probability and a second probability comprises: input each of the representative frames into a preset detection model to perform feature extraction on each of the representative frames, to obtain corresponding semantic features, wherein the detection model comprises a space-time feature extraction network, a key frame detection head, and a structure classification head; fuse the semantic features with position encoding results to obtain fused features, and input the fused features into the space-time feature extraction network to obtain deep features, wherein the position encoding results are obtained by performing position encoding on each of the representative frames; input the deep features into the key frame detection head and the structure classification head respectively to obtain first probabilities and second probabilities.

3. The capsule endoscopy-based medical report automatic generation method of claim 1, wherein, the classification results to perform fine shrinking on the initial key frames, to obtain a plurality of target key frames, specifically as follows: in each of the classification results, select the initial key frame with the highest first probability as a candidate key frame; calculate feature similarities between the initial key frames with the same classification result as the candidate key frame and the candidate key frame, remove the initial key frames with a feature similarity greater than a preset similarity threshold, and if the number of candidate key frames corresponding to each classification result reaches a preset number threshold or the first probability corresponding to each initial key frame is less than a preset probability threshold, determine that all selected candidate key frames are final target key frames.

4. The capsule endoscopy-based medical report automatic generation method of claim 1, wherein, the target text description into a large language model, specifically as follows: obtain time points corresponding to each of the target key frames, and form a description pair by combining the time points with the target text description; splice the description pair with a preset text prompt to obtain a spliced result, and input the spliced result into a large language model.

5. The capsule endoscopy based medical report automatic generation method of claim 1, wherein, based on the first segmentation sequence, generate a corresponding structured medical report, specifically as follows: decode the first segmentation sequence to obtain a first text; extract key information from the first text, and fill the key information into a preset report template to obtain a corresponding structured medical report.

6. A capsule endoscopy based medical report automatic generation system characterized by, comprise: an acquisition module, a processing module, a first prediction module, and a second prediction module; the acquisition module is configured to acquire a capsule endoscope video, and remove redundant frames of the capsule endoscope video to obtain a plurality of representative frames; the processing module is configured to input each of the representative frames into a preset detection model, determine initial key frames and classification results, and perform fine shrinking on the initial key frames by using the classification results to obtain a plurality of target key frames; the first prediction module is configured to input each of the target key frames into a preset multi-modal model, extract corresponding first visual features, splice the first visual features with text features to obtain an initial text description, perform iterative prediction on the initial text description until a first iteration termination condition is met, and output a target text description, wherein the text features are obtained by mapping a preset prompt text after segmentation to the multi-modal model; the second prediction module is configured to input the target text description into a large language model, use a self-recurrent method to gradually predict a first segmentation sequence until a second iteration termination condition is met, and generate a corresponding structured medical report based on a second segmentation sequence. The redundant frames of the capsule endoscopy video are removed to obtain a plurality of representative frames, specifically: the capsule endoscopy video is input into a preset analysis model to obtain second visual features corresponding to each frame of image in the capsule endoscopy video; the similarity of second visual features of adjacent frames is calculated, a plurality of clusters are determined according to a comparison result of the similarity and a preset similarity threshold, and representative frames are determined based on the clusters; The detection model is trained, specifically: features corresponding to the representative frames are extracted by using a preset feature encoder, and the features are normalized to obtain frame features; the cosine similarity between the frame features corresponding to each representative frame is calculated to obtain a similarity matrix, and the reference key frames are augmented based on the similarity matrix and a preset video segmentation true value to obtain augmented key frames; the key frame detection network is supervisedly trained by using the augmented key frames and a preset structure segmentation label, and the detection model is determined when the loss function is minimized; The initial text description is iteratively predicted until a first iteration termination condition is met, and a target text description is output, specifically: a probability distribution value of word segmentation is generated by using an autoregressive method based on the initial text description, and a third word segmentation sequence is determined based on the probability distribution value; the word segmentation sequence is constantly updated based on the second word segmentation sequence and the initial text description until the length of the fourth word segmentation sequence is greater than a preset length threshold, and then the target text description is determined based on the fourth word segmentation sequence. The detection model is trained, specifically: features corresponding to the representative frames are extracted by using a preset feature encoder, and the features are normalized to obtain frame features; the cosine similarity between the frame features corresponding to each representative frame is calculated to obtain a similarity matrix, and the reference key frames are augmented based on the similarity matrix and a preset video segmentation true value to obtain augmented key frames; the key frame detection network is supervisedly trained by using the augmented key frames and a preset structure segmentation label, and the detection model is determined when the loss function is minimized;

Images