Medical report automatic generation method and system based on capsule endoscope

Abstract

The invention discloses an automatic medical report generation method and system based on a capsule endoscope, and the method comprises the steps: obtaining a capsule endoscope video, and removing redundant frames of the capsule endoscope video, and obtaining a plurality of representative frames; inputting each representative frame into a preset detection model, determining an initial key frame and a classification result, and performing fine reduction on the initial key frame by using the classification result to obtain a plurality of target key frames; inputting each target key frame into a preset multi-modal model, extracting a corresponding first visual feature, splicing the first visual feature and the text feature to obtain an initial text description, performing iterative prediction on the initial text description until a first iteration termination condition is met, and outputting a target text description; and inputting the target text description into the large language model, predicting the first word segmentation sequence step by step by using an autoregression method until a second iteration termination condition is met, and generating a corresponding structured medical report based on the second word segmentation sequence. The diagnosis report can be automatically generated.

Description

Technical Field

[0001] The present application relates to the field of report generation, and in particular to a method and system for automatically generating medical reports based on capsule endoscopy. Background Art

[0002] Capsule endoscopy is an advanced medical device used for the early diagnosis of gastrointestinal diseases. Patients simply swallow a capsule equipped with a camera to obtain comprehensive images from the esophagus to the colon. Although capsule endoscopy has many advantages in disease diagnosis, a complete record of the digestive tract requires continuous capture of a large number of normal tissue images. This requires doctors to spend a lot of time and effort to screen key information, such as anatomical landmarks or lesion sites, during analysis. This not only increases the doctor's workload but also may lead to the omission of some important information.

[0003] Currently, although machine learning has been used to analyze endoscopic videos, it has failed to effectively detect potential lesions in the gastrointestinal tract and cannot provide detailed information such as the size, location, and morphology of the lesions. In addition, after the analysis is completed, it still depends on doctors to manually integrate the analysis results, which is not only time-consuming and labor-intensive, but may also lead to incomplete or inaccurate information, thereby affecting the efficiency and quality of diagnosis. Summary of the Invention

[0004] The present application provides a method and system for automatically generating medical reports based on capsule endoscopy to automatically generate diagnostic reports and improve the efficiency and quality of diagnosis.

[0005] In a first aspect, the present application provides a method for automatically generating a medical report based on capsule endoscopy, comprising: Acquire a capsule endoscopy video, and remove redundant frames of the capsule endoscopy video to obtain a plurality of representative frames; Inputting each representative frame into a preset detection model to determine an initial key frame and a classification result, and using the classification result to refine the initial key frame to obtain a plurality of target key frames; Inputting each target keyframe into a preset multimodal model, extracting corresponding first visual features, and concatenating the first visual features with text features to obtain an initial text description, iteratively predicting the initial text description until a first iteration termination condition is satisfied, and outputting a target text description, wherein the text features are obtained by mapping the preset prompt text to the multimodal model after word segmentation; The target text description is input into a large language model, and the first word segmentation sequence is gradually predicted using an autoregressive method until the second iteration termination condition is met, and a corresponding structured medical report is generated based on the second word segmentation sequence.

[0006] The embodiment of the present application can retain the key information in the video by eliminating redundant frames of the capsule endoscopy video, while avoiding repeated calculation of repeated or meaningless frames, thereby improving the subsequent processing speed and accuracy; by identifying the initial key frames that may contain important information through the detection model, it is possible to initially obtain relevant frames, which is convenient for subsequent analysis, and the classification results can help distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained; by refining the initial key frames, the number of key frames can be further reduced, ensuring that the amount of data processed in subsequent steps is minimized while retaining the most important information; by The first visual feature is spliced with the text feature to combine visual and textual information to generate richer and more accurate descriptions, thereby enhancing the clinical value of the report; generating detailed text descriptions through iterative prediction can ensure that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention, automatically generating high-quality text descriptions, and improving the speed and consistency of report generation; gradually constructing the report through the autoregressive method can integrate the text descriptions of all key frames into a coherent and structured medical report, ensuring that the report content is complete and the logic is clear. Doctors can directly use the generated report for diagnosis without manual organization and writing, saving time and energy. Compared with the existing technology, this application can automatically generate diagnostic reports and improve the efficiency and quality of diagnosis.

[0007] Furthermore, the redundant frames of the capsule endoscopy video are removed to obtain several representative frames, specifically: Inputting the capsule endoscopy video into a preset analysis model to obtain a second visual feature corresponding to each frame image in the capsule endoscopy video; The similarity of the second visual features of adjacent frames is calculated, a number of clusters are determined based on a comparison result of the similarity with a preset similarity threshold, and a representative frame is determined based on each of the clusters.

[0008] In this way, by eliminating redundant frames of the capsule endoscopy video, key information in the video can be retained, while repeated calculations of repeated or meaningless frames can be avoided, thereby improving subsequent processing speed and accuracy.

[0009] Furthermore, the representative frames are input into a preset detection model to determine the initial key frames and classification results, specifically: Inputting each of the representative frames into a preset detection model to determine a first probability and a second probability; Comparing the first probability with a preset probability threshold, and if the probability is greater than the predetermined probability, determining an initial key frame; A classification result is determined based on the second probability.

[0010] In this way, the detection model can identify initial key frames that may contain important information, and relevant frames can be initially obtained to facilitate subsequent analysis. The classification results can help distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained.

[0011] Furthermore, the inputting of each representative frame into a preset detection model to determine the first probability and the second probability is specifically as follows: Inputting each representative frame into a preset detection model to extract features from each representative frame to obtain corresponding semantic features, wherein the detection model includes a spatiotemporal feature extraction network, a key frame detection head, and a structure classification head; Fusing the semantic features with the position encoding results to obtain fused features, and inputting the fused features into the spatiotemporal feature extraction network to obtain deep features, wherein the position encoding results are obtained by position encoding each of the representative frames; The deep features are input into the key frame detection head and the structure classification head respectively to obtain a first probability and a second probability.

[0012] In this way, by determining the first probability and the second probability, it is possible to facilitate subsequent identification of initial key frames and classification results that may contain important information, thereby facilitating subsequent analysis.

[0013] Furthermore, the training process of the detection model is specifically as follows: Extracting features corresponding to the representative frame using a preset feature encoder, and normalizing the features to obtain frame features; Calculating the cosine similarity between the frame features corresponding to each of the representative frames to obtain a similarity matrix, and amplifying the predetermined reference key frame based on the similarity matrix and a preset video segment true value to obtain an amplified key frame; The key frame detection network is supervisedly trained using the amplified key frames and preset structure segmentation labels until the loss function is minimized, thereby determining the detection model.

[0014] In this way, by training the detection model, the initial key frames and classification results that may contain important information can be accurately identified, which facilitates subsequent analysis.

[0015] Furthermore, the initial key frames are refined using the classification results to obtain several target key frames, specifically: Iteratively selecting the initial key frame with the highest first probability from among the classification results as a candidate key frame; Calculate the feature similarity between the initial key frame and the candidate key frame that is the same as the classification result of the candidate key frame, and remove the initial key frame whose feature similarity is greater than a preset similarity threshold. If the number of the candidate key frames corresponding to each of the classification results reaches a preset number threshold or the first probability corresponding to each of the initial key frames is less than a preset probability threshold, determine all the selected candidate key frames as the final target key frames.

[0016] In this way, by condensing the initial key frames, the number of key frames can be further reduced, ensuring that the amount of data processed in subsequent steps is minimized while retaining the most important information.

[0017] Furthermore, the initial text description is iteratively predicted until a first iteration termination condition is met, and a target text description is output, specifically: Based on the initial text description, generating a probability distribution value of word segmentation by an autoregressive method, and determining a third word segmentation sequence based on the probability distribution value; The word segmentation sequence is continuously updated based on the second word segmentation sequence and the initial text description until the length of the fourth word segmentation sequence is greater than a preset length threshold, and the target text description is determined based on the fourth word segmentation sequence.

[0018] In this way, detailed text descriptions are generated through iterative prediction, which ensures that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention and automatically generating high-quality text descriptions.

[0019] Furthermore, the target text description is input into the large language model, specifically: Obtaining a time point corresponding to each target keyframe, and forming a description pair with the time point and the target text description; The description pair is spliced with a preset text prompt to obtain a splicing result, and the splicing result is input into a large language model.

[0020] In this way, by inputting the target text description into a large language model, the text descriptions of all key frames can be accurately integrated into a coherent, structured medical report, ensuring that the report content is complete and the logic is clear. Doctors can directly use the generated report for diagnosis without manual organization and writing, saving time and energy.

[0021] In a second aspect, the present application further provides a system for automatically generating medical reports based on capsule endoscopy, comprising: an acquisition module, a processing module, a first prediction module, and a second prediction module; The acquisition module is used to acquire capsule endoscopy video and remove redundant frames of the capsule endoscopy video to obtain a plurality of representative frames; The processing module is configured to input each representative frame into a preset detection model, determine an initial key frame and a classification result, and refine the initial key frame using the classification result to obtain a plurality of target key frames; The first prediction module is configured to input each target keyframe into a preset multimodal model, extract corresponding first visual features, and concatenate the first visual features with text features to obtain an initial text description, iteratively predict the initial text description until a first iteration termination condition is satisfied, and output a target text description, wherein the text features are obtained by mapping the preset prompt text to the multimodal model after word segmentation; The second prediction module is used to input the target text description into the large language model, use the autoregressive method to gradually predict the first word segmentation sequence until the second iteration termination condition is met, and generate a corresponding structured medical report based on the second word segmentation sequence.

[0022] The embodiment of the present application can retain the key information in the video by eliminating redundant frames of the capsule endoscopy video, while avoiding repeated calculation of repeated or meaningless frames, thereby improving the subsequent processing speed and accuracy; by identifying the initial key frames that may contain important information through the detection model, it is possible to initially obtain relevant frames, which is convenient for subsequent analysis, and the classification results can help distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained; by refining the initial key frames, the number of key frames can be further reduced, ensuring that the amount of data processed in subsequent steps is minimized while retaining the most important information; by The first visual feature is spliced with the text feature to combine visual and textual information to generate richer and more accurate descriptions, thereby enhancing the clinical value of the report; generating detailed text descriptions through iterative prediction can ensure that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention, automatically generating high-quality text descriptions, and improving the speed and consistency of report generation; gradually constructing the report through the autoregressive method can integrate the text descriptions of all key frames into a coherent and structured medical report, ensuring that the report content is complete and the logic is clear. Doctors can directly use the generated report for diagnosis without manual organization and writing, saving time and energy. Compared with the existing technology, this application can automatically generate diagnostic reports and improve the efficiency and quality of diagnosis.

[0023] In a third aspect, the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for automatically generating a medical report based on capsule endoscopy as described in the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1This is a flow chart of an embodiment of a method for automatically generating a medical report based on capsule endoscopy provided by the present application; Figure 2 This is a flowchart of removing redundant frames provided by this application; Figure 3 is a schematic diagram of the key frame amplification algorithm provided by this application; Figure 4 It is a structural diagram of the detection model provided by this application; Figure 5 This is a schematic diagram of the key frame reduction process provided by this application; Figure 6 This is a structural diagram of an embodiment of a capsule endoscope-based medical report automatic generation system provided in this application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0027] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] Capsule endoscopy, an advanced diagnostic tool for gastrointestinal diseases, enables visual monitoring of the entire digestive tract by swallowing a camera capsule. While its advantages lie in its noninvasive and convenient nature, it faces two major bottlenecks: First, the extensive imaging process generates a massive amount of redundant images, requiring physicians to time-consumingly screen for key anatomical landmarks or lesions, which can easily lead to missing important information. Second, existing machine learning analysis systems can only assist in identifying abnormalities but cannot accurately provide key clinical parameters such as lesion morphology and location. Furthermore, they rely on manual integration of results to generate reports, resulting in low diagnostic efficiency and limited accuracy.

[0031] Next, the nouns involved in this application are analyzed: EndoViT is a Vision Transformer (ViT) pre-trained on hundreds of thousands of endoscopic images using a Masked Autoencoder (MAE). Although EndoViT is not trained on WCE images, which prevents it from fully understanding their semantic information, its strong visual representation capabilities enable it to effectively distinguish visual differences between different WCE images, thereby achieving simple redundancy removal.

[0032] L2 normalization is a technique for scaling vectors in vector space so that the L2 norm (also known as the Euclidean norm) of the normalized vector is equal to 1. The L2 norm is the square root of the sum of the squares of the elements in the vector. Normalization is done by dividing each element in the vector by the L2 norm of the vector.

[0033] Llava-v1.5-7b is a multimodal model that combines vision and language processing capabilities for generating image descriptions. It consists of a visual encoder (based on CLIP, specifically CLIP-ViT-L / 14) and a language model (based on Vicuna, derived from Llama).

[0034] Based on this, the embodiments of the present application provide a method and system for automatically generating medical reports based on capsule endoscopy, which can automatically generate diagnostic reports and improve the efficiency and quality of diagnosis.

[0035] The method and system for automatically generating medical reports based on capsule endoscopy provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the method for automatically generating medical reports based on capsule endoscopy in the embodiments of the present application is described.

[0036] The method for automatically generating medical reports based on capsule endoscopy provided in the embodiment of the present application relates to the field of report generation. The method for automatically generating medical reports based on capsule endoscopy provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for automatically generating medical reports based on capsule endoscopy, etc., but is not limited to the above forms.

[0037] The present application can also be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0038] Example 1 Please refer to Figure 1 , Figure 1 This is a flowchart of an embodiment of a method for automatically generating a medical report based on capsule endoscopy provided by the present application, including steps S101 to S103; Step S101, obtaining a capsule endoscopy video, and removing redundant frames of the capsule endoscopy video to obtain a plurality of representative frames; In some embodiments, the patient swallows a capsule equipped with a camera to obtain a comprehensive capsule endoscopy video from the esophagus to the colon. Since the capsule endoscopy video contains a large number of redundant frames (such as normal tissue images), and key information (such as anatomical landmarks or lesions) is often scattered among tens of thousands of frames, and the redundant frames are highly similar in visual content and have limited contribution to diagnosis, the redundant frames need to be removed to reduce the number of video frames while retaining diagnosis-related content.

[0039] In some embodiments, the method of removing redundant frames of the capsule endoscopy video to obtain a plurality of representative frames includes: inputting the capsule endoscopy video into a preset analysis model to obtain a second visual feature corresponding to each frame of the capsule endoscopy video; calculating the similarity of the second visual features of adjacent frames, determining a plurality of clusters based on the comparison results of the similarity with a preset similarity threshold, and determining a representative frame based on each cluster. Specifically, first, the capsule endoscopy video is input into a preset analysis model to obtain a second visual feature corresponding to each frame of the capsule endoscopy video; calculating the similarity of the second visual features of adjacent frames, determining a plurality of clusters based on the comparison results of the similarity with a preset similarity threshold, and determining a representative frame based on each cluster. Importing a preset analysis model , to extract the second visual features of each frame image , ; Next, calculate the second visual feature of the current frame The second visual feature of the previous frame The cosine similarity between them is calculated as follows: , where For the Frame and Cosine similarity of frames, and Respectively Frame and The second visual feature of the frame; then, the similarity is compared with the preset similarity threshold Compare, if the similarity , if the two frames are considered similar, they will be grouped into the same cluster. , then the current frame forms a new cluster. In this way, all frames are classified according to this method to obtain several clusters. Finally, since the frame images in each cluster are highly similar, a frame is randomly selected from each cluster as a representative frame, and finally the de-redundant frame is obtained. Through this step, the number of video frames is significantly reduced, thereby greatly reducing the computational load of subsequent processing.

[0040] It should be noted that the preset similarity threshold can be automatically adjusted according to the complexity or feature distribution of the video content, and this application does not impose any restrictions.

[0041] It should be noted that the analysis model It can be but not limited to EndoViT or convolutional neural network. When EndoViT is used, the flowchart of removing redundant frames is as follows: Figure 2 shown.

[0042] In this way, by eliminating redundant frames of the capsule endoscopy video, key information in the video can be retained, while repeated calculations of repeated or meaningless frames can be avoided, thereby improving subsequent processing speed and accuracy.

[0043] Step S102: inputting each representative frame into a preset detection model to determine an initial key frame and a classification result, and using the classification result to refine the initial key frame to obtain a plurality of target key frames; In some embodiments, the training process of the detection model includes: using a preset feature encoder to extract the features corresponding to the representative frames, and normalizing the features to obtain frame features; calculating the cosine similarity between the frame features corresponding to each of the representative frames to obtain a similarity matrix, and based on the similarity matrix and the preset video segmentation true value, amplifying the predetermined reference key frame to obtain an amplified key frame; using the amplified key frame and the preset structure segmentation label, supervised training is performed on the key frame detection network until the loss function is minimized, and the detection model is determined. First, use the trained feature encoder Extract the features of each representative frame and perform L2 normalization to obtain the frame features , wherein the frame features The calculation formula is: ; Secondly, calculate the frame features corresponding to each representative frame The cosine similarity between the frame features corresponding to other representative frames is obtained, thereby obtaining a The inter-frame cosine similarity matrix , the calculation formula is: , where is the inter-frame cosine similarity matrix , and Respectively Representative frames and The frame features corresponding to the representative frames. Then, when the similarity matrix is obtained Afterwards, based on the similarity matrix and video segmentation truth ( , where 0 represents others, 1 represents esophagus, 2 represents stomach, 3 represents small intestine, and 4 represents large intestine), for each predetermined reference key frame Amplify and obtain the amplified key frame , wherein the amplification condition is the non-key frame and the reference key frame The feature similarity The similarity between the non-key frame and the reference key frame must be greater than the preset similarity threshold (such as 2). Belonging to the same structure, if the conditions are met, the non-key frame is considered to be an augmented key frame Finally, when the amplified key frame is obtained Then, the key frame detection network is trained in a supervised manner using the preset structural segmentation labels (such as other, esophagus, stomach, small intestine and large intestine) until the loss function is minimized, and the detection model is determined. The schematic diagram of the key frame expansion algorithm is shown in the figure. Figure 3 shown.

[0044] It should be noted that the calculation formula of the loss function is: ; ; ; ; ; Where, is the total loss; is the keyframe detection loss, including the reference keyframe Guided loss and augmented keyframes Guided loss ; is the structural classification loss, using multi-class cross entropy loss, and is the structural segmentation label loss of supervision; is the cross entropy loss, the loss is only for Frame calculation means that we only care about the probability that the model predicts 1 at the key frame position; is the binary cross entropy loss; is the key frame probability vector predicted by the model; is the structural classification probability matrix predicted by the model; is the number of structural categories.

[0045] In this way, by training the detection model, the initial key frames and classification results that may contain important information can be accurately identified, which facilitates subsequent analysis.

[0046] In some embodiments, the inputting of each representative frame into a preset detection model to determine the initial key frame and classification result includes: inputting each representative frame into a preset detection model to determine a first probability and a second probability; comparing the first probability with a preset probability threshold, and if it is greater than, determining the initial key frame; and determining the classification result based on the second probability. Specifically: first, the representative frame is input into a preset detection model, and the model can output two probability values (first probability and second probability) at the same time, wherein the first probability is the probability that each frame output by the model is a key frame, and the second probability is the probability that each frame output by the model belongs to a different anatomical structure category; then, the first probability is compared with the preset probability threshold, and if the first probability is greater than the threshold, the frame is determined as the initial key frame; at the same time, for the prediction of each frame, the category with the second highest probability is selected as the final classification result, and the first probability is set as the final classification result. The second probability of the frame is , the classification results are: , where 0-4 correspond to the five categories of "other, esophagus, stomach, small intestine, large intestine" respectively. is the second probability vector.

[0047] It should be noted that the detection model includes the feature encoder, position encoder, spatiotemporal feature extraction network, key frame detection head and structure classification head, wherein the structural diagram of the detection model is as follows: Figure 4 shown.

[0048] In this way, the detection model can identify initial key frames that may contain important information, and relevant frames can be initially obtained to facilitate subsequent analysis. The classification results can help distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained.

[0049] In some embodiments, the inputting of each of the representative frames into a preset detection model to determine the first probability and the second probability includes: inputting each of the representative frames into a preset detection model to extract features of each of the representative frames to obtain corresponding semantic features, wherein the detection model includes a spatiotemporal feature extraction network, a key frame detection head, and a structural classification head; fusing the semantic features with the position encoding results to obtain fused features, and inputting the fused features into the spatiotemporal feature extraction network to obtain deep features, wherein the position encoding results are obtained after position encoding of each of the representative frames; inputting the deep features into the key frame detection head and the structural classification head respectively to obtain the first probability and the second probability. Specifically, first, each of the representative frames is input into the feature encoder of the preset detection model. To perform feature extraction, the semantic features of each representative frame are obtained ; Secondly, the position encoding result is obtained by position encoding each representative frame through the position encoder , the relevant formula is: , where is the position encoding result, is the learnable parameter matrix of the position encoder, is the current frame timestamp, is the total length of the video, is the dimension of position encoding; then, the semantic features The position encoding result of each frame Input the spatiotemporal feature extraction network ViT for fusion to obtain the fusion feature , where the fusion calculation formula is: , where To fusion features, is a semantic feature, is the position encoding result, is the semantic feature dimension, Represents vector concatenation; Finally, the deep features Input the key frame detection header respectively and the structure classification header , where the key frame detection head The output length is The probability vector (also known as the first probability), structural classification head The output size is The probability vector (Also the second probability).

[0050] In some embodiments, the first probability The calculation formula is: , where is the first probability, is the sigmoid activation function, and It is the key frame detection head The learnable parameter matrix for each frame prediction , if the Sigmoid activation value is greater than 0.5, then its predicted category If it is 1, it corresponds to a key frame; otherwise, it is 0, which corresponds to a non-key frame.

[0051] In some embodiments, the second probability The calculation formula is: , where is the second probability, represents the hidden layer output of the model, and It is the structural classification head The learnable parameter matrix, the Softmax function is calculated along the category dimension, and the second probability That is, the five values of each frame represent the probability that the frame structure belongs to the five categories of others, esophagus, stomach, small intestine and large intestine.

[0052] It should be noted that the feature encoder The training process is as follows: a WCE classification dataset containing 18 categories is constructed by combining the public dataset KvasirCapsule and the private dataset (including 14 lesion types: normal, congestion, erosion, ulcer, polyp, hemorrhage, vascular abnormality, raised lesion, scar, parasite, lymphangiectasia, lymph follicle hyperplasia, foreign body, reduced mucosal field of view and 4 anatomical landmarks: cardia, pylorus, duodenal bulb, ileocecal valve). Then, a semantic classifier SEResNet50 dedicated to WCE images is trained on this dataset, and its last fully connected layer and fixed parameters are removed to obtain the feature encoder. ,The encoder can extract the semantic features of the image, providing a basis for subsequent key frame expansion and detection.

[0053] In this way, by determining the first probability and the second probability, it is possible to facilitate subsequent identification of initial key frames and classification results that may contain important information, thereby facilitating subsequent analysis.

[0054] In some embodiments, the classification results are used to refine the initial key frames to obtain several target key frames, including: in each of the classification results, iteratively selecting the initial key frame with the highest first probability as a candidate key frame; calculating the feature similarity between the initial key frame and the candidate key frame that has the same classification result as the candidate key frame, and removing the initial key frame whose feature similarity is greater than a preset similarity threshold. If the number of candidate key frames corresponding to each of the classification results reaches a preset number threshold or the first probability corresponding to each of the initial key frames is less than a preset probability threshold, then all selected candidate key frames are determined to be the final target key frames. Specifically, when the prediction is completed, the structural classification results and more initial key frames can be obtained. However, in real clinical scenarios, such a complex initial keyframe is not required. , so it is necessary to refine the initial key frame, and the flowchart of key frame refinement is shown in Figure 5; since the classification result indicates the structure to which each frame belongs, in each classification result, the initial key frame with the highest first probability is iteratively selected as the candidate key frame (a set of refined key frames), and then the feature similarity between the initial key frame with the same classification result as the candidate key frame and the candidate key frame is calculated, and the initial key frame with the feature similarity greater than the preset similarity threshold is removed. If the iteration termination condition is met, the candidate key frames extracted by all structural classifications (that is, the set of refined key frames) are the refined target key frames. , wherein the iteration termination condition is when the number of the candidate key frames extracted from each structural classification reaches a preset number threshold (the upper limit of the number of key frames for each category is set in advance according to the information richness of different anatomical structures) or the first probability of the remaining initial key frames is less than a preset probability threshold (such as 4). The flowchart of the key frame reduction is as follows Figure 5 shown.

[0055] In this way, by condensing the initial key frames, the number of key frames can be further reduced, ensuring that the amount of data processed in subsequent steps is minimized while retaining the most important information.

[0056] Step S103: Input each target keyframe into a preset multimodal model, extract the corresponding first visual features, and concatenate the first visual features with text features to obtain an initial text description. Iteratively predict the initial text description until a first iteration termination condition is met, and output a target text description, wherein the text features are obtained by mapping the preset prompt text to the multimodal model after word segmentation. In some embodiments, first, when the target key frame is obtained Afterwards, each target keyframe needs to be Input a preset multimodal model (such as Llava-v1.5-7b) to extract each target keyframe through the CLIP encoder The first visual feature ; Secondly, segment the preset prompt text into token sequences , and then mapped to text features through Vicuna's word embedding layer ; Then, the first visual feature Through the trainable projection layer Mapping to the language model space to obtain visual embedding and embedding vision into With text features Splice to form a joint input sequence ; Finally, the model uses the input sequence Input as the initial context and generates the first The probability distribution of , the relevant formula is: , where, is the hidden layer representation of the Input by the Vicuna decoder, and W is the output vocabulary projection matrix. The model selects the token with the highest probability based on the probability distribution, which also determines the initial text description.

[0057] For example, the prompt text may be in the format of “This image appears in X% of the video. Render a clear and concise summary of the image”.

[0058] In some embodiments, the iterative prediction of the initial text description is performed until the first iteration termination condition is met, and the target text description is output, including: based on the initial text description, generating a probability distribution value of the word segmentation by an autoregressive method, and determining a third word segmentation sequence based on the probability distribution value; continuously updating the word segmentation sequence based on the second word segmentation sequence and the initial text description until the length of the fourth word segmentation sequence is greater than a preset length threshold, and then determining the target text description based on the fourth word segmentation sequence. Specifically, after the initial text description is input into the multimodal model, the multimodal model will generate a probability distribution value of the word segmentation token by an autoregressive method, and select the word segmentation token with the largest probability according to the probability distribution value to form a third word segmentation sequence; then, for each subsequent position t, based on the token sequence generated historically and the initial text description to generate the next token The probability distribution of , select the token with the largest probability, and gradually build a complete word segmentation sequence. <eos>The iteration stops when the length of the generated fourth word segmentation sequence exceeds the preset length threshold. Finally, the decoding function of the Vicuna word segmenter converts the generated fourth word segmentation sequence into natural text, forming the final descriptive caption, which is also the target text description. It should be noted that at each iteration step, the token with the highest probability is selected to gradually construct a complete token sequence.

[0059] In some embodiments, the calculation formula for continuously updating the word segmentation sequence is: ; , where is the hidden layer representation when generating the t-th token, The model generates the current token The probability distribution of and is the word segmentation sequence token, is the initial text description, is the output vocabulary projection matrix of the model.

[0060] It should be noted that the multimodal model can be fine-tuned using LoRA (Low-Rank Adaptation) technology. LoRA uses the attention weight matrix in Vicuna Add the low-rank update matrix ΔW to it, and freeze the original weight matrix at each weight update , only ΔW is updated, thus achieving lightweight adjustments to the model, significantly reducing the number of training parameters and computing resource requirements while maintaining the model's generalization ability. The goal of multimodal model training is to minimize the captioning loss. The standard cross-entropy loss function is used to measure the difference between the sequence tokens describing the caption generated by the model and the sequence tokens describing the caption: , where is the number of samples in the training batch, For the The number of tokens of the true caption of a sample, For the The first of the true captions of the samples tokens. For the model in a given graph and before In the case of tokens, predict The token is By minimizing this loss function, the model learns to generate descriptions that are as similar as possible to the true caption, thereby improving the accuracy and clinical relevance of the caption.

[0061] In this way, detailed text descriptions are generated through iterative prediction, which ensures that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention and automatically generating high-quality text descriptions.

[0062] Step S104: input the target text description into the large language model, use the autoregressive method to gradually predict the first word segmentation sequence until the second iteration termination condition is met, and generate a corresponding structured medical report based on the second word segmentation sequence.

[0063] In some embodiments, the inputting of the target text description into the large language model includes: obtaining the time point corresponding to each target key frame, and forming a description pair with the time point and the target text description; splicing the description pair with a preset text prompt to obtain a splicing result, and inputting the splicing result into the large language model. Specifically, obtaining the time point corresponding to each target key frame , and set the key frame time point and the corresponding target text description Forming time-description pairs , where m is the number of refined keyframes extracted; the time-description pair is then concatenated with the text prompt to form a complete input string for input into the large language model. For example, when the text prompt is "Generate a medical report based on the following keyframe description:", the complete input string is "Generate a medical report based on the keyframe description: ”.

[0064] In some embodiments, the first word segmentation sequence is gradually predicted using an autoregressive method. Specifically, the word segmenter of the large language model is used to convert the complete input string into a token sequence, and input it into the word embedding layer to obtain the corresponding embedding vector representation. , where the word embedding layer maps each token to a vector space of fixed dimension, for example, each token is mapped to a 768-dimensional vector, so It is a matrix whose number of rows is equal to the length of the token sequence and the number of columns is equal to the dimension of the embedding vector. After that, the large language model will iteratively predict the next token based on the input token sequence. For each position t, the model will predict the next token based on the previously generated token. and the text prompt and time-description pair entered , calculate the next token The probability distribution of Then select the token with the highest probability as the output word segmentation sequence .

[0065] In some embodiments, until the second iteration termination condition is met, specifically, when the generated <eos>(terminator) or when the length of the first word segmentation sequence reaches the maximum length, the iteration stops, wherein the maximum length can be preset in advance and is not limited in this application.

[0066] In some embodiments, a corresponding structured medical report is generated based on the second word segmentation sequence. Specifically, the generated token sequence, i.e., the second word segmentation sequence, is converted into natural text through a decoder of a large language model to form a final structured medical report, ensuring that the report content is accurate, coherent and meets clinical needs.

[0067] It should be noted that this stage uses the Llama-7B model to achieve the conversion of text captions to structured medical reports. The fine-tuning process is to collect the real key frame descriptions of each patient's key frame time as training input samples, and the corresponding structured reports as training output samples. When training the Llama-7B model, the goal is to minimize the cross entropy loss function to maximize the probability of the output sequence, thereby obtaining a large language model. The loss function is defined as: , where The total number of tokens reported in the summary. For the true report tokens, is the embedding vector representation of the complete input text.

[0068] It's important to note that to improve the model's robustness in real-world scenarios, the training data was augmented using the following strategies to simulate redundant captions and temporal deviations that may occur during testing: 1. Increasing the number of captions: This approach increases the number of input captions by duplicating existing captions or generating similar descriptions. 2. Randomly adjusting time: This approach randomly shifts the time of each keyframe forward or backward to generate new time points. These strategies enable the model to better handle uncertainty and redundant information in real-world scenarios, ensuring that generated reports are accurate and coherent.

[0069] The embodiment of the present application can retain the key information in the video by eliminating redundant frames of the capsule endoscopy video, while avoiding repeated calculation of repeated or meaningless frames, thereby improving the subsequent processing speed and accuracy; by identifying the initial key frames that may contain important information through the detection model, it is possible to initially obtain relevant frames, which is convenient for subsequent analysis, and the classification results can help distinguish different types of anatomical structures or lesions, thereby ensuring that only the most relevant and representative key frames are retained; by refining the initial key frames, the number of key frames can be further reduced, ensuring that the amount of data processed in subsequent steps is minimized while retaining the most important information; by The first visual feature is spliced with the text feature to combine visual and textual information to generate richer and more accurate descriptions, thereby enhancing the clinical value of the report; generating detailed text descriptions through iterative prediction can ensure that the visual content of each key frame is accurately converted into language information, thereby avoiding human intervention, automatically generating high-quality text descriptions, and improving the speed and consistency of report generation; gradually constructing the report through the autoregressive method can integrate the text descriptions of all key frames into a coherent and structured medical report, ensuring that the report content is complete and the logic is clear. Doctors can directly use the generated report for diagnosis without manual organization and writing, saving time and energy. Compared with the existing technology, this application can automatically generate diagnostic reports and improve the efficiency and quality of diagnosis.

[0070] Example 2 Please refer to Figure 6 , Figure 6 This application provides a capsule endoscopy-based medical report automatic generation system, comprising: an acquisition module 100, a processing module 200, a first prediction module 300 and a second prediction module 400; The acquisition module 100 is used to acquire capsule endoscopy video and remove redundant frames of the capsule endoscopy video to obtain a plurality of representative frames; The processing module 200 is configured to input each representative frame into a preset detection model, determine an initial key frame and a classification result, and refine the initial key frame using the classification result to obtain a plurality of target key frames; The first prediction module 300 is configured to input each target keyframe into a preset multimodal model, extract corresponding first visual features, and concatenate the first visual features with text features to obtain an initial text description, iteratively predict the initial text description until a first iteration termination condition is satisfied, and output a target text description, wherein the text features are obtained by mapping the preset prompt text to the multimodal model after word segmentation; The second prediction module 400 is used to input the target text description into the large language model, use the autoregressive method to gradually predict the first word segmentation sequence until the second iteration termination condition is met, and generate a corresponding structured medical report based on the second word segmentation sequence.

[0071] The information interaction, execution process, etc. between the modules in the above-mentioned automatic generation system of medical reports based on capsule endoscopy are based on the same concept as the embodiment of the automatic generation method of medical reports based on capsule endoscopy in the first aspect of the present invention, and the technical effects achieved are basically the same. For specific contents, please refer to the description in the first embodiment of the method of the present invention, and will not be repeated here.

[0072] The apparatus embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, i.e., they may be located in one location or distributed across multiple network elements. Some or all of these elements may be selected based on actual needs to achieve the objectives of the methods of this embodiment.

[0073] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-monitorable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0074] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application.

[0075] It is particularly pointed out that for those skilled in the art, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.< / eos> < / eos>

Claims

1. A method for automatically generating medical reports based on capsule endoscopy, characterized in that: include: Acquire a capsule endoscopy video, and remove redundant frames of the capsule endoscopy video to obtain a plurality of representative frames; Inputting each representative frame into a preset detection model to determine an initial key frame and a classification result, and using the classification result to refine the initial key frame to obtain a plurality of target key frames; Inputting each target keyframe into a preset multimodal model, extracting corresponding first visual features, and concatenating the first visual features with text features to obtain an initial text description, iteratively predicting the initial text description until a first iteration termination condition is satisfied, and outputting a target text description, wherein the text features are obtained by mapping the preset prompt text to the multimodal model after word segmentation; The target text description is input into a large language model, and the first word segmentation sequence is gradually predicted using an autoregressive method until the second iteration termination condition is met, and a corresponding structured medical report is generated based on the second word segmentation sequence.

2. The method for automatically generating medical reports based on capsule endoscopy according to claim 1, characterized in that: The redundant frames of the capsule endoscopy video are removed to obtain several representative frames, specifically: Inputting the capsule endoscopy video into a preset analysis model to obtain a second visual feature corresponding to each frame image in the capsule endoscopy video; The similarity of the second visual features of adjacent frames is calculated, a number of clusters are determined based on a comparison result of the similarity with a preset similarity threshold, and a representative frame is determined based on each of the clusters.

3. The method for automatically generating medical reports based on capsule endoscopy according to claim 1, characterized in that: The representative frames are input into a preset detection model to determine the initial key frames and classification results, specifically: Inputting each of the representative frames into a preset detection model to determine a first probability and a second probability; Comparing the first probability with a preset probability threshold, and if the probability is greater than the predetermined probability, determining an initial key frame; A classification result is determined based on the second probability.

4. The method for automatically generating medical reports based on capsule endoscopy according to claim 3, characterized in that: Inputting each representative frame into a preset detection model to determine the first probability and the second probability is specifically as follows: Inputting each representative frame into a preset detection model to extract features from each representative frame to obtain corresponding semantic features, wherein the detection model includes a spatiotemporal feature extraction network, a key frame detection head, and a structure classification head; Fusing the semantic features with the position encoding results to obtain fused features, and inputting the fused features into the spatiotemporal feature extraction network to obtain deep features, wherein the position encoding results are obtained by position encoding each of the representative frames; The deep features are input into the key frame detection head and the structure classification head respectively to obtain a first probability and a second probability.

5. The method for automatically generating medical reports based on capsule endoscopy according to claim 1, characterized in that: The training process of the detection model is specifically as follows: Extracting features corresponding to the representative frame using a preset feature encoder, and normalizing the features to obtain frame features; Calculating the cosine similarity between the frame features corresponding to each of the representative frames to obtain a similarity matrix, and amplifying the predetermined reference key frame based on the similarity matrix and a preset video segment true value to obtain an amplified key frame; The key frame detection network is supervisedly trained using the amplified key frames and preset structure segmentation labels until the loss function is minimized, thereby determining the detection model.

6. The method for automatically generating medical reports based on capsule endoscopy according to claim 3, characterized in that: The initial key frames are refined using the classification results to obtain several target key frames, specifically: Iteratively selecting the initial key frame with the highest first probability from among the classification results as a candidate key frame; Calculate the feature similarity between the initial key frame and the candidate key frame that is the same as the classification result of the candidate key frame, and remove the initial key frame whose feature similarity is greater than a preset similarity threshold. If the number of the candidate key frames corresponding to each of the classification results reaches a preset number threshold or the first probability corresponding to each of the initial key frames is less than a preset probability threshold, determine all the selected candidate key frames as the final target key frames.

7. The method for automatically generating medical reports based on capsule endoscopy according to claim 1, characterized in that: The iterative prediction of the initial text description is performed until a first iteration termination condition is satisfied, and the target text description is output, specifically: Based on the initial text description, generating a probability distribution value of word segmentation by an autoregressive method, and determining a third word segmentation sequence based on the probability distribution value; The word segmentation sequence is continuously updated based on the second word segmentation sequence and the initial text description until the length of the fourth word segmentation sequence is greater than a preset length threshold, and the target text description is determined based on the fourth word segmentation sequence.

8. The method for automatically generating medical reports based on capsule endoscopy according to claim 1, characterized in that: The inputting the target text description into the large language model is specifically as follows: Obtaining a time point corresponding to each target keyframe, and forming a description pair with the time point and the target text description; The description pair is spliced with a preset text prompt to obtain a splicing result, and the splicing result is input into a large language model.

9. The method for automatically generating medical reports based on capsule endoscopy according to claim 1, characterized in that: The generating of a corresponding structured medical report based on the first word segmentation sequence is specifically as follows: Decoding the first word segmentation sequence to obtain a first text; Extract key information from the first text, and fill the key information into a preset report template to obtain a corresponding structured medical report.

10. A capsule endoscopy-based medical report automatic generation system, characterized in that: include: an acquisition module, a processing module, a first prediction module and a second prediction module; The acquisition module is used to acquire capsule endoscopy video and remove redundant frames of the capsule endoscopy video to obtain a plurality of representative frames; The processing module is configured to input each representative frame into a preset detection model, determine an initial key frame and a classification result, and refine the initial key frame using the classification result to obtain a plurality of target key frames; The first prediction module is configured to input each target keyframe into a preset multimodal model, extract corresponding first visual features, and concatenate the first visual features with text features to obtain an initial text description, iteratively predict the initial text description until a first iteration termination condition is satisfied, and output a target text description, wherein the text features are obtained by mapping the preset prompt text to the multimodal model after word segmentation; The second prediction module is used to input the target text description into the large language model, use the autoregressive method to gradually predict the first word segmentation sequence until the second iteration termination condition is met, and generate a corresponding structured medical report based on the second word segmentation sequence.

Images