Endoscope report generation method, device and equipment and readable storage medium

Abstract

The invention discloses an endoscope report generation method, device and equipment and a readable storage medium, and is applied to the technical field of computer vision and deep learning, and the method comprises the steps: obtaining an endoscope real-time video stream, and determining a target inspection type based on the time sequence characteristics, spatial characteristics and motion characteristics of the endoscope real-time video stream; determining a global feature and a local feature of the endoscope real-time video stream, splicing the global feature and the local feature, and determining a target inspection part based on a spliced feature obtained by splicing; performing real-time lesion detection on the endoscope real-time video stream to obtain a lesion area; and generating an examination report based on the target examination type, the target examination part and the lesion area to obtain a target endoscope report. As the doctor does not need to manually determine each index generation report, the endoscope report generation efficiency can be improved, and the target examination part can be determined based on the global features and the local features, so that a plurality of examination parts can be accurately determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to an endoscope report generation method, device, equipment and readable storage medium. BACKGROUND

[0002] Endoscopy is an important minimally invasive diagnostic and treatment method. Currently, when performing endoscopic diagnosis, by inserting a long tube with a camera into the body cavity, the doctor can directly observe the internal condition of the organ and find and diagnose various lesions. The problem of the scheme of generating a report by manually observing and determining the result is that the doctor is easy to be tired after long time observation, the attention is difficult to be continuously concentrated, the sensitivity to subtle lesions is reduced, and the recognition ability of novice doctors is weak. Different doctors' diagnosis standards are difficult to unify, personal experience difference is large, and the training period of a professional doctor is long.

[0003] Therefore, the current technical problem is that the endoscopy report generation efficiency is low, and how to improve the efficiency of endoscopy report generation is an urgent technical problem for those skilled in the art. SUMMARY

[0004] Therefore, the present application provides an endoscope report generation method, device, equipment and readable storage medium, which solves the problem of low efficiency of endoscopy report generation in the prior art.

[0005] To solve the above technical problems, the present application provides an endoscope report generation method, comprising:

[0006] Obtaining an endoscope real-time video stream, determining a target examination type based on the time sequence feature, the spatial feature and the motion feature of the endoscope real-time video stream;

[0007] Determining the global feature and the local feature of the endoscope real-time video stream, and splicing the global feature and the local feature, determining a target examination site based on the spliced feature obtained by splicing;

[0008] Performing real-time lesion detection on the endoscope real-time video stream to obtain a lesion area;

[0009] Generating an examination report based on the target examination type, the target examination site and the lesion area to obtain a target endoscope report.

[0010] Optionally, before generating the target endoscope report by generating an examination report based on the target examination type, the target examination site and the lesion area, the method further comprises:

[0011] determine a comprehensive key parameter of the picture frame based on color abnormality, picture frame quality and key frame of the lesion of the picture frame in the endoscope real-time video stream, and determine the key frame in the endoscope real-time video stream based on the comprehensive key parameter;

[0012] monitoring the integrity of the examination is performed during the examination based on the endoscope, and an examination integrity parameter is obtained; wherein the examination integrity parameter is determined by the examination time length, image quality and coverage area of each examination point, and the weight corresponding to each parameter;

[0013] Correspondingly, based on the target examination type, the target examination site, and the lesion area, an examination report is generated to obtain a target endoscope report, including:

[0014] Based on the target examination type, the target examination site, the lesion area, the key frame and the examination integrity parameter, an examination report is generated to obtain the target endoscope report.

[0015] Optionally, the endoscope real-time video stream is detected in real time to obtain a lesion area, including:

[0016] Feature extraction is performed on the endoscope real-time video stream, and all extracted features are fused to obtain multi-scale fusion features;

[0017] Based on the multi-scale fusion features, a plurality of lesion prediction boxes are recognized, each including a bounding box coordinate, a confidence and classification information;

[0018] The lesion area is output based on the lesion prediction box; the lesion area includes a lesion position and a lesion classification result.

[0019] Optionally, based on the target examination type, the target examination site, and the lesion area, an examination report is generated to obtain a target endoscope report, including:

[0020] Based on the target examination type, the target examination site, and the lesion area, a large language model and its corresponding prompt word template are used to obtain the target endoscope report.

[0021] Optionally, after generating the target endoscope report based on the target examination type, the target examination site and the lesion area, the method further includes:

[0022] Based on the text and image in the target endoscope report, joint learning is performed to determine a timing feature;

[0023] Based on the timing feature, a multi-modal prediction is performed to obtain a diagnosis auxiliary disease prediction result.

[0024] Optionally, the method further comprises:

[0025] analyzing the endoscope real-time video stream by using a detection model to determine a suspicious detection event;

[0026] outputting an index and a highlight set according to the suspicious detection event; wherein the index is converting the endoscope real-time video stream into a structured index table, and the highlight set includes an abnormal time point and a related description.

[0027] The endoscope report generation device provided by the embodiment of the application comprises:

[0028] a target detection type determination module configured to acquire an endoscope real-time video stream, and determine a target detection type based on time sequence features, spatial features and motion features of the endoscope real-time video stream;

[0029] a target detection site determination module configured to determine global features and local features of the endoscope real-time video stream, splice the global features and the local features, and determine a target detection site based on spliced features obtained by splicing;

[0030] a lesion area determination module configured to perform real-time detection of lesions on the endoscope real-time video stream to obtain a lesion area;

[0031] a target endoscope report generation module configured to generate a detection report based on the target detection type, the target detection site and the lesion area to obtain a target endoscope report.

[0032] Optionally, the endoscope report generation device further comprises:

[0033] a time sequence feature determination module configured to jointly learn based on text and images in the target endoscope report to determine time sequence features;

[0034] a diagnosis auxiliary disease prediction result determination module configured to perform multi-modal prediction based on the time sequence features to obtain a diagnosis auxiliary disease prediction result.

[0035] The endoscope report generation device provided by the embodiment of the application comprises:

[0036] a memory configured to store a computer program;

[0037] a processor configured to execute the computer program to implement steps of the endoscope report generation method.

[0038] The embodiment of the application further provides a computer readable storage medium, the computer readable storage medium stores the computer program.

[0039] A computer program is stored in the storage, and the computer program is executed by the processor to implement the steps of the endoscope report generation method.

[0040] The embodiment of the present application also provides a computer program product, comprising computer programs / instructions, which are executed by a processor to implement the steps of the hardware description language generation method.

[0041] It can be seen that, by acquiring the endoscope real-time video stream, determining the target examination type based on the time sequence feature, the spatial feature and the motion feature of the endoscope real-time video stream, determining the global feature and the local feature of the endoscope real-time video stream, splicing the global feature and the local feature, determining the target examination part based on the spliced feature, performing real-time detection of lesions on the endoscope real-time video stream to obtain a lesion area, and generating an examination report based on the target examination type, the target examination part and the lesion area to obtain a target endoscope report, compared with the current endoscope examination which is completely observed by a doctor manually and the endoscope report is generated, the efficiency of the endoscope report generation can be improved since the doctor does not need to manually determine each index to generate the report, and the comprehensiveness of the endoscope report generation can be improved since the target examination part is determined based on the global feature and the local feature, so that the multiple examination parts can be accurately determined, and the comprehensiveness of the endoscope examination part determination is improved.

[0042] In addition, the present application also provides an endoscope report generation device, equipment and readable storage medium, which also have the above beneficial effects. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0044] Figure 1 A flowchart of the endoscope report generation provided by the embodiment of the present application is provided.

[0045] Figure 2 A front and back comparison chart of the edge removal operation provided by the embodiment of the present application is provided.

[0046] Figure 3 A flowchart of an endoscope report generation method provided by an embodiment of the present application is shown in FIG. 1.

[0047] Figure 4 A structural schematic diagram of an endoscope report generation system provided by an embodiment of the present application is shown in FIG. 2.

[0048] Figure 5 A structural schematic diagram of an endoscope report generation device provided by an embodiment of the present application is shown in FIG. 3.

[0049] Figure 6 A structural schematic diagram of an endoscope report generation device provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0051] Reference is made to Figure 1 , Figure 1 A flowchart of an endoscope report generation method provided by an embodiment of the present application is shown in FIG. 1. The method can include:

[0052] S101, acquiring an endoscope real-time video stream, and determining a target examination type based on time sequence features, spatial features and motion features of the endoscope real-time video stream.

[0053] The execution subject of this embodiment can be an electronic device. The electronic device in this embodiment can be a computer, a mobile phone, etc. The endoscope real-time video stream in this embodiment is a video stream obtained by a doctor based on an endoscope during examination. The target examination type in this embodiment can include: digestive tract endoscopes: enteroscope, gastroscope, cholangioscope; respiratory endoscopes: bronchoscope, nasopharyngoscope; cystoscope, hysteroscope, etc. The examination type recognition in this embodiment is performed by using an endoscope real-time video stream analysis method: input data: the endoscope real-time video stream is V(t), t∈[0,T], where T is the total examination time length. Feature extraction: feature extraction is performed on the endoscope real-time video stream to obtain a multi-dimensional feature vector F, including: a time sequence feature vector f t by analyzing the change law between consecutive frames; a spatial feature vector extracting spatial structure information in a single frame of image, including tissue texture, edge; a motion feature vector , analyze the motion trajectory and speed characteristics of the device. . Type judgment: use a deep learning model to check the type judgment, that is argmax represents the independent variable when the function reaches the maximum value, that is, among all possible inspection types y, find the type that can make the probability P(y|F) maximum, P(y|F) represents the conditional probability, that is, under the condition of observing the feature F, the probability that the model judges the inspection type y. The above process of identifying the inspection type can include: collecting complete inspection video streams from multiple endoscopy cases, covering different types (such as gastroscopy and colonoscopy), labeling each video with a type label (the whole video is "gastroscopy" or "colonoscopy"), and slicing each video according to a fixed time window (such as 5 seconds) to form training samples. Process the video frame sequence in each time window, extract three types of features: spatial features: use CNN (convolutional neural network) to extract texture, color, and other tissue structure information of single-frame images; time sequence features: use LSTM (long short-term memory network) to process frame sequences to capture motion rules and image evolution trends; motion features: calculate lens movement trajectory, speed, and other dynamic information through optical flow estimation; concatenate the above features into a unified multi-dimensional vector F for subsequent classification tasks. The model uses a time sequence encoding + multi-feature fusion + classification head structure; CNN processes single frames, LSTM encodes the time sequence information of frame sequences, and motion features are integrated through a lightweight MLP (multilayer perceptron), and finally output the inspection type prediction probability.

[0054] S102, determine the global feature and local feature of the endoscope real-time video stream, and concatenate the global feature and the local feature, and determine the target inspection site based on the concatenated feature.

[0055] The global feature in this embodiment refers to a feature representation extracted from the entire video frame or a larger region thereof, reflecting the overall structure, layout, profile and macro context; the local feature in this embodiment refers to a feature representation extracted from a relatively small region of the image (such as an image block or a local receptive field region on a feature map), reflecting the detailed texture, edge, microstructure, specific pattern. In this embodiment, a pre-trained EfficientNet (Efficient Network) can be used as a branch to extract the convolutional features of the local image. At the same time, a pre-trained ViT is used as another branch to extract the global attention features of the image using the Encoder. ViT is a model that innovatively migrates the Transformer architecture in natural language processing (originally used for processing sequence data such as text) to image recognition tasks. For ease of understanding, this embodiment can combine the structure based on the self-attention mechanism and the lightweight neural network model to extract image features and determine the target examination site, wherein the self-attention (self-attention) is a mechanism of neural network, which is used to calculate the importance of the association between elements in a sequence, and is widely used in natural language processing. The famous Transformer is a neural network architecture based on the self-attention mechanism, which is originally applied to language models and does not use convolutional neural networks at all. The self-attention mechanism used in the patent comes from the vision transformer (ViT), a variant of the Transformer model, which mainly divides the input image into many small blocks and calculates the relevance between these small blocks through some convolution operations. The attention mechanism in the patent is mainly reflected in the checkpoint classification model in the enteroscope. The checkpoint classification model combines the Encoder of ViT and EfficientNet, extracts image features through ViT and EfficientNet respectively, and finally connects the feature vectors of the image in a splicing manner as the final classification features of the image. Use multiple fully connected layers to classify the categories of the image (which examination point, rectum, colon, ileocecal valve, etc.). The training process of the structure based on the self-attention mechanism and the lightweight neural network model can include: picture preprocessing, first remove the black border in the picture returned by the endoscope video stream. For ease of understanding, please refer to Figure 2 , Figure 2A front and back comparison chart of a side removal operation provided by an embodiment of the present application. Before performing examination site identification, the embodiment can uniformly input the picture size, and uniformly adjust to the size of 224 pixels by 224 pixels through the Resize() method. The feature extraction network construction of the model for identifying the examination site can include: using a pre-trained EfficientNet as a branch for extracting the convolutional features of the image. At the same time, a pre-trained ViT is used as another branch to extract the global attention features of the image. The output features of the two models are spliced in the feature dimension to form a joint feature representation. The feature vectors extracted by the above two branches are spliced in the channel dimension (the output features of the two models are spliced in the feature dimension). The spliced features are input to a multi-layer perceptron (MLP) classifier, which usually includes one or more fully connected layers, ReLU activation functions and Dropout (Dropout) layers. The dimension of the output of the last layer is consistent with the number of categories, and the prediction probability of each category is obtained through Softmax. The cross-entropy loss function can be used as the objective function during the training process. The Adam optimizer is used for back propagation and parameter update, and the initial learning rate is set to 1e-4. After the training is completed, the optimal model weight is selected according to the validation set performance.

[0056] S103, real-time lesion detection is performed on the endoscope real-time video stream to obtain a lesion region.

[0057] The embodiment is not limited to a specific model for real-time lesion detection. For example, the model for real-time lesion detection can use a YOLOv11 (single-stage detection, end-to-end inference model) model for target detection. The structure of the YOLOv11 model includes: (1) an input layer: after the image is normalized and preprocessed, it is input into the YOLOv11 model; (2) a feature extraction layer: feature extraction is performed based on a CSPDarknet (cross-stage local darknet network), EfficientNet, etc. backbone network; (3) a neck network: an FPN (feature pyramid network) or PANet (path aggregation network) structure is used for multi-scale feature fusion; an output layer: a plurality of prediction boxes are generated, each box includes a bounding box coordinate, a confidence and classification information, and an abnormal position and classification result of a lesion point are output.

[0058] It should be further explained that, in order to improve the accuracy of the determination of the lesion region, the real-time lesion detection on the endoscope real-time video stream to obtain the lesion region can include:

[0059] S1031, feature extraction is performed on the endoscope real-time video stream, and multi-scale feature fusion is performed on all the extracted features to obtain multi-scale fusion features;

[0060] S1032, performing recognition based on the multi-scale fusion features to obtain a plurality of lesion prediction boxes, each of which includes a bounding box coordinate, a confidence and classification information;

[0061] S1033, outputting a lesion region based on the lesion prediction box; the lesion region includes a lesion position and a lesion classification result.

[0062] The spatial features in this embodiment reveal a single frame of anatomical structure, the temporal features model the pathological dynamic evolution, and the motion features quantify the operator's operation method (fast zoom-out / shaking), and the three are complementary to overcome the limitations of a single modality.

[0063] S104, generating an endoscopy report based on the target examination type, the target examination site and the lesion region.

[0064] In this embodiment, the endoscopy report can be generated based on a large model when generating an examination report based on the target examination type, the target examination site and the lesion region. The content of the endoscopy report in this embodiment can include: basic information, examination type, examination time, device information and operating physician. The examination process record includes the examination path, the examination time of each part and the examination integrity. The lesion information record includes the position description of the lesion, the morphological feature, the size of the lesion and the nature. The target endoscopy report D=Generate(Fimg, Floc, Fmeas); wherein Fimg is the feature of the picture in the examination, Floc is the feature of the examination point in the examination, and Fmeas is the feature of the measured lesion size. This embodiment can perform atlas analysis based on the pathological linkage knowledge base to make the report have medical logical coherence. In the process of examination, before outputting the report, this embodiment can automatically analyze the situation of blurred, overexposed, occluded and the like of the photographed picture, and propose a retake suggestion, improve the picture quality consistency, and effectively reduce the information loss risk. According to the known anatomical structure changes and the photographed picture, the current part where the field of view is located is indicated in real time, and the progress bar and the coverage of the examined area are displayed synchronously to prevent omission. At the same time, through the error sample output and the new result feedback, the model (all models) is automatically enhanced and learned to improve the generalization ability of the system. A multi-language interface, a custom shortcut operation key and a voice control auxiliary are provided to reduce the operation threshold. This embodiment is not limited to a specific target endoscopy report, for example, the target endoscopy report is:

[0065] Findings: The patient was placed in a left lateral position, and after successful intravenous anesthesia, the cavity was entered and the scope was inserted 80 cm to the ileocecal junction, 5 cm into the terminal ileum, and no abnormalities were found. The ileocecal junction showed smooth mucosa, normal color, clear vascular texture, clear appendiceal opening, and ileocecal valve lip shape.

[0066] Retrospective observation: There are more loose stool water in the intestinal tract, and part of the mucosa cannot be observed carefully. Two 4-5mm mucosal elevations are visible in the ascending colon, and the surface is smooth. A snare cold ring is removed, and the tissue is recovered for pathology. About 6cm from the anus, a 6mm mucosal elevation is visible in the rectum, and the surface is smooth. A snare cold ring is removed, and the tissue is recovered for pathology. About 5cm from the teeth, a 5mm mucosal elevation is visible in the rectum, and the surface is smooth. Blood vessels can be seen crawling, and the lesion is lifted by 1:10000 XXX injection. The lifting sign is positive, and the lesion is removed by snare high-frequency electrocoagulation. Three harmony clips are placed to close the wound, and the tissue is recovered for pathology. The rest of the transverse colon, descending colon, sigmoid colon, and rectum mucosa is smooth, with normal color and clear vascular texture.

[0067] Conclusion: Multiple colorectal polyps + cold resection; rectal NET + high-frequency electroresection.

[0068] It should be further explained that the target endoscopic report can be obtained based on the target examination type, target examination site and lesion area. The target endoscopic report can be obtained by using a large language model and its corresponding prompt word template based on the target examination type, target examination site and lesion area. This embodiment can include a heat map in the target endoscopic report. By analyzing the moving track of the lens and the distribution of the observation area in the endoscopic video, it is determined whether all the areas to be examined are sufficiently examined, and the heat map is used to visualize the "where and how long it is observed". Real-time data collection includes: each frame of image (frame and timestamp), direction and speed of lens movement (derived from inter-frame motion), image center / boundary of each frame (representing the current observation area). The lens displacement is estimated by a feature matching algorithm. When constructing the coverage heat map, the number of frames in which each area appears is calculated, which can estimate how long the doctor stays in a certain area, and through feature matching calculation, the number of frames in which the overlapping area is observed can be calculated, and a visual coverage heat map can be made. The average coverage rate, the number of missed scanning areas, the minimum dwell time and the lens repetition rate can be calculated, and the quality control score can be calculated to give the quality of this examination in the target endoscopic report.

[0069] It needs to be further explained that based on any of the above embodiments, before the target endoscopic report is generated based on the target examination type, the target examination site, and the lesion area, the following steps can also be included: determining the comprehensive key parameters of the picture frames based on the color abnormality degree, the picture frame quality, and the lesion key frames in the endoscopic real-time video stream, and determining the key frames in the endoscopic real-time video stream based on the comprehensive key parameters; monitoring the examination integrity during the examination based on the endoscope to obtain an examination integrity parameter; wherein the examination integrity parameter is determined by the examination time length, the image quality, and the coverage area of each examination point, and the weights corresponding to each parameter; accordingly, the target endoscopic report is generated based on the target examination type, the target examination site, and the lesion area, and the target endoscopic report is obtained, which can include: generating the target endoscopic report based on the target examination type, the target examination site, the lesion area, the key frames, and the examination integrity parameter. The process of determining the comprehensive key parameters of the picture frames based on the color abnormality degree, the picture frame quality, and the lesion key frames in the endoscopic real-time video stream, and determining the key frames in the endoscopic real-time video stream based on the comprehensive key parameters can include the determination of the color abnormality degree: , Ω is the color abnormality; c is the current frame HSV color vector, c ref is the normal tissue color vector; the color abnormality degree is calculated using the Euclidean distance. Picture frame quality screening: , is the sharpness score, which is calculated based on image gradient, is the focus score, which is calculated based on frequency domain analysis, is the stability score, which is calculated based on inter-frame motion estimation, is the illumination score, which is calculated based on light distribution evaluation. is the adaptive weight coefficient of the model. In summary, the frame quality is evaluated by combining multiple quality indicators. Lesion key frame capture: morphological feature vector, where A is the lesion area, P is the perimeter, C is the circularity, E is the eccentricity, and S is the symmetry, the geometric features of the lesion area are extracted. Texture feature vector: where is the gray level histogram feature, LBP is the local binary pattern, and GLCM is the gray level co-occurrence matrix feature, the feature vector is analyzed by multi-scale texture feature analysis. Comprehensive feature vector F: key frame selection strategy where S is the comprehensive score of key frame selection, , and are the weights of the three indicators corresponding to the deep learning model, if S is greater than a set threshold, the key frame is selected.

[0070] The examination integrity monitoring in the process of endoscopic examination in this embodiment can include: checkpoint management: generating an examination checklist according to the target examination type; examination progress tracking: recording the examined parts; calculating the examination time; evaluating the examination quality; marking the unexamined areas. Omission monitoring: monitoring whether each checkpoint is examined and photographed; calculating the integrity score for each checkpoint p (combining the time point with the requirement of the part examination time, and each examination part will not exceed the set time prepayment): ; wherein T(p) is the examination time score, Q(p) is the image quality score, A(p) is the coverage area score, 、 、 are weight coefficients, respectively.

[0071] It should be further pointed out that based on any of the above embodiments, in order to assist doctors in improving disease diagnosis, after the above target endoscopic report is generated based on the target examination type, the target examination part and the lesion area, joint learning based on the text and images in the target endoscopic report can be performed to determine the timing feature; multi-modal prediction based on the timing feature can be performed to obtain a diagnosis auxiliary disease prediction result. This embodiment can combine the text and image data in the target endoscopic report for disease diagnosis, use a multi-modal prediction method, capture timing features based on LSTM (Long Short Term Memory Network) or RNN (Recurrent Neural Network), and output corresponding disease prediction results. Specifically, a LLama large language model can be used for multi-modal disease prediction, the LLama model is based on joint learning of image data and text description, uses a deep neural network for timing modeling and context understanding, and outputs disease categories, including but not limited to benign and malignant polyps, whether there are signs of cancer, etc. Diagnosis suggestions related to the patient's condition can also be provided according to the adjustment of the weight of different input information. At the end of each examination, the scene pictures collected during the examination and the corresponding prediction results are stored in a database or a picture-text system, and doctors can view the visualized existing model annotation results from the backend webpage, and modify or confirm the results. Every week or every month, the system adds the results confirmed by the doctors to the original training data for model training, thereby improving the performance of the model. If the performance of the model is improved, the current model weight is updated. This embodiment can assist in disease diagnosis, improving the user experience.

[0072] It needs to be further explained that based on any of the above embodiments, in order to improve the accuracy of the examination, the above endoscope report generation method can also include: analyzing the endoscope real-time video stream by using a detection model to determine a suspicious detection event; outputting an index and a highlight set according to the suspicious detection event; wherein the index is to convert the endoscope real-time video stream into a structured index table, and the highlight set includes an abnormal time point and a related description. The detection model in this embodiment can be based on YOLO. The index in this embodiment is like turning the entire examination into a structured index table for the doctor to quickly view the key segments: the highlight set is like outputting the time point when the anomaly occurs and the related description, for example: time point: 00:01:14; event: automatic detection of polyp, detection confidence 93%; screenshot: 012.jpg; description: about 5mm protrusion is found in the ascending colon, the edge is clear, the surrounding mucosa is smooth at this time, and the blood vessel texture is clear. The index and the highlight set in this embodiment can help the doctor review the examination process and locate.

[0073] The endoscope report generation method provided by the embodiment of the present application can include: S101, acquiring an endoscope real-time video stream, determining a target examination type based on the time sequence feature, the spatial feature and the motion feature of the endoscope real-time video stream; S102, determining the global feature and the local feature of the endoscope real-time video stream, and splicing the global feature and the local feature, determining a target examination site based on the spliced feature obtained by splicing; S103, performing real-time detection of lesions on the endoscope real-time video stream to obtain a lesion area; S104, generating an examination report based on the target examination type, the target examination site and the lesion area to obtain a target endoscope report. Compared with the current endoscope examination which is completely observed by a doctor and the endoscope report is generated by the doctor, the present application determines a target examination type based on the time sequence feature, the spatial feature and the motion feature of the endoscope real-time video stream; determines the global feature and the local feature of the endoscope real-time video stream, and splices the global feature and the local feature, determines a target examination site based on the spliced feature obtained by splicing; performs real-time detection of lesions on the endoscope real-time video stream to obtain a lesion area; generates an examination report based on the target examination type, the target examination site and the lesion area to obtain a target endoscope report. Since the doctor does not need to manually determine each index to generate a report, the efficiency of endoscope report generation can be improved, and since the target examination site is determined based on the global feature and the local feature, multiple examination sites can be accurately determined, the comprehensiveness of endoscope detection site determination is improved, and thus the comprehensiveness of endoscope report generation is improved.

[0074] In order to make the present application more convenient to understand, please refer to Figure 3 , Figure 3 The flowchart of the endoscope report generation method provided by the embodiment of the present application can specifically include:

[0075] S201, real-time acquisition is performed on an endoscopy process to obtain an endoscopy real-time video stream; the endoscopy real-time video stream is composed of picture frames.

[0076] S202, a target examination type is determined based on a time sequence feature, a space feature and a motion feature of the endoscopy real-time video stream.

[0077] S203, global features and local features of the endoscopy real-time video stream are recognized based on an improved neural network, and the global features and the local features are spliced to determine a target examination part based on spliced features obtained by the splicing.

[0078] The improved neural network in this embodiment includes an Encoder (encoder) and an EfficientNet (efficient network).

[0079] S204, real-time detection is performed on the endoscopy real-time video stream to obtain a lesion area.

[0080] S205, a comprehensive key parameter of a picture frame is determined based on a color abnormality degree of the picture frame, a picture frame quality and a lesion key frame in the endoscopy real-time video stream, and a key frame in the endoscopy real-time video stream is determined based on the comprehensive key parameter.

[0081] S206, a target endoscopy report is generated based on a large model based on the target examination type, the target examination part, the lesion area and the key frame.

[0082] S207, picture and text information in the target endoscopy report are taken as multi-modal input information, and input to a large model for multi-modal prediction to obtain a diagnosis auxiliary disease prediction result.

[0083] When the endoscopy report is generated in the embodiment of the application, the picture and the text with context connection can be taken as multi-modal input information, and then multi-modal prediction is performed, the accuracy of disease prediction is improved, and the experience of a user can be improved.

[0084] Please refer to Figure 4 , Figure 4 A structure diagram of an endoscopy report generation system provided by the embodiment of the application can include an endoscopy handle, an endoscopy host, a video acquisition card and an AI system server, wherein the AI system server is a server for generating an endoscopy report based on the above endoscopy report generation method.

[0085] The endoscopy report generation device provided by the embodiment of the application is described below, and the endoscopy report generation device described below can be correspondingly referred to the endoscopy report generation method described above.

[0086] Please refer to Figure 5 ,Figure 5 A structural diagram of an endoscope report generation device provided by an embodiment of the present application can include:

[0087] A target detection type determination module 100 is configured to acquire an endoscope real-time video stream, determine a target examination type based on time sequence features, spatial features and motion features of the endoscope real-time video stream;

[0088] A target examination site determination module 200 is configured to determine global features and local features of the endoscope real-time video stream, splice the global features and the local features, and determine a target examination site based on spliced features obtained by splicing;

[0089] A lesion area determination module 300 is configured to perform real-time lesion detection on the endoscope real-time video stream to obtain a lesion area;

[0090] A target endoscope report generation module 400 is configured to generate an examination report based on the target examination type, the target examination site and the lesion area to obtain a target endoscope report.

[0091] Further, based on any of the above embodiments, the endoscope report generation device can further include:

[0092] A key frame determination module is configured to determine a comprehensive key parameter of a picture frame based on color abnormality degree, picture frame quality and lesion key frame of the picture frame in the endoscope real-time video stream, and determine a key frame in the endoscope real-time video stream based on the comprehensive key parameter;

[0093] An examination integrity parameter determination module is configured to perform examination integrity monitoring during an examination based on an endoscope to obtain an examination integrity parameter; wherein the examination integrity parameter is determined by examination time length, image quality and coverage area of each examination point, and weights corresponding to each parameter;

[0094] Correspondingly, the target endoscope report generation module 400 can include:

[0095] A target endoscope report generation unit is configured to generate an examination report based on the target examination type, the target examination site, the lesion area, the key frame and the examination integrity parameter to obtain the target endoscope report.

[0096] Further, based on any of the above embodiments, the lesion area determination module 300 can include:

[0097] A multi-scale fusion feature determination unit is configured to perform feature extraction on the endoscope real-time video stream, perform multi-scale feature fusion on all features obtained by extraction to obtain multi-scale fusion features;

[0098] a lesion prediction box determination unit configured to determine a plurality of lesion prediction boxes based on the multi-scale fusion features, each of the lesion prediction boxes including a bounding box coordinate, a confidence score, and classification information;

[0099] a lesion region determination unit configured to output the lesion region based on the lesion prediction boxes, the lesion region including a lesion location and a lesion classification result.

[0100] Further, based on any of the above embodiments, the target endoscopy report generation module 400 can include:

[0101] a large model-based target endoscopy report generation unit configured to obtain the target endoscopy report based on the target examination type, the target examination site, and the lesion region using a large language model and a corresponding prompt word template.

[0102] Further, based on any of the above embodiments, the endoscopy report generation method can further include:

[0103] a time sequence feature determination module configured to determine time sequence features based on joint learning of text and images in the target endoscopy report;

[0104] a diagnosis auxiliary disease condition prediction result determination module configured to obtain diagnosis auxiliary disease condition prediction results based on multi-modal prediction of the time sequence features.

[0105] Further, based on any of the above embodiments, the endoscopy report generation method can further include:

[0106] a suspicious detection event determination module configured to analyze the endoscopy real-time video stream using a detection model to determine suspicious detection events;

[0107] an index and highlight reel determination module configured to output an index and a highlight reel based on the suspicious detection events, wherein the index is a structured index table converted from the endoscopy real-time video stream, and the highlight reel includes abnormal time points and related descriptions.

[0108] It should be noted that the order of the modules and units in the endoscopy report generation device can be changed without affecting the logic.

[0109] The endoscope report generation device provided by the embodiment of the present application can comprise: a target detection type determination module 100, configured to acquire an endoscope real-time video stream, determine a target detection type based on time sequence features, spatial features and motion features of the endoscope real-time video stream; a target detection site determination module 200, configured to determine global features and local features of the endoscope real-time video stream, splice the global features and the local features, and determine a target detection site based on spliced features; a lesion area determination module 300, configured to perform real-time detection of lesions on the endoscope real-time video stream to obtain a lesion area; and a target endoscope report generation module 400, configured to generate a detection report based on the target detection type, the target detection site and the lesion area to obtain a target endoscope report. Compared with the current endoscope detection which is completely observed by a doctor manually and an endoscope report is generated, the present application determines a target detection type based on time sequence features, spatial features and motion features of an endoscope real-time video stream, determines global features and local features of the endoscope real-time video stream, splices the global features and the local features, determines a target detection site based on spliced features, performs real-time detection of lesions on the endoscope real-time video stream to obtain a lesion area, generates a detection report based on the target detection type, the target detection site and the lesion area to obtain a target endoscope report. Since the doctor does not need to manually determine various indexes to generate a report, the efficiency of endoscope report generation can be improved. Since the target detection site is determined based on the global features and the local features, multiple detection sites can be accurately determined, the comprehensiveness of endoscope detection site determination is improved, and thus the comprehensiveness of endoscope report generation is improved.

[0110] The endoscope report generation device provided by the embodiment of the present application will be described below. The endoscope report generation device described below can be referred to in correspondence with the endoscope report generation method described above.

[0111] Please refer to Figure 6 , Figure 6 The endoscope report generation device provided by the embodiment of the present application can comprise:

[0112] The memory 10 is configured to store a computer program.

[0113] The processor 20 is configured to execute the computer program to implement the endoscope report generation method described above.

[0114] The memory 10, the processor 20 and the communication interface 30 can communicate with each other through the communication bus 40.

[0115] In the embodiment of the present application, the memory 10 stores one or more programs, which can include program codes including computer operation instructions. In the embodiment of the present application, the memory 10 can store programs for implementing the following functions:

[0116] acquiring the endoscope real-time video stream, determining the target examination type based on the time sequence feature, the spatial feature and the motion feature of the endoscope real-time video stream;

[0117] determining the global feature and the local feature of the endoscope real-time video stream, splicing the global feature and the local feature, and determining the target examination part based on the spliced feature;

[0118] performing real-time lesion detection on the endoscope real-time video stream to obtain a lesion area;

[0119] generating an examination report based on the target examination type, the target examination part and the lesion area to obtain a target endoscope report.

[0120] In a possible implementation, the memory 10 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application program required by a function, etc. The data storage area can store data created during use.

[0121] In addition, the memory 10 can include a read-only memory and a random access memory, and provide instructions and data for the processor. A part of the memory can also include an NVRAM. The memory stores an operating system and operation instructions, executable modules or data structures, or a subset or an extended set thereof, wherein the operation instructions can include various operation instructions for implementing various operations. The operating system can include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0122] The processor 20 can be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device. The processor 20 can be a microprocessor or any conventional processor, etc. The processor 20 can invoke the program stored in the memory 10.

[0123] The communication interface 30 can be an interface of a communication module, used for connecting with other devices or systems.

[0124] Of course, it should be noted that, Figure 6 The structure shown does not constitute a limitation on the endoscope report generation device in the embodiment of the present application. In actual application, the endoscope report generation device can include more or fewer components than those shown. Figure 6more or less components shown, or combinations of certain components.

[0125] The computer readable storage medium provided by the embodiments of the present application is described below, and the computer readable storage medium described below can be referred to the endoscope report generation method described above.

[0126] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the endoscope report generation method described above.

[0127] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0128] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0129] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present text can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in the above description. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0130] Finally, it should be noted that in the present text, relationships such as first and second are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0131] The endoscope report generation method, device, equipment and readable storage medium provided by the present application are described in detail above, the principle and implementation mode of the present application are described by applying specific examples in this paper, and the above example description is only used to help understand the method of the present application and its core idea; at the same time, for the general technical personnel in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A method for generating an endoscopy report, characterized in that: include: Acquiring a real-time endoscopic video stream, and determining a target inspection type based on temporal features, spatial features, and motion features of the real-time endoscopic video stream; Determining global features and local features of the real-time endoscope video stream, splicing the global features and the local features, and determining a target inspection part based on the spliced ​​features obtained by splicing; Performing real-time lesion detection on the real-time video stream of the endoscope to obtain a lesion area; An inspection report is generated based on the target inspection type, the target inspection site, and the lesion area to obtain a target endoscopic report.

2. The method for generating an endoscope report according to claim 1, wherein: Before generating an inspection report based on the target inspection type, the target inspection site, and the lesion area to obtain a target endoscopic report, the method further includes: Determining comprehensive key parameters of picture frames based on color abnormality, picture frame quality, and lesion key frames in the real-time endoscopic video stream, and determining key frames in the real-time endoscopic video stream based on the comprehensive key parameters; Performing inspection integrity monitoring during the endoscopic inspection process to obtain inspection integrity parameters; wherein the inspection integrity parameters are determined by the inspection time, image quality and coverage area of ​​each inspection point, and the weights corresponding to the parameters; Accordingly, an inspection report is generated based on the target inspection type, the target inspection site, and the lesion area to obtain a target endoscopic report, including: An inspection report is generated based on the target inspection type, the target inspection site, the lesion area, the key frame and the inspection integrity parameter to obtain the target endoscopy report.

3. The method for generating an endoscope report according to claim 1, wherein: Performing real-time lesion detection on the real-time video stream of the endoscope to obtain the lesion area includes: Extracting features from the real-time video stream of the endoscope, and fusing all the extracted features at multiple scales to obtain multi-scale fused features; Identifying and obtaining a plurality of lesion prediction frames based on the multi-scale fusion features, each lesion prediction frame including bounding box coordinates, confidence level, and classification information; The lesion area is output based on the lesion prediction frame; the lesion area includes a lesion location and a lesion classification result.

4. The method for generating an endoscope report according to claim 1, wherein: An inspection report is generated based on the target inspection type, the target inspection site, and the lesion area to obtain a target endoscopic report, including: The target endoscopy report is obtained based on the target inspection type, the target inspection site, and the lesion area using a large language model and its corresponding prompt word template.

5. The method for generating an endoscopy report according to any one of claims 1 to 4, characterized in that: After generating an inspection report based on the target inspection type, the target inspection site, and the lesion area to obtain a target endoscopic report, the method further includes: performing joint learning based on text and images in the target endoscopy report to determine temporal features; Multimodal prediction is performed based on the time series features to obtain diagnosis-aided disease prediction results.

6. The method for generating an endoscope report according to claim 1, wherein: Also includes: Analyzing the real-time video stream of the endoscope using a detection model to determine suspicious detection events; An index and a highlight collection are output according to the suspicious detection event; wherein the index is obtained by converting the real-time endoscope video stream into a structured index table, and the highlight collection includes abnormal time points and related descriptions.

7. An endoscope report generating device, characterized in that: include: A target detection type determination module is used to obtain a real-time endoscope video stream and determine the target inspection type based on the temporal characteristics, spatial characteristics and motion characteristics of the real-time endoscope video stream; a target inspection part determination module, configured to determine global features and local features of the real-time endoscope video stream, splice the global features and the local features, and determine the target inspection part based on the spliced ​​features; a lesion area determination module, configured to perform real-time lesion detection on the real-time video stream of the endoscope to obtain a lesion area; The target endoscope report generating module is used to generate an inspection report based on the target inspection type, the target inspection part, and the lesion area to obtain a target endoscope report.

8. The endoscope report generating device according to claim 7, characterized in that: Also includes: a time series feature determination module, configured to perform joint learning based on text and images in the target endoscopy report to determine time series features; The diagnosis-aided disease prediction result determination module is used to perform multimodal prediction based on the time series characteristics to obtain the diagnosis-aided disease prediction result.

9. An endoscope report generating device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the endoscopy report generating method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the endoscope report generation method according to any one of claims 1 to 6.

Images