Capsule endoscope diagnosis report generation device, method, electronic device, and storage medium

Abstract

The application discloses a capsule endoscope diagnosis report generation device, method, electronic equipment and storage medium, wherein the diagnosis report generation device comprises: a part segmentation module, which inputs each frame of image collected by a capsule endoscope in a target area to a segmentation model, identifies candidate parts existing in the image, and stores each frame of image as a candidate image in a corresponding candidate part set; a part representative image screening module, which screens corresponding part representative images from candidate images contained in each candidate part set according to screening conditions corresponding to each candidate part; the screening conditions corresponding to at least two candidate parts are different, and the screening condition corresponding to at least one candidate part is associated with a geometric parameter, and the geometric parameter represents the spatial feature of the candidate part in the corresponding candidate image; and a report output module, which generates a diagnosis report corresponding to the capsule endoscope by using the screened part representative images.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a capsule endoscopy diagnostic report generation device, a capsule endoscopy diagnostic report generation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Capsule endoscopy can acquire a massive amount of imaging images of the digestive tract at once, making it a core tool for non-invasive screening of gastric diseases. In clinical practice, representative anatomical structure images need to be selected from the massive amount of images and compiled into diagnostic reports to visually present the morphological characteristics of key areas of the stomach such as the cardia, gastric angle, and gastric fundus.

[0003] Currently, the methods for selecting images for capsule endoscopy diagnostic reports are mainly divided into two categories: one is a purely manual screening mode, and the other is an automated solution that relies on site recognition models combined with general keyframe selection logic. However, both existing solutions have inherent technical defects.

[0004] For manual screening, physicians or dedicated image readers need to browse tens of thousands of endoscopic images frame by frame and manually select representative images that match the report. This not only results in a huge workload and low image reading efficiency, but also the image selection effect depends entirely on the operator's personal clinical experience and real-time working status. There are significant differences in the image selection scale between different people and even between the same person at different times, resulting in poor image selection stability.

[0005] For automated screening schemes equipped with site recognition models, existing schemes only use image clarity as the sole indicator to construct general screening rules. They apply the same set of judgment criteria to all gastric anatomical images, resulting in significant fluctuations in the quality of the site-representing images obtained through screening. It is difficult to unify the selection criteria for images in diagnostic reports for different cases, leading to a low degree of report standardization. Summary of the Invention

[0006] In order to solve at least one of the above-mentioned technical problems in the prior art, the present invention provides a capsule endoscopy diagnostic report generation device, a capsule endoscopy diagnostic report generation method, an electronic device, and a computer-readable storage medium.

[0007] The first aspect of this application provides a capsule endoscopy diagnostic report generation device, comprising: Part segmentation module: Input each frame of image acquired by the capsule endoscope in the target area into the segmentation model, identify the candidate parts present in the image, and store each frame of the image as a candidate image in the corresponding candidate part set according to the different candidate parts in the image; The region-representative image filtering module: Based on the filtering conditions corresponding to each candidate region, it filters the candidate images contained in each candidate region set to obtain the corresponding region-representative image; among the filtering conditions corresponding to each of the candidate regions, at least two candidate regions have different filtering conditions, and at least one candidate region's filtering condition is associated with geometric parameters. These geometric parameters characterize the spatial features of the candidate region within the corresponding candidate image, and are calculated from the segmentation mask output by the segmentation model after identifying the candidate region in the candidate image; and Report output module: Generates a diagnostic report corresponding to the capsule endoscope using the filtered images representing the affected areas.

[0008] A second aspect of this application provides a method for generating a capsule endoscopy diagnostic report, comprising: Each frame of image acquired by the capsule endoscope in the target area is input into the segmentation model to identify the candidate regions in the image. Based on the different candidate regions in the image, each frame of the image is stored as a candidate image in the corresponding candidate region set. Based on the filtering conditions corresponding to each candidate region, a representative image of the corresponding region is obtained from the candidate images contained in each candidate region set; among the filtering conditions corresponding to each of the candidate regions, at least two candidate regions have different filtering conditions, and at least one candidate region's filtering condition is associated with geometric parameters, the geometric parameters representing the spatial features of the candidate region within the corresponding candidate image, the geometric parameters being calculated from the segmentation mask output by the segmentation model after identifying the candidate region in the candidate image; and A diagnostic report corresponding to the capsule endoscope is generated using the selected site representative images.

[0009] A third aspect of this application provides an electronic device, including a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the capsule endoscopy diagnostic report generation method as described above.

[0010] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the endoscopy report generation method described above.

[0011] In the capsule endoscopy diagnostic report generation apparatus, method, electronic device, and computer-readable storage medium provided in this application, among the screening conditions for the representative images of each candidate site, at least one candidate site is associated with a geometric parameter. The geometric parameter characterizes the spatial features of the candidate site within the corresponding candidate image. The geometric parameter is calculated by the segmentation mask output by the segmentation model after identifying the candidate site in the candidate image. Thus, the representative images of the site can be screened based on the geometric parameters characterizing the imaging performance of the candidate site, effectively reducing the random fluctuation of the image selection results, unifying the selection scale of site images in different case diagnostic reports, and improving the stability and standardization of automatic image screening in diagnostic reports. Attached Figure Description

[0012] The accompanying drawings are provided to further illustrate the embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. It should be noted that, in order to clearly illustrate the technical solutions of this application, some drawings are not drawn strictly according to the scale, and key parts have been enlarged or shifted.

[0013] Figure 1 A schematic diagram of a capsule endoscopy diagnostic report generation device provided for an embodiment of this application; Figure 2 A schematic flowchart of the capsule endoscopy diagnostic report generation method provided for embodiments of this application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0015] The technical solution of this patent will be further described in detail below with reference to specific embodiments.

[0016] The embodiments of this patent are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this patent, and should not be construed as limiting this patent.

[0017] This application provides a capsule endoscopy diagnostic report generation device and a capsule endoscopy diagnostic report generation method, which are used to automatically select representative images of the body parts during or after the capsule endoscopy examination in the body of the subject, and generate a diagnostic report using the representative images of the body parts.

[0018] Please refer to Figure 1 The capsule endoscopy diagnostic report generation device 100 provided in this application includes: Part segmentation module 101: Input each frame of image acquired by the capsule endoscope in the target area into the segmentation model, identify the candidate parts in the image, and store each frame of image as a candidate image in the corresponding candidate part set according to the different candidate parts in the image; The part representative image filtering module 102: Based on the filtering conditions corresponding to each candidate part, it filters the corresponding part representative images from the candidate images contained in each candidate part set; among the filtering conditions corresponding to each candidate part, at least two candidate parts have different filtering conditions, and at least one candidate part's filtering condition is associated with geometric parameters. The geometric parameters characterize the spatial features of the candidate part within the corresponding candidate image, and are calculated by the segmentation mask output by the segmentation model after identifying the candidate parts in the candidate image; and Report output module 109: Generates a diagnostic report corresponding to the capsule endoscope using the filtered site representative images.

[0019] In the capsule endoscopy diagnostic report generation device 100, method, electronic device, and computer-readable storage medium provided in this application, among the screening conditions for the representative images of each candidate site, at least one candidate site is associated with a geometric parameter. The geometric parameter characterizes the spatial features of the candidate site within the corresponding candidate image. The geometric parameter is calculated by the segmentation mask output by the segmentation model after identifying the candidate site in the candidate image. Thus, the representative images of the site can be screened based on the geometric parameters characterizing the imaging performance of the candidate site, effectively reducing the random fluctuation of the image selection results, unifying the selection scale of site images in different case diagnostic reports, and improving the stability and standardization of automatic image screening in diagnostic reports.

[0020] Capsule endoscopy acquires a continuous sequence of images during the examination. This sequence can originate from the real-time video stream transmitted back by the capsule endoscope, or from image or video files stored after the examination. The continuous image sequence comprises multiple consecutive frames, each containing image data, a frame number, and the capsule endoscope's serial number, among other basic information.

[0021] The segmentation model in the part segmentation module 101 can be Mask-RCNN, or other instance segmentation or semantic segmentation models, to identify candidate parts present in the image.

[0022] In this embodiment, the target region is the stomach, and the candidate sites are six key anatomical regions of the stomach, including the cardia, fundus, stomach body, angle, antrum, and pylorus. It is understood that in other embodiments, the target region may be the esophagus or intestine, and correspondingly, the candidate sites are key anatomical regions of the esophagus or intestine.

[0023] The part segmentation module 101 is used to store each frame of the image as a candidate image in the corresponding candidate part set according to the different candidate parts identified in the image.

[0024] For example, the site segmentation module 101 inputs the first frame image acquired by the capsule endoscope in the stomach into the segmentation model. The segmentation model identifies that the first frame image contains the cardia. The site segmentation module 101 or the segmentation model stores the first frame image as a candidate image in the candidate site set of the cardia.

[0025] Multiple candidate images can be stored within the candidate region set for any candidate region. In some implementations, some candidate regions are not identified in the images during the entire examination process of capsule endoscopy. For example, if the segmentation model does not identify the pylorus in all images during the examination process, the candidate region set corresponding to the pylorus is empty and no candidate image is stored.

[0026] The report output module 109 generates a diagnostic report for the capsule endoscope using the site-representative image filtering module 102, which filters the site-representative images. The diagnostic report can include a site-representative image section, which displays the site-representative images for each candidate site filtered by the site-representative image filtering module 102. The diagnostic report output supports multiple formats, including PDF files, structured electronic data (such as JSON), system-specific data formats, and graphic templates adapted for electronic medical record integration.

[0027] The part representative image filtering module 102 filters the corresponding part representative images from the candidate images in each candidate part set according to the filtering conditions corresponding to each candidate part.

[0028] The filtering conditions corresponding to each of the candidate regions can be the same or different. In the embodiments of this application, among the filtering conditions corresponding to each of the candidate regions, at least two candidate regions have different filtering conditions for the images they represent.

[0029] At least one candidate region corresponds to a region representing the image's filtering criteria associated with geometric parameters. These geometric parameters characterize the spatial features of the candidate region within its corresponding candidate image, thus representing the imaging performance of the candidate region in the candidate image. The geometric parameters are calculated from the segmentation mask output by the segmentation model after identifying the candidate region in the candidate image. The geometric parameters include at least one of the following: the boundary contact state of the identified candidate region in its respective image, region centering, size, and shape.

[0030] The segmentation model outputs a binary image as a segmentation mask. All pixels belonging to a candidate region in the mask (including internal filling pixels and outer contour pixels) correspond to complete pixel coordinates, fully recording the global pixel distribution and closed contour boundaries of the candidate region in the candidate image. Simultaneously, this segmentation mask can generate bounding boxes adapted to the candidate regions. These bounding boxes are the smallest bounding rectangles that fit the complete imaging area of ​​the candidate region in the candidate image, accurately enclosing the identified candidate region. The extreme values ​​(corner coordinates) of the bounding boxes are the baseline position information inherent in the segmentation mask, forming an important component of the mask's spatial feature data. Based on the global pixel coordinate data of the candidate region and / or the extreme values ​​of the bounding boxes, the contact between the candidate region and the image edge can be determined, the contour morphology can be calculated to characterize the region's shape, and dimensional parameters such as the candidate region's center position, overall pixel area, and extension length can be solved. This allows for precise quantification and calculation of various geometric parameters, including boundary contact state, centering, size, and shape.

[0031] Geometric parameters include at least one of the following: boundary contact state, location centering, size, and shape.

[0032] Boundary contact state is used to characterize whether the contour of a candidate region touches the boundary of its parent candidate image. The boundary contact state is defined as follows: if a candidate region touches the boundary of its parent candidate image, the boundary contact state is 0; otherwise, the boundary contact state is 1. For example, if a capsule endoscope captures an image with a resolution of 480*480, and the candidate region segmentation mask shows boundary points with x=0, x=479, y=0, or y=479, then the candidate region is considered to be within its parent candidate image, and the boundary contact state is 0; otherwise, the boundary contact state is 1. Generally, selecting representative images requires a boundary contact state of 1 to facilitate the complete display of candidate regions within the candidate image.

[0033] Location centering refers to the degree to which a candidate location is centered within the candidate image. The formula for calculating location centering is: , , S center (r i )=1-d i / Dmax , in: d i Let be the distance between the center of the i-th candidate region and the center of its corresponding candidate image. (x c ,y c () represents the center coordinates of the candidate image to which the i-th candidate region belongs. (x i ,y i Let be the center coordinates of the i-th candidate region within its respective candidate image. D max The length of the candidate image's diagonal. S center (r i ) represents the i-th candidate region r i The closer the value is to 1, the stronger the candidate part r is. i The closer the center is to the center of its corresponding candidate image.

[0034] The shape is characterized by the roundness of the candidate region's contour. The formula for calculating roundness is: S circle (r i )=4πA i / P i 2 , in: S circle (r i ) represents the i-th candidate region r i Circularity in its candidate image A i For the i-th candidate part r i The pixel area in its candidate image. P i For candidate regions r in candidate images i The pixel perimeter.

[0035] Candidate part r i The pixel area and pixel perimeter refer to the candidate region r within the segmentation mask. i The quantization metrics are calculated from the coordinates of all pixels. The pixel area represents the total number of pixels constituting the candidate region, and the pixel perimeter is calculated from the total number of pixels along the closed contour line of the candidate region. These two metrics directly reflect the actual imaging size and contour extension of the candidate region within the image, serving as a geometric criterion for selecting representative images.

[0036] The dimensions include the pixel length, pixel area, and / or area fraction of the target subject corresponding to the candidate region within its respective candidate image. The target subject includes the candidate region, the segmentation unit within the candidate region, or the candidate region related to the candidate region. The candidate region related to the candidate region can be a candidate region adjacent to the candidate region or a non-adjacent candidate region. For example, in some embodiments, the candidate region is the cardia, and its related candidate region can be the fundus of the stomach adjacent to the cardia.

[0037] Area fractions include larger area fractions and smaller area fractions, and the calculation formulas are as follows: S area (r i )=AR i / A max , S small (r i )=1-AR i / A max , In the formulas for calculating larger and smaller area fractions: AR i For the i-th candidate part r i The corresponding pixel area of ​​the target subject in its respective candidate image. A max For the i-th candidate region r among all candidate images in the same candidate region set, i The maximum pixel area of ​​the corresponding target subject, or the maximum preset pixel area of ​​the target subject. S area (r i Let r be the i-th candidate part. i The larger the area of ​​the target subject within its respective candidate image, the higher the S value. area (r i The closer the value of ) is to 1, the better. S small (r i ) represents the i-th candidate region r within the candidate image. i The smaller the area of ​​the target subject, the smaller the area of ​​the target subject within the candidate image, the lower the S value. small (r i The closer the value of ) is to 1.

[0038] The formula for calculating the extension length is: , in: L elong (r i ) represents the i-th candidate region r iThe corresponding extension length of the target subject is used to characterize the elongation of the target subject identified within the candidate image. x i max x i min y i max y i min The extreme values ​​of the bounding box corresponding to the target body, output by the segmentation model.

[0039] In this embodiment, instead of using a single, uniform image selection standard for different candidate sites, a differentiated screening function and a hierarchical backoff mechanism are defined based on the medical morphological characteristics of each candidate site.

[0040] The weights in the screening formulas for different sites can be adjusted based on the training set, validation set, or clinical experience.

[0041] In this application, the target region is the stomach, and the candidate sites included in the stomach are the cardia, fundus, body, angle, antrum, and pylorus.

[0042] In some implementations, it is necessary to filter candidate images that identify the cardia to obtain a representative image of the cardia. In this case, the candidate region is the cardia, and the filtering criteria for the cardia are: Calculate the comprehensive score of each candidate image in the candidate image set corresponding to the cardia as the representative image of the area. Select several candidate images as representative images of the area according to the comprehensive score from high to low. The formula for calculating the comprehensive score of the area is as follows: Score cardia (r i )=b1*S center (r i )+ b2*S circle (r i )+ b3*S area (r i ), Among them, Score cardia (r i The candidate image containing the cardia is used as the representative image of the region for a comprehensive score. S area (r i ) represents the i-th candidate region r i (i.e., the cardia) has the largest area fraction within its candidate image. b1, b2, and b3 are weighting coefficients.

[0043] In some implementations, 0.4 ≤ b1 ≤ 0.8; and b1 ≥ b2, b1 ≥ b3.

[0044] The image of the cardia emphasizes that the cardia should be located in the center of the image as much as possible, with a relatively rounded outline and a moderate area. Therefore, the three indicators of centering, roundness and area are used for comprehensive scoring.

[0045] One to five candidate images with the highest overall scores for the affected area can be selected as representative images of the cardia. Preferably, two candidate images with the highest overall scores for the affected area are selected as representative images of the cardia and presented in the diagnostic report.

[0046] In some implementations, the candidate site is the gastric angle, and the screening conditions corresponding to the gastric angle include a first screening condition and a second screening condition. When the location-representative image filtering module 102 filters location-representative images from the candidate image set corresponding to the gastric angle, there are two execution modes: Mode 1: Select either the first or second filter condition to execute; Mode 2: First, the first screening condition is executed first. If there is no candidate image in the candidate image set corresponding to the gastric angle that meets the first screening condition, then the second screening condition is executed. The first screening criterion is: Obtain the boundary contact state of the gastric angle in each candidate image within the candidate image set corresponding to the gastric angle. Calculate the comprehensive score of each candidate image as a representative image of the region. Sort all candidate images from high to low according to the comprehensive score of the region. Select a number of candidate images with a comprehensive score of non-zero as representative images of the region. The formula for calculating the comprehensive score of the region is as follows: Score angle (r i )=B(ri)*[ d1*L elong (r i )+ d2*S area (r i )], Among them, Score angle (r i The candidate image containing the gastric angle is used as the site representative image for a comprehensive site score. L elong (r i ) represents the i-th candidate region r i (i.e., the gastric angle) is the length of its extension within its candidate image. S area (r i Let r be the i-th candidate part. i (i.e., the gastric angle) has the largest area fraction within its respective candidate image. d1 and d2 are weighting coefficients; in some implementations, d1 ≥ d2.

[0047] The second screening criterion is: Calculate the extension length of the gastric angle in each candidate image in the candidate image set corresponding to the gastric angle, sort the candidate images from largest to smallest according to the extension length value, and select several candidate images as representative images of the site.

[0048] The gastric angle is first selected based on candidate images that are complete and not touching the edges. Then, images with a more slender shape and a more suitable area are chosen. If no images that are not touching the edges are found, the constraints are relaxed, and the gastric angle image with the most morphological features is selected.

[0049] One to five candidate images can be selected as representative images of the gastric angle. Preferably, two representative images of the gastric angle are selected and presented in the diagnostic report.

[0050] In some implementations, the candidate site is the gastric antrum, and the corresponding screening criteria for the gastric antrum are: Calculate the overall site score for each candidate image in the candidate image set corresponding to the gastric antrum, and select several candidate images with a non-zero overall site score as the site representative images in descending order of overall site score. The formula for calculating the overall site score is as follows: Score antrum (r i )=B(r i )*[ g1*S small (r i )+ g2*S center (r i )+ g3*S circle (r i )] , Among them, Score antrum (r i The candidate image containing the gastric antrum is used as the site representative image for a comprehensive site score. S small (r i ) represents the i-th candidate region r i (i.e., the small fraction of the area of ​​the gastric antrum within its respective candidate image) g1, g2, and g3 are weighting coefficients. In some implementations, g1 ≥ g2 ≥ g3, and 0.3 ≤ g1 ≤ 0.8.

[0051] The screening of the gastric antrum emphasizes that the target should not be close to the edge and should be located in the central area as much as possible. At the same time, the candidate results with smaller area and more regular shape are more consistent with the typical gastric antrum view.

[0052] In some implementations, the candidate site is the pylorus, and the screening criteria for the pylorus are: Images with a boundary contact state of 1 within the candidate image set corresponding to the pylorus are selected. The images are sorted from high to low according to the confidence of the segmentation model in identifying the pylorus within the candidate images. Several candidate images are selected as representative images of the part.

[0053] Pyloric sphincter often presents with short detection times and high identification difficulty in capsule endoscopy. Therefore, priority is given to ensuring that the target is not too close to the edge, and the most reliable result is selected directly based on the model confidence level.

[0054] One to five candidate images can be selected as representative images of the affected area. Preferably, two representative images of the pylorus are selected and presented in the diagnostic report.

[0055] In some implementations, among the screening criteria corresponding to each of the candidate regions, at least one candidate region's screening criteria is associated with the coexistence relationship between the candidate region and other candidate regions. In the implementation of this application, the screening criteria for the region-representing images of the gastric fundus and gastric body are associated with the coexistence relationship between the candidate region and other candidate regions.

[0056] In some implementations, the candidate site is the gastric body, the target region is the stomach, and other candidate sites included in the stomach include the cardia, fundus, antrum, and angle. The gastric body includes two segmented units: the lesser curvature and the greater curvature. The screening criteria for the gastric body include a third, a fourth, and a fifth screening criterion. When the site-representative image filtering module filters site-representative images from the candidate image set corresponding to the stomach body, there are three execution modes: Mode 3: If the candidate image contains the lesser curvature of the stomach and at least one of the cardia, fundus, and gastric cavity, then the third screening condition is executed; Mode 4: If the candidate image contains the greater curvature and gastric cavity, then the fourth screening condition is applied; Mode 5: If a gastric cavity or antrum exists in the candidate image, and a gastric angle with a boundary contact state of 1 exists, then the fifth screening condition is executed; The gastric cavity is the dark area presented in the candidate image of a cavity enclosed by at least one of the gastric body mucosal folds, lesser curvature, and greater curvature. The third screening criterion is: Calculate the overall site score for each candidate image in the candidate image set corresponding to the stomach body, and select several candidate images as site representative images according to the overall site score from high to low. The formula for calculating the overall site score is as follows: N region (I t )=∑ k 1(Region k existsinI t ), Score body (It )=h1N region (I t )+ h2L lesser (I t ), N region (I t ) represents the candidate image I in frame t. t The number of visible structures includes the lesser curvature, cardia, fundus, and gastric cavity. Region k For the kth visible structure, k = 1, 2, 3 or 4, Region k existsinI t This represents the k-th visible structure in the candidate image I of frame t. t It was identified in the middle. 1(·) is an indicator function, which is 1 when the condition is met and 0 otherwise.

[0057] For example: in the t-th frame, candidate image I t In the middle, three visible structures are the lesser curvature of the stomach, the cardia, and the fundus. Then N... region (I t )=3. N region (I t The number of values ​​is used to measure the coverage of gastric body-related anatomical information by the current frame candidate image. The larger the value, the richer the gastric body auxiliary discrimination information that the candidate image can present at the same time.

[0058] Score body (I t The candidate image containing the stomach body is used as a representative image of the region for a comprehensive score. h1 and h2 are weighting coefficients, and h1 > h2 > 0; L lesser (I t ) represents the extension length of the lesser curvature of the stomach in the candidate image of frame t.

[0059] The fourth screening criterion is: Several candidate images were selected as representative images of the gastric cavity in ascending order of pixel area.

[0060] The fifth screening criterion is: Several candidate images were selected as representative images of the gastric region in ascending order of pixel area.

[0061] In some implementations, at least one representative image of the gastric body showing the lesser curvature can be obtained by screening according to the third screening condition, at least one representative image of the gastric body showing the greater curvature can be obtained by screening according to the fourth screening condition, and at least one representative image of the gastric body showing the angle of the stomach can be obtained by screening according to the fifth screening condition. The representative images of the gastric body from the above three perspectives are all presented in the diagnostic report, thereby realizing a complete display of multi-view gastric body images.

[0062] It should be noted that after training, the segmentation model can identify tissue structures within an image, enabling the identification of six candidate sites in the stomach. After identifying the candidate sites, it can further identify the internal subdivisions (segmentation units) of the candidate sites based on image information, such as identifying the greater curvature and lesser curvature of the stomach body. The segmentation model is also used to identify the gastric cavity in the image. The gastric cavity is the dark area presented in the candidate image, which is the cavity formed by at least one of the gastric mucosal folds, lesser curvature, and greater curvature.

[0063] In this embodiment, the gastric cavity serves as a preferred auxiliary structure in the gastric body screening criteria, used to characterize the open area within the cavity, the central hollow area, or auxiliary segmentation areas related to the morphology of the gastric body. The device can perform gastric body keyframe screening based on the coexistence relationship between the gastric cavity and the lesser curvature, greater curvature, gastric angle, gastric fundus, or gastric antrum. When the gastric cavity segmentation result output by the segmentation model is unavailable or does not participate in the above screening criteria calculation, representative images of the gastric body from various perspectives can also complete the gastric body keyframe screening based on the coexistence relationship between the gastric body and other structures, pixel area, extension length, location centering, and boundary contact state in the above screening criteria.

[0064] In some implementations, the candidate site is the fundus of the stomach; the target region is the stomach, and other candidate sites included in the stomach include the cardia. The site-representing image filtering module 102 filters candidate images that meet the first preferred condition from the candidate image set corresponding to the fundus of the stomach and copies them to the first preferred set, and filters candidate images that meet the second preferred condition and copies them to the second preferred set. First preferred condition: The candidate image contains the cardia and the boundary contact state of the gastric fundus is 1; Second preferred condition: The boundary contact state of the gastric fundus in the candidate image is 1; When selecting images representing the corresponding areas of the gastric fundus, use the seventh or eighth selection criteria. The seventh screening criterion is to sort the candidate images from largest to smallest according to the pixel area of ​​the gastric fundus, and select a number of candidate images from the first preferred set, the second preferred set, or the candidate image set corresponding to the gastric fundus as representative images of the gastric fundus. Eighth screening criterion: The candidate images are sorted from largest to smallest according to the pixel area of ​​the gastric fundus, and a number of candidate images are selected from the first preferred set as representative images of the site. If there are no candidate images in the first preferred set, then select several candidate images from the second preferred set as representative images of the body parts; If there are no candidate images in the second preferred set, then select several candidate images from the candidate image set corresponding to the gastric fundus as representative images of the site.

[0065] The representative image of the gastric fundus should ideally include the gastric fundus and the cardia together, and the gastric fundus should not be attached to the edge, in order to improve the reliability of gastric fundus localization. If this condition is not met, the constraints are gradually relaxed, and the candidate gastric fundus image with the larger gastric fundus area is ultimately used as the representative image (keyframe).

[0066] The site representative image screening module uses a differentiated screening mechanism for six candidate gastric sites to ensure that the selected site representative images are more in line with clinical reading habits and improve the standardization of reports.

[0067] In some implementations, the diagnostic report includes a lesion representative image section in addition to the site representative image section. The lesion representative image section is used to display the lesion representative images of each lesion instance obtained through screening.

[0068] Accordingly, the capsule endoscopy diagnostic report generation device 100 includes: Lesion Image Report Supplement Module: Used to receive a set of lesion instances and use them as optional input to the lesion image section of the diagnostic report; Lesion representative image selection module: When a set of lesion instances exists, calculates a comprehensive lesion score for each candidate image in each lesion instance set as a representative lesion image, and selects several representative lesion images from the candidate images corresponding to the same lesion instance based on the comprehensive lesion score; and Report output module: Generates a diagnostic report corresponding to the capsule endoscopy using the representative images of the lesions obtained from the screening.

[0069] The device can be equipped with a lesion instance identification and association module to obtain a set of lesion instances.

[0070] Specifically, the lesion instance identification and association module uses the target detection model to detect lesion instances in the candidate images in each candidate site set, and copies the candidate images that detect the same lesion instance to the corresponding lesion instance set; The lesion image report supplement module is used to receive the set of lesion instances output by the lesion instance identification and association module, and use it as an optional input to the lesion image section of the diagnostic report; The report output module is used to generate a diagnostic report by binding a set of lesion instances with their corresponding candidate sites.

[0071] The lesion instance identification and association module may include a lesion detection submodule, a feature extraction submodule, and an instance association submodule. The lesion detection submodule can be implemented using an object detection model, and the instance association submodule can be implemented using feature similarity-based matching methods, DeepSORT, MOTR, Siamese networks, or other multi-object association methods to distinguish between the same lesion instance and newly added lesion instances.

[0072] The lesion instance identification and association module performs instance-level identification of lesions in candidate images, distinguishing whether the currently detected lesion belongs to a historical lesion instance or a newly added lesion instance; then, the lesion representative image selection module selects representative images for each lesion instance, which can avoid the same lesion from being repeatedly entered into the diagnostic report and reduce the information redundancy between the images and diagnostic conclusions in the report.

[0073] After completing the identification and association of lesion instances, for each lesion instance, a representative image (representative frame) for displaying the diagnostic report is selected from its corresponding lesion instance set. The lesion instance set is derived from the lesion instance identification and association results and contains multiple observation images of the same lesion instance obtained at different times.

[0074] The lesion representative image selection module calculates the comprehensive lesion score for each candidate image as a representative lesion image based on the clarity of the candidate image, the pixel area or pixel area ratio of the lesion region in the candidate image, the centering of the lesion region in the candidate image, and the representativeness score of the lesion region in the candidate image.

[0075] Specifically, the formula for calculating the comprehensive lesion score is as follows: Score(f i )=k1Q i +k2AE i + k3C i + k4R i , , , C i =1-d i / D max , in: Score(f i ) is the candidate image f i Comprehensive lesion score as a representative image of the lesion; Q i For candidate image f i The image quality score is related to the sharpness, exposure quality, and degree of occlusion of the candidate image.

[0076] AE iFor candidate image f i The pixel area of ​​the lesion region, or the lesion region in the candidate image f. i The percentage of pixel area within the image. (AE) i To evaluate the adequacy of lesion display, it can be calculated using the mask information output by the target detection model that corresponds to the lesion regions detected in each image.

[0077] C i For candidate image f i The degree of lesion centering in the lesion area is indicated by a value closer to 1, meaning the center of the lesion area is closer to the candidate image f. i The center of the lesion. Centrality C i For the calculation method, please refer to the centering of the part. Simply replace the center coordinates of the candidate part in the candidate image with the center coordinates of the lesion area in the candidate image.

[0078] R i For candidate image f i The representativeness score of the lesion region is correlated with at least one of the following: the confidence level of the lesion region, the boundary contact state of the lesion region, and the candidate site to which the lesion region belongs. The higher the score, the more suitable the candidate image is as a representative image of the lesion.

[0079] The confidence score of the lesion region is additional information output by the target detection model when it detects a lesion region.

[0080] The boundary contact state of the lesion area can be calculated based on the mask information corresponding to the lesion areas detected in each image, output by the target detection model.

[0081] Since the lesion instance identification and association module copies the candidate images of the same lesion instance detected in each candidate site set to the corresponding lesion instance set, each candidate image in the lesion instance set is matched with its corresponding candidate site. For any candidate image in the lesion instance set, its corresponding candidate site is known information, and the corresponding candidate site can be displayed in the diagnostic report as the classification identifier information of the candidate image.

[0082] k1, k2, k3, and k4 are the weights corresponding to each parameter, all of which are greater than or equal to 0 and satisfy k1+k2+k3+k4=1. They can be adjusted according to the training set, validation set, or clinical experience. d i Let be the distance between the center of the i-th lesion region and the center of its corresponding candidate image. (x c ,y c () represents the center coordinates of the candidate image to which the i-th lesion region belongs. (xi ,y i Let be the center coordinates of the i-th lesion region within its respective candidate image. D max is the length of the diagonal of the candidate image.

[0083] The coordinates (x) in this formula c ,y c ) and (x i ,y i It can be calculated based on the mask information corresponding to the lesion areas detected in each image, output by the target detection model.

[0084] The meaning of the comprehensive lesion score calculation formula is that the representative image of the lesion should meet the requirements of clear image, sufficient display of lesion, easy observation of lesion location, and typical medical expression.

[0085] The specific formula for calculating image quality score is as follows: Q i =v1B i + v2E i + v3M i , v1 + v2 + v3 = 1, v1, v2, v3 ≥ 0, The meanings of each letter / symbol are as follows: Q i Candidate image f i Image quality score; B i Candidate image f i Clarity metrics; E i Candidate image f i Exposure compliance index; M i Candidate image f i The mucus / obstruction compliance index; v1, v2, v3: These are the weighting coefficients for the sharpness index, exposure passability index, and mucus / occlusion passability index, respectively, and can be adjusted based on the training set, validation set, or clinical experience.

[0086] Clarity Index B i Used to characterize candidate image f i The clarity of the lesion region's boundaries, texture, and local structure. Intuitively, this can be understood as: the clarity of the candidate image f i The sharper the edges and the richer the texture of the lesion area, the higher the clarity index B. i The higher.

[0087] One specific calculation formula is as follows: B i =clip{[Var(Lap(f i ))] / T B ,0,1}, The meanings of each letter / symbol are as follows: B i Represents candidate image f i Clarity metrics; f i Represents the i-th candidate image; Lap(·) represents the Laplacian operator, used to extract the second-order transformation of image grayscale. The more obvious the edges and details, the stronger the Laplacian response usually is. Var(·) represents variance, which measures the dispersion of the Laplace response. A larger variance usually indicates a clearer image. T B This indicates a preset sharpness threshold, used to normalize the sharpness response. It can be adjusted based on training data, validation data, or practical application experience. clip(x,0,1) means restricting x to the range of 0 to 1.

[0088] Exposure compliance index E i For candidate image f i The percentage of pixels with medium brightness falling within the effective observation range is used to characterize whether the candidate image is too dark or overexposed.

[0089] One specific calculation formula is as follows: E i =clip{1-[(N low,i +N high,i ) / N i ],0,1}, The meanings of each letter / symbol are as follows: E i Represents candidate image f i Exposure compliance index; N i Represents candidate image f i Total number of pixels within; Y min and Y max These represent the lower and upper limits of the effective brightness range, respectively. N low,i This indicates that the brightness in the candidate image is lower than Y. min The number of pixels, i.e., the number of overly dark pixels; N high,i This indicates that the brightness of the candidate image is higher than Y. max The number of pixels, i.e., the number of overexposed pixels; (Nlow,i +N high,i ) / N i Indicates the percentage of pixels with poor exposure, 1-[(N low,i +N high,i ) / N i [] indicates the percentage of pixels that passed the exposure test.

[0090] The clip function is used to limit the exposure pass rate index between 0 and 1, thereby avoiding the occurrence of N pixels with unacceptable exposure due to calculation errors. low,i +N high,i Greater than the total number of pixels N in the candidate image i This situation has occurred.

[0091] Mucus / Coverage Compliance Index M i The percentage of the visible lesion area in the entire candidate image is used to characterize the degree to which the lesion area is covered by mucus, air bubbles, food residue, reflective spots, or other obstructions. Intuitively, it can be understood as: Mucus / Occlusion Acceptance Index M. i The higher the value, the higher the mucus / coverage qualification. One specific calculation formula is: M i = clip{1-[A blocked,i / ( A lesion,i +ε)],0,1}, The meanings of each letter / symbol are as follows: M i Represents candidate image f i The mucus / obstruction compliance index; A blocked,i This represents the pixel area of ​​the occluded region within the lesion detection box identified by color, texture rules, or mucus / occlusion segmentation models (such as traditional OpenCV image processing methods and CNN deep learning methods). A lesion,i Represents candidate image f i The pixel area of ​​the lesion region; ε represents a minimal constant to prevent the denominator from being zero; A blocked,i / ( A lesion,i +ε) represents the percentage of the obstructed area; 1-[A blocked,i / ( A lesion,i +ε)] represents the percentage of the visible area within the lesion detection frame.

[0092] The clip function is used to limit the visible area of ​​a lesion to between 0 and 1.

[0093] The above sections provide quantitative acquisition algorithms for the sharpness indicator, the exposure qualification indicator, and the mucus / occlusion qualification indicator respectively. It is understandable that those skilled in the art can make adaptive adjustments to the calculation methods of the three types of indicators described above in combination with actual image acquisition conditions and algorithm debugging experience.

[0094] Thus, the image quality score Q of the candidate image f i i can simultaneously reflect whether the candidate image f i is clear, whether the brightness is qualified, and whether the lesion area is occluded, which is beneficial to exclude candidate images with blurred frames, abnormal exposure, and severe occlusion, and exclude them from the diagnostic report.

[0095] In some embodiments, when multiple representative lesion images need to be selected for the same lesion instance, a similarity penalty term between candidate images is introduced to reduce the probability that a plurality of highly similar images are selected at the same time; based on the existing representative lesion images, a combination of representative lesion images with lower information redundancy is screened out for report display.

[0096] The representative lesion image selection module stores the representative lesion images into the alternative lesion image set, and under the condition that the constraint on the number of candidate images in the alternative image subset |Rl|≤Ml is satisfied, the alternative image subset is screened from the alternative lesion image set according to Formula 1.

[0097] (Formula 1), wherein, λ is a diversity constraint coefficient, which is used to control the strength of reducing redundant representative lesion images.

[0098] F l is the alternative lesion image set of the l-th lesion instance; R l is the alternative image subset selected from the alternative lesion image set F l , and each alternative image subset stores a plurality of representative lesion images; the alternative image subset R l is a subset of the alternative lesion image set F l . M l is the maximum number of representative lesion images allowed to enter the diagnostic report for each lesion instance.

[0099] is the sum of the comprehensive lesion scores of all representative lesion images in one alternative image subset R l ; is the sum of the similarities between any two different representative lesion images in one alternative image subset R l , and the calculation is performed only once in the manner of i<j; a larger value indicates that the content of the two images is more similar.​

[0100] argmax(·) selects the subset of candidate images R with the largest numerical value. l ; The report output module uses the candidate image subset R obtained by filtering using Formula 1. l Generate a diagnostic report corresponding to the capsule endoscopy.

[0101] Formula 1 means that while ensuring that the representative images of lesions have a high comprehensive lesion score, combinations of highly similar representative images of lesions are penalized, thereby avoiding the appearance of multiple highly repetitive representative images of lesions in the diagnostic report.

[0102] In some implementations, the candidate lesion image set F l The system stores three representative images of the lesion: lesion representative image f1, lesion representative image f5, and lesion representative image f7.

[0103] Candidate lesion image set F l Includes 6 alternative image subsets R l Let them be denoted as R1={f1}, R2={f5}, R3={f7}, R4={f1,f5}, R5={f1,f7}, and R6={f1,f5,f7}, respectively. l =5, calculate the comprehensive score of multiple candidate image subsets according to Formula 1, and then select a candidate image subset.

[0104] According to Formula 1, The overall score of the candidate image subset R1 = Score(f1), The overall score for the candidate image subset R2 is Score(f5). The overall score for the candidate image subset R3 = Score(f7), The overall score of the candidate image subset R4 = Score(f1) + Score(f5) - λ*sim(f1,f5), The overall score of the candidate image subset R5 = Score(f1) + Score(f7) - λ*sim(f1,f7), The overall score of the candidate image subset R6 = Score(f1) + Score(f5) + Score(f7) - λ*[sim(f1,f5) + sim(f1,f7) + sim(f5,f7)].

[0105] Formula 1 ultimately returns the subset of candidate images that yields the highest possible score. Based on the combined scores of the six candidate image subsets, the subset with the highest score is selected and presented in the diagnostic report.

[0106] The target detection model is used to detect lesion instances in candidate images from each candidate region set. In some implementations, the target detection model is specifically used for: Lesion detection is performed on candidate images in each candidate region set to detect at least one lesion region; in the candidate images, the current candidate image to be processed and its corresponding at least one current candidate lesion region are determined; For each current candidate lesion region, feature extraction is performed to generate the corresponding current candidate feature vector; Calculate the similarity between the current candidate feature vector and each historical feature vector in the lesion instance database; If the similarity is greater than or equal to the preset threshold, the current candidate lesion region is determined to belong to the lesion instance corresponding to the historical feature vector, and the historical feature vector corresponding to the lesion instance is updated iteratively using the current candidate feature vector. If the similarity between the current candidate feature vector and all historical feature vectors in the lesion instance database is less than a preset threshold, then the current candidate lesion region is determined to be a newly added lesion instance, and the current candidate feature vector is stored in the lesion instance database as a historical feature vector of the newly added lesion instance.

[0107] The target detection model provided in this application performs lesion detection, feature extraction and similarity calculation sequentially on continuous image sequences acquired by endoscopy; based on the similarity results, it determines the attribution relationship between the current candidate lesion region and the corresponding lesion instances of each historical feature vector, realizes dynamic iterative update of historical feature vectors, and stores the current candidate feature vector corresponding to the newly added lesion instance into the lesion instance library.

[0108] The target detection model provided in this application can effectively distinguish between recurring lesions and newly added lesions. This helps to reduce the misjudgment of the same lesion observed under different imaging conditions and observation dimensions as a new lesion, and alleviates the phenomenon of repeated labeling and redundant statistics of the same lesion in intelligent diagnostic reports. This can significantly reduce the burden of doctors in interpreting images and also improve the overall reliability of endoscopic clinical screening and auxiliary diagnosis.

[0109] The target detection model provided in this application facilitates unified instantiation management of individual lesions. It fuses the historical feature vectors corresponding to each lesion instance with newly added current candidate feature vectors, enabling dynamic updating and fusion of lesion features across multiple frames for the same lesion instance. This allows the historical feature vectors corresponding to lesion instances to gradually stabilize and become more complete as imaging conditions and observation dimensions become richer, thus constructing a robust feature representation of lesion instances. With this improved feature representation, the differentiation accuracy between different lesion instances can be significantly improved, reducing instances mismatch and misclassification problems, and comprehensively enhancing the overall judgment accuracy and practical application reliability of lesion instance identification and matching.

[0110] During the diagnostic report generation process, multiple candidate images of the same lesion at different times and angles cannot be simply written into the diagnostic report as multiple independent lesion entries. Instead, lesion instance association should be performed first. Let f be the feature vector of the current lesion. i The historical feature vector of existing lesion instances in the instance library is denoted as f. j Then, feature similarity is defined as: , in: S ij The similarity between the lesion region identified in the current candidate image and existing lesion instances in the lesion instance database; f i This is the feature vector of the lesion region to be matched. f j Let be the feature vector of the j-th lesion instance in the lesion instance database; f i ·f j It is the dot product of two eigenvectors; ||f i || represents the eigenvector f i The norm; ||f j || represents the historical feature vector f j The norm of .

[0111] This formula essentially uses cosine similarity to measure the consistency of feature representations between two lesions. Since the same lesion can appear differently in different frames due to factors such as angle, distance, lighting, and local occlusion, this invention first encodes the features of the current lesion and then performs instance matching based on the consistency of feature directions, thereby improving the robustness of cross-frame recognition.

[0112] In this embodiment, cosine similarity is used to calculate the similarity between the current lesion and historical lesion instances. In other embodiments, Euclidean distance or learned similarity measurement methods can also be used.

[0113] The feature vectors of the current lesion and the historical feature vectors can be represented by a single feature vector, a set of multiple feature vectors, or a probability distribution model.

[0114] After calculating the similarity between the current lesion feature vector and all historical feature vectors in the lesion instance database, the similarity of the lesion instance most similar to the current lesion feature vector is selected for threshold determination. If the similarity is greater than or equal to the preset threshold, the current lesion is determined to belong to the lesion instance corresponding to that historical feature vector, and the candidate lesion can be assigned the identifier of the lesion instance corresponding to that historical feature vector; if the similarity between the current lesion feature vector and all historical feature vectors in the lesion instance database is less than the preset threshold, the current lesion is determined to be a newly added lesion instance, and a new lesion instance identifier can be created for the current lesion.

[0115] , , The meanings of each letter / symbol are as follows: ID(i): The instance identifier that the current lesion is ultimately assigned.

[0116] The instance identifier (ID) is a unique identifier for each lesion instance, used to distinguish different lesion instances. The instance identifier (ID) can consist of at least one of letters, numbers, or symbols, such as 0001 or poly0001. When the variable j takes the values ​​0, 1, and 2 in sequence, the corresponding lesion instance identifiers can be set to poly0000, poly0001, and poly0002 in sequence.

[0117] j * : The lesion instance most similar to the current lesion, where j is a natural number.

[0118] argmax(·): The argument when (·) reaches its maximum value.

[0119] ID(j) * ): The instance identifier of the lesion instance in the lesion instance database that is most similar to the current lesion; T s The preset threshold can be determined by methods such as training set statistics, ROC analysis, or human experience. new_ID: The instance identifier of the newly created lesion instance.

[0120] When the maximum similarity is not less than the preset threshold T s When the maximum similarity is less than a preset threshold T, it indicates that the current lesion has a sufficiently high degree of similarity to an already identified lesion instance, and therefore the current lesion is classified as belonging to that already identified lesion instance. s When this occurs, it indicates that the current lesion is more likely a newly observed lesion instance, thus creating an instance identifier for it to create a new lesion instance.

[0121] Preset threshold T sIt can be set based on at least one method, such as training set statistical results, validation set ROC curve, or human experience, thus balancing recognition accuracy and robustness in practical applications.

[0122] The lesion instance database is not a static table, but a dynamically updated data structure that records not only lesion information but also the observation process of that lesion. In this embodiment, after instance matching is completed, the lesion instance database is updated using relevant data from the current candidate lesion region.

[0123] The lesion instance library is used to store the set of lesion instances corresponding to the identified lesions. In the embodiments of this application, the lesion instance library includes multiple lesion sets corresponding to the identified lesion instances. If two lesion instances have been identified, the lesion instance library will contain two lesion instance sets, and there is a one-to-one correspondence between the lesion instances and the lesion instance sets.

[0124] Through the above-mentioned lesion instance identification and association mechanism, the observation results of the same lesion in multiple frames can be merged into a single lesion instance, providing a basis for subsequent selection of representative lesion images and deduplication of diagnostic reports.

[0125] Lesion instance identification and association can be achieved by dynamically updating single-vector or multi-vector sets.

[0126] Specifically, the target detection model uses the following formula to iteratively update the historical feature vector corresponding to the lesion instance: f l (t+1) =(1-p)f l (t) +pf new , in: f l (t+1) : The historical feature vector of the l-th lesion instance in the lesion instance database after the next time step; f l (t) : The historical feature vector of the l-th lesion instance in the lesion instance database at the current time; f new : The current candidate feature vector extracted from the current candidate lesion region; p: Update coefficient, 0 < p < 1.

[0127] The model states that the features of the current lesion instance are obtained by weighted fusion of historical feature vectors and the current candidate feature vectors corresponding to newly observed current candidate lesion regions. As imaging conditions and observation dimensions become richer, the features of the lesion instance become more stable and complete.

[0128] The essence of this formula is to dynamically fuse the features of the same lesion across multiple frames using an exponential moving average, thereby constructing a stable feature representation of the lesion instance.

[0129] In some implementations, the capsule endoscopy diagnostic report generation device includes a time information binding module.

[0130] The time information binding module is used to obtain the acquisition time of the site representative image based on the frame number of the filtered site representative image and the capsule endoscope running time mapping function, and to use the acquisition time as the display attribute of the site representative image in the diagnostic report. The report output module generates a diagnostic report corresponding to the capsule endoscopy using images representing the body parts and their acquisition time.

[0131] In some implementations, the time information binding module is used to obtain the acquisition time of each image based on the frame number of each image and the capsule endoscope running time mapping function; classify each image to determine the digestive tract region captured by each image, the digestive tract region including the oral cavity, esophagus and target region, the target region being the stomach; The report output module generates a diagnostic report corresponding to the capsule endoscopy using the earliest time of images acquired by the capsule endoscopy in each digestive tract region, and / or the earliest time of images acquired by the capsule endoscopy outside a biological body.

[0132] The time information binding module can automatically generate time information consistent with the examination process by recognizing the starting frame of each digestive tract region and modeling the time axis, thus reducing the amount of manual data entry.

[0133] Each frame acquired by a capsule endoscope includes at least the following information: image frame number, image acquisition timestamp, and raw image data. For any digestive tract region and in vitro environment, the starting frame image is f. start The corresponding time is obtained by using the frame number of each image and the capsule endoscope runtime mapping function.

[0134] T region =FrameToTime(f start ), The meanings of each letter / symbol are as follows: T region The start time of a specific digestive tract region; FrameToTime is a function that maps frame numbers to time. f start This is the starting frame image corresponding to a certain digestive tract region, as identified by the algorithm.

[0135] Therefore, this embodiment can output the swallowing time of the capsule endoscope, the time it enters the esophagus, the time it passes through the cardia and enters the stomach, and the examination end time. This time information can be used not only for the report's first page summary but also for the time binding of representative images of the site and lesion in the subsequent diagnostic report.

[0136] T lesion =[t min , t max ], The meanings of each letter / symbol are as follows: T lesion The timeframe during which a specific lesion appears during the examination process; t min This refers to the time when the lesion first appeared. t max This refers to the time when the lesion was last observed.

[0137] Understandably, a similar method can be used to obtain the time range in which each candidate site appears during the inspection process.

[0138] The time information binding module can classify each image using image classification models and / or image processing to determine the digestive tract region captured in each image.

[0139] In some implementations, the time information binding module obtains the acquisition time of each candidate image based on the frame number of each candidate image and the capsule endoscope running time mapping function. The report output module uses the acquisition time of the site-representing image and / or the acquisition time of the lesion-representing image as display fields for the corresponding images in the diagnostic report, generating a diagnostic report for the capsule endoscopy. Some diagnostic reports can display both the site-representing image and its acquisition time, as well as the lesion-representing image and its acquisition time.

[0140] The lesion instance set includes at least one of the following: lesion instance identifier, candidate site of origin, representative image of lesion, and acquisition time, for use in the lesion image section of the diagnostic report.

[0141] In some implementations, the diagnostic report is used to present candidate sites, site information corresponding to the candidate sites, and lesion instance information within the candidate sites.

[0142] The location information corresponding to the candidate location may include the candidate location identifier, the time range in which the candidate location appears, the representative image of the candidate location and its acquisition time, etc.

[0143] The lesion instance information within the candidate site may include the lesion instance identifier, the time range in which the lesion instance appeared, the acquisition time of the representative image of the lesion, the set of lesion instances, and the textual description information corresponding to the lesion instance.

[0144] The textual description of the lesion instance may include at least the location of the lesion instance, the lesion category of the lesion instance, and its corresponding phrase or sentence-style description. In different implementations, the description content for labels below the lesion representative images, report text, structured fields, or electronic medical record interfaces may be generated by combining the lesion occurrence time range, the selection results of representative lesion images, the lesion morphology, and / or report template requirements.

[0145] In a preferred embodiment, the report output module can also synchronously generate image tag information below the representative lesion image and text description information in the report body based on the binding relationship between the same lesion instance and the corresponding candidate site. The image tag information includes at least the candidate site name and the lesion category name, and may further include the lesion instance identifier and / or the time of appearance, so that the image display area of ​​the representative lesion image and the text description area are consistent in terms of location, lesion category, and time information.

[0146] For example, when a lesion instance is associated with the gastric antrum region and the lesion type is polypoid elevation, the device can display "Gastric Antrum | Polypoid Elevation" below the corresponding image and generate the corresponding location-lesion description in the report text description, thereby reducing manual re-entry and inconsistencies between image and text information.

[0147] The above structured report organization method can achieve unified output of "location - lesion - time - text description", which is convenient for clinical image reading and follow-up.

[0148] It should be noted that the site segmentation module, site representative image filtering module, lesion image report supplementation module, lesion instance identification and association module, lesion representative image selection module, time information binding module, and report output module can be implemented in hardware.

[0149] The computational logic of each module can be carried out by the internal logic unit of a dedicated processing chip. The deployment architecture includes three feasible forms: a single module is independently configured with a single chip; a single module is collaboratively distributed among multiple chips; and multiple modules reuse the partitioned logic unit within the same chip for parallel or time-sharing processing. The modules can also be distributed across different hardware carriers within the system. In some implementations, some low-power real-time processing modules are integrated into the capsule endoscope's built-in chip to perform pre-processing image processing, while post-processing modules such as lesion matching, temporal modeling, and report generation are located within the chip of the in vitro testing machine. The two devices interact with each other via wireless or wired links, exchanging images and intermediate data.

[0150] Please refer to Figure 2 This application also provides a method for generating a capsule endoscopy diagnostic report, comprising: S1: Input each frame of image acquired by the capsule endoscope in the target area into the segmentation model, identify the candidate parts in the image, and store each frame of image as a candidate image in the corresponding candidate part set according to the different candidate parts in the image; S2: Based on the screening conditions corresponding to each candidate region, the corresponding region representative image is obtained from the candidate images contained in each candidate region set; among the screening conditions corresponding to each of all candidate regions, at least two candidate regions have different screening conditions, and at least one candidate region's screening condition is associated with geometric parameters. The geometric parameters characterize the spatial features of the candidate region within the corresponding candidate image, and are calculated by the segmentation mask output by the segmentation model after identifying the candidate region in the candidate image; and S3: Generate a diagnostic report corresponding to the capsule endoscope using the selected site representative images.

[0151] It should be noted that the steps in the method of the present invention can be executed sequentially, or partially parallelized or the order adjusted according to the actual system implementation. Those skilled in the art can make modifications or substitutions without departing from the spirit of the present invention.

[0152] For specific implementation details of the above methods, please refer to the detailed description of the above device implementation details, which will not be repeated here.

[0153] Please see Figure 3 This application provides an electronic device 300, including a memory 301 and a processor 302, wherein: Memory 301 is used to store computer programs; Processor 302 is configured to execute a computer program to implement the capsule endoscopy diagnostic report generation method as described in the embodiments of this application.

[0154] The memory 301 is a dedicated data and program storage hardware for electronic device 300. It can use high-speed flash memory, solid-state storage media or embedded read-only memory devices to persistently store lesion detection logic programs, various preset judgment thresholds, lesion feature sample data, life cycle status parameters and real-time acquired raw image data. At the same time, it temporarily caches feature vectors, instance matching results and acquisition completion rate data generated during the operation, ensuring stable access and retrieval of various types of data.

[0155] The processor 302, as the core computing and control hardware of the system, can be an embedded microprocessor chip, a multi-core processing unit or a dedicated intelligent computing processor. It is responsible for calling and reading program instructions and data in the memory 301, completing the entire process of lesion detection, lesion feature extraction, lesion instance similarity comparison, acquisition completion calculation, life cycle status bit update, etc., and coordinating the orderly operation of various functions of the electronic device 300.

[0156] This application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the capsule endoscopy diagnostic report generation method described above.

[0157] The computer-readable storage medium can be any medium with data read / write and persistent storage capabilities, such as solid-state drives, flash memory chips, read-only memory, random access memory, portable hard drives, and embedded storage modules. It can stably store program code, algorithm parameters, and business configuration data, meeting the program integration and data storage requirements of embedded devices and intelligent diagnostic and treatment devices.

[0158] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer storage medium, such as a read-only memory, a disk, or an optical disk.

[0159] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0160] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0161] Furthermore, various different embodiments of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the embodiments of the present invention, they should also be regarded as the content disclosed by the embodiments of the present invention.

Claims

1. A capsule endoscopy diagnostic report generation device, characterized in that, include: Part segmentation module: Input each frame of image acquired by the capsule endoscope in the target area into the segmentation model, identify the candidate parts present in the image, and store each frame of the image as a candidate image in the corresponding candidate part set according to the different candidate parts in the image; The part representative image filtering module: Based on the filtering conditions corresponding to each candidate part, it filters the candidate images contained in each candidate part set to obtain the corresponding part representative image; among the filtering conditions corresponding to each of the candidate parts, at least two candidate parts have different filtering conditions, and at least one candidate part has a filtering condition associated with geometric parameters. The geometric parameters characterize the spatial features of the candidate part in the corresponding candidate image. The geometric parameters are calculated by the segmentation mask output by the segmentation model after identifying the candidate part in the candidate image. and Report output module: Generates a diagnostic report corresponding to the capsule endoscope using the filtered images representing the affected areas.

2. The capsule endoscopy diagnostic report generation device as described in claim 1, characterized in that, The geometric parameters include at least one of boundary contact state, location centering, size, and shape; wherein: The boundary contact state is used to characterize whether the outline of the candidate part contacts the image boundary of the candidate image to which it belongs. The centering degree of the region refers to the degree to which the candidate region is centered in the candidate image; The size includes the pixel length, pixel area and / or area fraction of the target subject corresponding to the candidate part in its candidate image, and the target subject includes the candidate part, the segmentation unit in the candidate part, or the candidate part related to the candidate part; The shape is characterized by the roundness corresponding to the contour of the candidate part.

3. The capsule endoscopy diagnostic report generation device as described in claim 2, characterized in that, The boundary contact state is defined as follows: if the candidate part contacts the boundary of the candidate image to which it belongs, the boundary contact state is 0; otherwise, the boundary contact state is 1. The formula for calculating the centering of the aforementioned location is as follows: ， ， S center (r i )=1-d i / D max , in: d i Let be the distance between the center of the i-th candidate region and the center of its corresponding candidate image. (x c ,y c () represents the center coordinates of the candidate image to which the i-th candidate region belongs. (x i ,y i Let be the center coordinates of the i-th candidate region within its respective candidate image. D max The length of the diagonal of the candidate image. S center (r i ) represents the i-th candidate region r i The closer the value is to 1, the stronger the candidate part r is. i The closer the center is to the center of its corresponding candidate image; The formula for calculating the roundness is as follows: S circle (r i )=4πA i / P i 2 , in: S circle (r i ) represents the i-th candidate region r i Circularity in its candidate image A i For the i-th candidate part r i The pixel area in its candidate image. P i The candidate region r in the candidate image i The pixel perimeter; The area fraction includes a large area fraction and a small area fraction, and the calculation formula is as follows: S area (r i )=AR i / A max , S small (r i )=1-AR i / A max , in: AR i For the i-th candidate part r i The corresponding pixel area of ​​the target subject in its respective candidate image. A max The maximum pixel area of ​​the target subject among all candidate images in the same set of candidate regions, or the maximum preset pixel area of ​​the target subject. S area (r i Let r be the i-th candidate part. i The larger the area of ​​the target subject within its respective candidate image, the higher the S value. area (r i The closer the value of ) is to 1, the better. S small (r i ) represents the i-th candidate region r within the candidate image. i The smaller the area of ​​the target subject in the candidate image, the lower the S fraction. small (r i The closer the value of ) is to 1; The formula for calculating the extension length is: ， in: L elong (r i ) is the candidate region r i The corresponding extension length of the target subject is used to characterize the elongation of the target subject identified within the candidate image. x i max x i min y i max y i min The extreme coordinates of the bounding box corresponding to the target body, output by the segmentation model.

4. The capsule endoscopy diagnostic report generation device as described in claim 3, characterized in that, The candidate site is the cardia, and the corresponding screening criteria for the cardia are: Calculate the comprehensive score of each candidate image in the candidate image set corresponding to the cardia as the representative image of the region. Select several candidate images as the representative images of the region according to the comprehensive score from high to low. The formula for calculating the comprehensive score of the region is as follows: Score cardia (r i )=b1*S center (r i )+ b2*S circle (r i )+ b3*S area (r i ) , Among them, Score cardia (r i The candidate image containing the cardia is used as the representative image of the region for a comprehensive score. S area (r i ) represents the i-th candidate region r i The largest area fraction within its respective candidate image, b1, b2, and b3 are weighting coefficients.

5. The capsule endoscopy diagnostic report generation device as described in claim 3, characterized in that, The candidate site is the gastric angle, and the screening conditions corresponding to the gastric angle include a first screening condition and a second screening condition. When the location-representing image filtering module filters the location-representing images from the candidate image set corresponding to the gastric angle, there are two execution modes: Mode 1: Select either the first filter condition or the second filter condition to execute; Mode 2: The first filtering condition is executed first. If there is no candidate image in the candidate image set corresponding to the gastric angle that meets the first filtering condition, the second filtering condition is executed. The first filtering condition is: Obtain the boundary contact state of the gastric angle in each candidate image within the candidate image set corresponding to the gastric angle. Calculate the comprehensive score of each candidate image as the representative image of the region. Sort all candidate images from high to low according to the comprehensive score of the region. Select several candidate images with a comprehensive score of non-zero as the representative images of the region. The formula for calculating the comprehensive score of the region is as follows: Score angle (r i )=B(ri)*[ d1*L elong (r i )+ d2*S area (r i )], Among them, Score angle (r i The candidate images containing the gastric angle are used as the representative images of the region for a comprehensive score. L elong (r i ) represents the i-th candidate region r i The extension length within its respective candidate image, S area (r i Let r be the i-th candidate part. i The largest area fraction within its respective candidate image, d1 and d2 are weighting coefficients; The second screening criterion is: Calculate the extension length of the gastric angle in each candidate image in the candidate image set corresponding to the gastric angle, sort the candidate images from largest to smallest according to the extension length values, and select several candidate images as representative images of the region.

6. The capsule endoscopy diagnostic report generation device as described in claim 3, characterized in that, The candidate site is the gastric antrum, and the corresponding screening criteria for the gastric antrum are: Calculate the comprehensive site score for each candidate image in the candidate image set corresponding to the gastric antrum as a representative image of the site. Then, select several candidate images with a non-zero comprehensive site score as the representative images of the site, in descending order of the comprehensive site score. The formula for calculating the comprehensive site score is as follows: Score antrum (r i )=B(r i )*[ g1*S small (r i )+ g2*S center (r i )+ g3*S circle (r i )] , Among them, Score antrum (r i The candidate images containing the gastric antrum are used as site representative images for a comprehensive site score. S small (r i ) represents the i-th candidate region r i The small fraction of the area within its candidate image. g1, g2, and g3 are weighting coefficients.

7. The capsule endoscopy diagnostic report generation device as described in claim 3, characterized in that, The candidate site is the pylorus, and the corresponding screening criteria for the pylorus are: Images with a boundary contact state of 1 within the candidate image set corresponding to the pylorus are selected. The images are sorted from high to low according to the confidence level of identifying the pylorus in the candidate images corresponding to the segmentation model. Several candidate images are selected as representative images of the region.

8. The capsule endoscopy diagnostic report generation device as described in claim 3, characterized in that, Among the filtering conditions corresponding to each of the candidate parts, at least one filtering condition corresponding to a candidate part is associated with the coexistence relationship between the candidate part and other candidate parts.

9. The capsule endoscopy diagnostic report generation device as described in claim 8, characterized in that, The candidate site is the gastric body, and the target region is the stomach. Other candidate sites included in the stomach include the cardia, fundus, antrum, and angle. The gastric body comprises two segmented units: the lesser curvature and the greater curvature. The screening criteria for the gastric body include a third, a fourth, and a fifth screening criterion. When the site-representative image filtering module filters the site-representative images from the candidate image set corresponding to the stomach body, there are three execution modes: Mode 3: If the candidate image contains the lesser curvature of the stomach and at least one of the cardia, fundus, and gastric cavity, then the third screening condition is executed; Mode 4: If the candidate image contains the greater curvature of the stomach and the gastric cavity, then the fourth screening condition is executed; Mode 5: If the candidate image contains a gastric cavity or antrum and a gastric angle with a boundary contact state of 1, then the fifth screening condition is executed; The gastric cavity is a dark area presented in the candidate image, formed by at least one of the gastric mucosal folds, lesser curvature, and greater curvature. The third screening criterion is: Calculate the comprehensive score of each candidate image in the candidate image set corresponding to the stomach body as the representative image of the region. Select several candidate images as the representative images of the region according to the comprehensive score from high to low. The formula for calculating the comprehensive score of the region is as follows: N region (I t )=∑ k 1(Region k existsinI t ), Score body (I t )=h1N region (I t )+ h2L lesser (I t ), N region (I t ) represents the candidate image I in frame t. t The number of visible structures, including the lesser curvature, cardia, fundus, and gastric cavity, is specified. Region k For the kth visible structure, Region k existsinI t This represents the k-th visible structure in the candidate image I of frame t. t It was identified in the middle. 1(·) is an indicator function; it is 1 if the condition is met, and 0 otherwise. Score body (I t The candidate image containing the stomach body is used as a representative image of the region for a comprehensive score. h1 and h2 are weighting coefficients, and h1 > h2 > 0; L lesser (I t Let be the extension length of the lesser curvature of the stomach in the candidate image of frame t; The fourth screening criterion is: Several candidate images are selected as representative images of the region in ascending order of pixel area of ​​the gastric cavity; The fifth screening condition is: Several candidate images are selected as representative images of the region in ascending order of pixel area of ​​the gastric angle.

10. The capsule endoscopy diagnostic report generation device as described in claim 8, characterized in that, The candidate site is the fundus of the stomach; the target region is the stomach, and other candidate sites included in the stomach include the cardia. The site represents the image filtering module filtering candidate images that meet the first preferred condition from the candidate image set corresponding to the fundus of the stomach and copying them to the first preferred set, and filtering candidate images that meet the second preferred condition and copying them to the second preferred set. The first preferred condition is that the candidate image contains the cardia and the boundary contact state of the gastric fundus is 1. The second preferred condition is that the boundary contact state of the gastric fundus in the candidate image is 1. When filtering the images representing the corresponding parts of the gastric fundus, the seventh or eighth filtering condition is used; The seventh screening condition is to sort the candidate images in descending order of the pixel area of ​​the gastric fundus, and select several candidate images from the first preferred set, the second preferred set, or the candidate image set corresponding to the gastric fundus as representative images of the gastric fundus. The eighth screening condition: The candidate images are sorted from largest to smallest according to the pixel area of ​​the gastric fundus, and a number of candidate images are selected from the first preferred set as representative images of the region. If the first preferred set does not contain any candidate images, then a number of candidate images are selected from the second preferred set as representative images of the region. If the second preferred set does not contain the candidate image, then select several candidate images from the candidate image set corresponding to the gastric fundus as the representative images of the region.

11. The capsule endoscopy diagnostic report generation device according to any one of claims 1-10, characterized in that, include: Lesion Image Report Supplement Module: Used to receive a set of lesion instances and use them as optional input to the lesion image section of the diagnostic report; Lesion representative image selection module: When there is a set of lesion instances, calculate the comprehensive lesion score of each candidate image in each lesion instance set as the representative image of the lesion, and select several representative images of the lesion from the candidate images corresponding to the same lesion instance according to the comprehensive lesion score; and The report output module generates a diagnostic report corresponding to the capsule endoscope using the selected representative images of lesions.

12. The capsule endoscopy diagnostic report generation device as described in claim 11, characterized in that, The lesion representative image selection module calculates the comprehensive lesion score for each candidate image as a representative lesion image based on the clarity of the candidate image, the pixel area or pixel area ratio of the lesion region in the candidate image, the lesion centering of the lesion region in the candidate image, and the representativeness score of the lesion region in the candidate image.

13. The capsule endoscopy diagnostic report generation device as described in claim 12, characterized in that, The formula for calculating the comprehensive lesion score is as follows: Score(f i )=k1Q i +k2AE i + k3C i + k4R i , ， ， C i =1-d i / D max , in: Score(f i Candidate image f i Comprehensive lesion score as a representative image of the lesion; Q i Candidate image f i The image quality score is related to the candidate image's sharpness, exposure quality, and degree of occlusion. AE i Candidate image f i The pixel area of ​​the lesion region, or the lesion region in the candidate image f. i The percentage of pixel area within; C i Candidate image f i The degree of lesion centering in the lesion area is indicated by a value closer to 1, meaning the center of the lesion area is closer to the candidate image f. i The center; R i Candidate image f i A representative score for the lesion region, wherein the representative score is correlated with at least one of the confidence level of the lesion region, the boundary contact state of the lesion region, and the candidate site to which the lesion region belongs; k1, k2, k3, and k4 are the weights corresponding to each parameter, all of which are greater than or equal to 0, satisfying k1+k2+k3+k4=1; d i Let be the distance between the center of the i-th lesion region and the center of its corresponding candidate image. (x c ,y c () represents the center coordinates of the candidate image to which the i-th lesion region belongs. (x i ,y i Let be the center coordinates of the i-th lesion region within its respective candidate image. D max The length of the diagonal of the candidate image.

14. The capsule endoscopy diagnostic report generation device as described in claim 12, characterized in that, The lesion representative image selection module stores the lesion representative image into a candidate lesion image set, satisfying the constraint |R| on the number of candidate images in the candidate image subset. l |≤M l In this case, a subset of candidate images is selected from the set of lesion images according to the following formula: ， in, λ is the diversity constraint coefficient. F l This is the set of candidate lesion images for the l-th lesion instance; R l To select from the set of candidate lesion images F l The selected candidate image subsets, each candidate image subset storing several representative images of lesions; M l The maximum number of lesion-representing images allowed to be included in the diagnostic report for each lesion instance; For one of the candidate image subsets R l The sum of the comprehensive lesion scores of all lesion-representing images within the image; is the sum of the similarities of any two different lesion representative images in the candidate image subset R that are calculated only once in the manner of i<j l ; argmax(·) selects the subset of candidate images R with the largest numerical value. l ; The report output module utilizes the filtered candidate image subset R l Generate a diagnostic report corresponding to the capsule endoscope.

15. The capsule endoscopy diagnostic report generation device as described in claim 11, characterized in that, include: Lesion instance identification and association module: Utilizes the target detection model to detect lesion instances in candidate images in each candidate site set, and copies candidate images with the same lesion instance to the corresponding lesion instance set; The lesion image report supplement module is used to receive the set of lesion instances output by the lesion instance identification and association module, and use it as an optional input to the lesion image section of the diagnostic report; The report output module is used to generate a diagnostic report by binding the set of lesion instances with their respective candidate sites.

16. The capsule endoscopy diagnostic report generation device according to any one of claims 1-10, characterized in that, include: The time information binding module is used to obtain the acquisition time of the representative image of the part based on the frame number of the filtered part representative image and the capsule endoscope running time mapping function, and to use the acquisition time as the display attribute of the part representative image in the diagnostic report. The report output module generates a diagnostic report corresponding to the capsule endoscope using the representative image of the location and its acquisition time.

17. The capsule endoscopy diagnostic report generation device according to any one of claims 1-10, characterized in that, include: The time information binding module obtains the acquisition time of each candidate image based on the frame number of each candidate image and the capsule endoscope running time mapping function; The report output module uses the acquisition time of the image representing the location and / or the acquisition time of the image representing the lesion as the display field of the corresponding image in the diagnostic report to generate the diagnostic report corresponding to the capsule endoscope.

18. A method for generating a capsule endoscopy diagnostic report, characterized in that, include: Each frame of image acquired by the capsule endoscope in the target area is input into the segmentation model to identify the candidate regions in the image. Based on the different candidate regions in the image, each frame of the image is stored as a candidate image in the corresponding candidate region set. Based on the filtering conditions corresponding to each candidate region, a representative image of the corresponding region is obtained from the candidate images contained in each candidate region set; among the filtering conditions corresponding to each of the candidate regions, at least two candidate regions have different filtering conditions, and at least one candidate region has a filtering condition associated with a geometric parameter. The geometric parameter characterizes the spatial features of the candidate region in the corresponding candidate image. The geometric parameter is calculated by the segmentation mask output by the segmentation model after identifying the candidate region in the candidate image. and A diagnostic report corresponding to the capsule endoscope is generated using the selected site representative images.

19. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the capsule endoscopy diagnostic report generation method as described in claim 18.

20. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the capsule endoscopy diagnostic report generation method as described in claim 18.

Images