Intelligent Grading Method and System for Endoscopic Images of Ulcerative Colitis

By detecting and matching intervention events in endoscopic image sequences, identifying mucosal regions and constructing evidence co-location maps, the problem of unstable endoscopic image grading in existing technologies is solved, achieving more accurate inflammation grading.

CN123118565APending Publication Date: 2026-11-17THE SECOND AFFILIATED HOSPITAL OF HUNAN UNIV OF TRADITIONAL CHINESE MEDICINE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611366939.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing endoscopic image grading methods for ulcerative colitis are difficult to accurately correspond to image areas of the same mucosal site, especially under the influence of flushing or suction events, resulting in unstable grading results.

Method used

By detecting intervention events, matching image sequences before and after the intervention, identifying mucosal regions and assigning them the same labels, extracting local features to generate candidate inflammatory evidence objects, constructing a co-bitmap of inflammatory evidence, and performing image classification to obtain accurate inflammatory grading results.

Benefits of technology

It reduces the unstable impact of intervention events on image representation and improves the accuracy and reliability of endoscopic image inflammation grading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118565A_ABST
    Figure CN123118565A_ABST
Patent Text Reader

Abstract

This invention discloses an intelligent grading method and system for endoscopic images of ulcerative colitis. The method includes: acquiring pre-collected target colonoscopy videos; detecting intervention events; determining the pre-intervention image sequence and post-intervention image sequence corresponding to the intervention events; identifying corresponding mucosal regions and assigning identical mucosal location identifiers; extracting local image regions from the corresponding mucosal regions; determining the evidence type; generating candidate inflammatory evidence objects; generating authenticity verification results based on mucosal location identifiers, image matching results, and the image positions and local features of candidate inflammatory evidence objects in consecutive video frames; determining the candidate inflammatory evidence objects to be retained and their evidence reliability based on the authenticity verification results; and constructing an inflammatory evidence co-location map; classifying the corresponding mucosal regions to obtain ordered image categories, which serve as the endoscopic image inflammation grading results. This invention can reduce the impact of intervention events on the endoscopic image inflammation grading results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an intelligent grading method and system for endoscopic images of ulcerative colitis. Background Technology

[0002] Existing methods for grading endoscopic images of ulcerative colitis typically extract color, texture, shape, or edge features from a single endoscopic image or multiple video frames of an endoscopic video, and obtain the inflammation grading result of the endoscopic image through a classification model or fusion of multi-frame results.

[0003] However, pre-acquired colonoscopy videos may record flushing or suction events. Influenced by changes in flushing fluid, suctioned fluid, and mucosal surface deposits, inflammation-related image representations of the same mucosal site may remain, disappear, reappear, or shift position before and after the intervention event. Existing methods typically focus on extracting image features from each video frame and fusing classification results. This makes it difficult to accurately correlate image regions representing the same mucosal site before and after the intervention event, and also difficult to determine the mode of participation in subsequent image classification based on changes in the state of the corresponding image representations. Therefore, unstable image representations formed under the influence of the intervention event may interfere with the endoscopic image inflammation grading results. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an intelligent grading method for endoscopic images of ulcerative colitis, comprising the following steps:

[0006] Acquire pre-collected target colonoscopy videos, detect intervention events including flushing or suction events, and determine the pre-intervention image sequence and post-intervention image sequence corresponding to the intervention events;

[0007] Match the pre-intervention image sequence and the post-intervention image sequence to determine the corresponding mucosal regions that characterize the same mucosal site, and assign the same mucosal location identifier to the corresponding mucosal regions that match each other;

[0008] Local image regions are extracted from the corresponding mucosal regions, and the evidence type is determined based on the color, texture, shape, or edge features of the local image regions. Candidate inflammatory evidence objects with the evidence type and mucosal location identifier are generated.

[0009] Based on the mucosal location identifier, image matching results, and the image position and local features of the candidate inflammatory evidence objects in consecutive video frames, the candidate inflammatory evidence objects before and after the intervention event are determined to be in a state of retention, disappearance, reappearance, or position movement, and an authenticity verification result is generated.

[0010] Based on the authenticity verification results, the candidate inflammatory evidence objects to be retained and their evidence reliability are determined. The retained candidate inflammatory evidence objects are used as graph nodes, and their evidence type, mucosal location identifier and evidence reliability are set as graph node attributes. Spatial colocation connection edges and temporal continuity connection edges are established between the graph nodes to construct an inflammatory evidence colocation graph.

[0011] Based on the evidence combination formed by the attributes of the graph nodes, spatial co-location connections, and temporal persistence connections, the corresponding mucosal regions are classified to obtain ordered image categories that characterize the degree of inflammation in the images, which serve as the endoscopic image inflammation grading results.

[0012] In a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, the detection of the intervention event includes:

[0013] Calculate the optical flow field of adjacent video frames in the target colonoscopy video, and extract the fluid motion region and fluid coverage region;

[0014] When the optical flow vector in the fluid motion region moves from the diffusion center region to the periphery and the area of ​​the fluid coverage region increases with time, it is determined as a flushing event;

[0015] When the optical flow vector moves from the periphery to the convergence center region and the area of ​​the fluid-covered region decreases over time, it is determined to be an attraction event;

[0016] Video frames containing visible mucosal regions are selected from before and after the intervention event to form the pre-intervention image sequence and the post-intervention image sequence.

[0017] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, determining the corresponding mucosal region includes:

[0018] Brightness correction and color standardization were performed on the unobstructed mucosal background area before and after the intervention. At least two feature points were extracted from blood vessel bifurcation, fold edges and local mucosal texture and matched.

[0019] The non-rigid deformation field is calculated based on the matched feature points. The mucosal region in the post-intervention image sequence is mapped to the pre-intervention image sequence, and the same mucosal location identifier is assigned to the overlapping mucosal regions after mapping, thus obtaining the corresponding mucosal region.

[0020] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, the evidence types include one or more of the following: reduced vascular texture, mucosal erythema, bleeding, exudation, erosion, and ulceration.

[0021] The determination of the hold state, disappearance state, reappearance state, and position movement state includes:

[0022] If, under the same mucosal location marker, candidate inflammatory evidence objects of the same type exist both before and after the intervention and do not meet the criteria for reappearance, the state is determined to be "maintained". If, only the corresponding candidate inflammatory evidence objects exist before the intervention, and the corresponding mucosal area meets the extraction criteria for the candidate inflammatory evidence objects after the intervention but does not meet the criteria for positional movement, the state is determined to be "disappeared". If the candidate inflammatory evidence objects are visible before the intervention, are occluded or invisible during the intervention, and reappear under the same mucosal location marker after the intervention, the state is determined to be "reappeared".

[0023] The predicted displacement of the mucosal background determined by the non-rigid deformation field is subtracted from the actual displacement of the candidate inflammatory evidence object. When the direction of the remaining displacement is consistent with the optical flow direction of the fluid motion region and the mucosal position marker changes, it is determined to be a position movement state. When both the reappearance state and the maintenance state are satisfied, it is preferentially determined to be a reappearance state. When both the position movement state and the disappearance state are satisfied, it is preferentially determined to be a position movement state.

[0024] The authenticity verification results corresponding to the maintained state or the reappearance state are identified as fixed mucosal origin; the authenticity verification results corresponding to the positional movement state and the bleeding or exudation type that disappeared after intervention and did not reappear are identified as fluid interference; the authenticity verification results of other candidate inflammatory evidence objects are identified as pending verification; candidate inflammatory evidence objects whose authenticity verification results are fixed mucosal origin or pending verification are retained, and candidate inflammatory evidence objects whose authenticity verification results are fluid interference are excluded.

[0025] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, the construction of the inflammatory evidence co-bitmap and image classification include:

[0026] Spatial colocation connection edges are established between graph nodes whose mucosal location identifiers correspond to the same or overlapping mucosal regions, and temporal continuity connection edges are established between graph nodes whose mucosal location identifiers and evidence types are the same in different video frames.

[0027] From samples labeled with image inflammation levels, determine the evidence type, evidence reliability, and connection edge combination corresponding to each ordered image category to form a preset level evidence pattern;

[0028] Based on the degree of matching between the connection subgraph formed by candidate inflammatory evidence objects from fixed mucosa and the preset level evidence pattern, the ordered image category corresponding to the preset level evidence pattern with the highest matching degree is determined as the endoscopic image inflammation grading result.

[0029] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, it further includes:

[0030] Extract at least two contact characterization data from the target colonoscopy video, including local image magnification change, radial optical flow, mucosal color change, reflective area change, and degree of defocus;

[0031] When at least two contact characterization data change synchronously within the same continuous video frame segment and the corresponding change areas have a common spatial center, a lens contact event is determined. A lens contact trajectory is formed based on the position change of the common spatial center, and the video frame corresponding to when the at least two contact characterization data no longer change synchronously is determined as the video frame corresponding to the lens release event.

[0032] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, it further includes:

[0033] Map the lens contact trajectory to the mucosal location marker;

[0034] For candidate inflammatory evidence objects of bleeding type that were not extracted from the visible mucosal area before the lens contact event, and candidate inflammatory evidence objects of bleeding type that appeared for the first time after the lens contact event and whose mucosal position identifiers coincided with the mucosal position identifiers corresponding to the lens contact trajectory, contact association markers were set, and their authenticity verification results were determined as to be verified and retained as graph nodes.

[0035] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, it further includes:

[0036] The target colonoscopy video is divided into the advance stage and the withdrawal stage based on the direction of camera movement and the order in which intestinal segments appear.

[0037] Based on at least two region matching features among intestinal segment sequence, fold topology, vascular bifurcation, and local mucosal texture, match mucosal regions representing the same mucosal site in two stages and assign the same mucosal location identifier.

[0038] Video frames that are not in contact with the lens, are not obstructed, do not have saturated reflection or severe defocus, and meet the extraction conditions for candidate inflammatory evidence objects of bleeding type are selected from the matched mucosal region to form a two-way composite image.

[0039] As a preferred embodiment of the intelligent grading method for endoscopic images of ulcerative colitis described in this invention, it further includes:

[0040] Two-way diplopia verification was performed on candidate inflammatory evidence objects with contact association markers for bleeding types;

[0041] When candidate inflammatory evidence objects with the same mucosal location identifier, overlapping local image regions, and similar color or shape are extracted from other bi-path composite images, they are identified as duplicates, their authenticity verification results are modified to fix the mucosal source and retained.

[0042] When the corresponding mucosal areas in both the advance and retreat phases meet the extraction conditions and do not repeat, the authenticity verification result is modified to mechanical interference related to lens contact and excluded.

[0043] If a video frame that meets the extraction conditions cannot be obtained from both stages, its authenticity verification result remains unverified.

[0044] The inflammatory evidence co-bitmap is updated based on the revised authenticity verification results, and the images are reclassified.

[0045] The present invention also provides an intelligent grading system for endoscopic images of ulcerative colitis, including a processor and a memory storing a computer program. When the computer program is executed by the processor, it implements the steps of the above-described intelligent grading method for endoscopic images of ulcerative colitis.

[0046] The beneficial effects of this invention are:

[0047] 1. This invention generates authenticity verification results based on the response status of candidate inflammatory evidence objects before and after the intervention event, and determines the candidate inflammatory evidence objects to be retained and the reliability of the evidence accordingly, thereby reducing the impact of unstable image representation caused by the intervention event on the grading results.

[0048] 2. This invention defines the spatial co-location relationship and temporal persistence relationship between graph nodes by using spatial co-location connection edges and temporal persistence connection edges, and combines the evidence type and evidence reliability of graph nodes to form evidence combinations, thereby reducing the situation where candidate inflammatory evidence objects lacking corresponding spatial co-location relationships or temporal persistence relationships are combined for image classification.

[0049] 3. This invention sets a contact association marker for the first bleeding object after lens contact and performs dual-path diplopia verification, thereby updating the authenticity verification results and the co-location map of inflammatory evidence, reducing the interference of lens contact-related bleeding characteristics. Attached Figure Description

[0050] The following is a brief description of the accompanying drawings used in the embodiments of the present invention:

[0051] Figure 1 This is a flowchart illustrating the overall process of the intelligent grading method for endoscopic images of ulcerative colitis according to the present invention.

[0052] Figure 2 This is a flowchart illustrating the determination of intervention events and image sequences before and after intervention in the intelligent grading method for endoscopic images of ulcerative colitis according to the present invention.

[0053] Figure 3 This is a flowchart illustrating the verification results, retention processing, and determination of the reliability of candidate inflammatory evidence objects in the intelligent grading method for endoscopic images of ulcerative colitis of the present invention.

[0054] Figure 4 This is a flowchart illustrating the construction of the inflammatory evidence co-bitmap and image classification process for the intelligent grading method of endoscopic images of ulcerative colitis according to the present invention.

[0055] Figure 5 This is a flowchart illustrating the lens contact event determination and contact association marker setting process for the intelligent grading method for endoscopic images of ulcerative colitis according to the present invention.

[0056] Figure 6 This is a flowchart illustrating the dual-path diplopia verification and inflammatory evidence co-bitmap update process of the intelligent grading method for endoscopic images of ulcerative colitis according to the present invention. Detailed Implementation

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0059] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0060] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Example 1, referring to Figures 1 to 4This embodiment provides an intelligent grading method for endoscopic images of ulcerative colitis. The method is executed by a computing device, which reads pre-acquired and stored target colonoscopy videos and performs offline or near-real-time image processing. It does not control flushing, suction, endoscope insertion, endoscope withdrawal, or lens contact, and outputs ordered image categories characterizing the degree of inflammation in the images.

[0062] The thresholds or parameters for motion direction, area change, image quality, region overlap, feature similarity, object correspondence, evidence reliability, and pattern matching involved in this embodiment are all predetermined. Candidate thresholds or parameters are determined on training samples labeled with intervention events, mucosal regions, candidate inflammatory evidence objects, and image inflammation levels, and validated on independent validation samples. Specifically, thresholds related to motion direction and area change are determined based on the event detection F1 score; thresholds related to region overlap and feature similarity are determined based on the region matching accuracy; thresholds related to object correspondence are determined based on the object correspondence accuracy; evidence reliability mapping parameters are determined based on the accuracy and reliability calibration error determined by the authenticity verification results; and preset level evidence patterns, pattern matching thresholds, and corresponding weights are determined based on the image category consistency rate. Thresholds or parameters that enable the corresponding evaluation indicators to meet preset requirements or achieve optimal validation results are selected and fixed before processing the target colonoscopy video.

[0063] S1: Identify the intervention event and the pre-intervention image sequence and post-intervention image sequence.

[0064] S11: Acquire and process the target colonoscopy video.

[0065] Specifically, the target colonoscopy video is decoded into time-stamped video frames, with uniform size and color space. Based on local gradient energy or Laplacian response, the proportion of saturated highlights and dark pixels, and the proportion of mucosa not covered by fluid or severely out of focus, sharpness, exposure status, and visible proportions are determined and normalized to obtain image quality information. .

[0066] S12: Extract the fluid movement area and the fluid coverage area.

[0067] The optical flow field of adjacent video frames is calculated; feature points are matched from identifiable and unobstructed mucosal background regions, and background motion transformation is obtained through robust estimation. The remaining optical flow is obtained by subtracting the background motion component from the optical flow. Regions whose remaining optical flow amplitude, direction consistency, and connectivity area meet preset conditions are identified as fluid motion regions; combined with the color or texture changes of adjacent frames after mapping, candidate regions are connected and merged, hole-filled, and boundary smoothed to obtain the fluid-covered region and its frame-by-frame area.

[0068] S13: Identify the diffusion center area, convergence center area, and intervention events.

[0069] Specifically, radial voting is performed on the remaining optical flow in the fluid motion region: a diffusion center region is determined when the overall optical flow moves outward from the candidate center and the radial consistency reaches a threshold; a convergence center region is determined when the overall optical flow moves from the periphery towards the candidate center. The slope of the fluid coverage area change is fitted within a sliding time window; video frame segments that continuously satisfy outward movement and increase in area are determined as flushing events, and video frame segments that continuously satisfy inward movement and decrease in area are determined as attraction events. The first and last frames are used as event boundaries, and adjacent events of the same type are merged when the interval is less than a preset value.

[0070] S14: Generate pre-intervention image sequences and post-intervention image sequences.

[0071] Specifically, within preset time windows before the start frame and after the end frame of the intervention event, video frames that are not obscured by fluid, saturated reflection, or severe defocusing, and that retain vascular bifurcation, fold edges, or local mucosal texture, are selected; priority is given to selecting frames with a larger visible mucosal area and higher image quality information. The high number of video frames forms the pre-intervention image sequence and the post-intervention image sequence.

[0072] S2: Determine the corresponding mucosal region and mucosal location marker.

[0073] Based on the pre-intervention image sequence, post-intervention image sequence, and fluid coverage area obtained in S1, S2 is executed, which specifically includes the following sub-steps:

[0074] S21: Obtain the unobstructed mucosal background area and perform image correction.

[0075] A fluid coverage mask is generated based on the fluid coverage area obtained in S1, and an invalid area mask is generated by combining the saturated reflective area and the severely out-of-focus area. The mucous membrane area outside the fluid coverage mask and the invalid area mask is identified as the unmasked mucous membrane background area.

[0076] Using the unobstructed mucosal background area in the pre-intervention image sequence as a reference, brightness correction and color standardization were performed on the video frames before and after the intervention based on the brightness statistics and color channel statistics of the unobstructed mucosal background area in the pre-intervention and post-intervention image sequences.

[0077] S22: Extract and match mucosal feature points.

[0078] Specifically, at least two types of feature points are extracted from the corrected video frames, including vascular skeleton bifurcation points, fold edge direction change points, and stable local mucosal texture points. Descriptive information including grayscale, color, gradient direction, and neighborhood structure is established. Candidate matching point pairs are formed according to the descriptive distance, and incorrect matches are excluded based on relative distance, direction, and local geometric consistency.

[0079] S23: Calculate the non-rigid deformation field and assign mucosal position identifiers.

[0080] The retained post-intervention feature points were used as control points. The target point is the feature point before intervention that matches it. Calculate the non-rigid transformation from the coordinate system of the post-intervention image sequence to the coordinate system of the pre-intervention image sequence. .

[0081] In one specific implementation, thin-plate spline transformation is used as a non-rigid transformation. Its form is:

[0082]

[0083] in, and For affine components, To control points The corresponding non-rigid weights, To prevent a default positive number where the logarithmic input is zero, the value is determined by minimizing the sum of the control point mapping error and the bending energy regularization term. , and The regularization term weights are determined based on the feature point mapping error and deformation smoothness on the validation samples. (Non-rigid transformation) A non-rigid deformation field is formed between the corresponding video frames.

[0084] A non-rigid deformation field is used to map the mucosal regions in the post-intervention image sequence to the pre-intervention image sequence. When the cross-union ratio of the two mapped mucosal regions reaches a preset region overlap threshold, and the proportion of effective matching feature points within the region reaches a preset feature consistency threshold, the two mucosal regions are determined to be mutually matched and represent the same mucosal site. They are then identified as corresponding mucosal regions and assigned the same mucosal location identifier.

[0085] Mucosal location identifiers are used to identify mucosal sites rather than image coordinates. For corresponding mucosal regions representing the same mucosal site in different video frames, the same mucosal location identifier is used even if their image position, area, or shape changes; different mucosal sites use different mucosal location identifiers.

[0086] S3: Generate candidate inflammation evidence objects.

[0087] Based on the corresponding mucosal region, non-rigid deformation field, and mucosal location marker determined in S2, S3 is executed, specifically including the following steps:

[0088] S31: Extract local image regions.

[0089] Specifically, initial regions are formed in the corresponding mucosal regions before and after intervention based on pixel color and local gradient, and then merged into local image regions according to color, texture similarity and edge continuity; pixels in the invalid region mask do not participate in recognition.

[0090] S32: Determine the type of evidence based on local image features.

[0091] Color, texture, morphology, or edge features are extracted from each local image region. Color features include color channel statistics, local color distribution, and color differences between the local image region and the surrounding mucosa; texture features include local gradient distribution, texture direction, vascular linear structure density, and vascular bifurcation density; morphological features include region area, aspect ratio, roundness, boundary shape, and region center location; edge features include edge strength, continuity, closure degree, and gradient differences between the inside and outside of the region. These features are then arranged in a fixed order to form a feature vector for the local image region. .

[0092] Evidence types include one or more of the following: reduced vascular texture, mucosal erythema, hemorrhage, exudation, erosion, and ulceration. Reduced vascular texture mainly corresponds to a decrease in the density of linear or bifurcated blood vessels relative to the surrounding mucosa; mucosal erythema mainly corresponds to a local increase in red-related color channels and is distributed on the mucosal surface; hemorrhage mainly corresponds to red or dark red areas with irregular edges; exudation mainly corresponds to white or yellowish-white covered areas with corresponding texture; erosion mainly corresponds to the edges of superficial mucosal defects; and ulceration mainly corresponds to mucosal defect areas with relatively intact boundaries and color or morphological differences at the base compared to the surrounding mucosa. The above image representations are used to form evidence type labels and are not used as the sole basis for clinical diagnosis.

[0093] In an executable classifier implementation, local image region samples with one or more evidence type labels are obtained, and a feature vector is generated for each sample according to the method described above. For each type of evidence, a binary classification gradient boosting tree classifier is trained. Each classifier consists of a predetermined number of decision trees in sequence, with each subsequent decision tree fitting the classification residual of its predecessor. The number of decision trees, maximum depth, and learning rate are determined and fixed through cross-validation of training and validation samples. The input to the classifier is a feature vector. The output is the initial recognition score for the corresponding evidence type. , The value range is from 0 to 1.

[0094] For each type of evidence, select the evidence that enables that type of evidence to be identified on the verification sample. The value reaches the recognition threshold of the preset optimization target. When the corresponding initial recognition score is not lower than the recognition threshold, the local image region is determined as the corresponding evidence type. Since each evidence type is judged separately, the same local image region can correspond to multiple evidence types; at this time, candidate inflammatory evidence objects sharing the same region coordinates and mucosal location identifiers are generated respectively. In addition to the gradient boosting tree classifier mentioned above, other classifiers that can receive the same feature vector and output the initial recognition scores of each evidence type can also be used.

[0095] S33: Generate candidate inflammation evidence objects.

[0096] Specifically, local image regions, evidence types, and mucosal location identifiers are associated as candidate inflammatory evidence objects, and the corresponding video frame and timestamp, pixel mask and center position, and initial recognition score are recorded. Image quality information obtained by normalizing sharpness, exposure status, and occlusion level. .

[0097] S4: Generate authenticity verification results.

[0098] Based on the candidate inflammation evidence objects generated in S3, the non-rigid deformation field and mucosal location markers obtained in S2, and the fluid motion region and fluid coverage region obtained in S1, S4 is executed. Specifically, it includes the following sub-steps:

[0099] S41: Candidate inflammatory evidence objects corresponding to the intervention event before and after the intervention event.

[0100] Specifically, a non-rigid deformation field is used to map the local image regions of candidate inflammatory evidence objects after intervention to the coordinate system of the image sequence before intervention. Candidate inflammatory evidence objects before intervention and those after intervention with the same mucosal location identification and evidence type are identified as candidate corresponding objects.

[0101] For the The first candidate inflammatory evidence subjects before intervention and the first For each candidate inflammatory evidence object after intervention, a corresponding score is calculated based on the intersection-union ratio (IU / UGROUP) of the mapped local image regions, the distance between region centers, and the similarity of color, texture, morphology, and edge features. Each score component is first normalized to 0 to 1, and then summed according to a preset non-negative weight, which sums to 1. When the corresponding score of an object reaches a preset correspondence threshold, a candidate correspondence is established.

[0102] When multiple candidate inflammatory evidence objects of the same type exist under the same mucosal location identifier, the goal is to maximize the sum of the corresponding scores of the objects. This involves solving for a one-to-one correspondence between objects before and after intervention, ensuring that each candidate inflammatory evidence object corresponds to at most one independent candidate inflammatory evidence object. Objects that do not form a pre- and post-intervention correspondence retain their object trajectories in consecutive video frames for determining disappearance, reappearance, or positional movement states.

[0103] S42: Determine the response status of the candidate inflammatory evidence object.

[0104] The corresponding mucosal region meets the extraction criteria for candidate inflammatory evidence, meaning that the region is not obscured by fluid, saturated reflection, or severe defocusing, and the image quality information is satisfactory. The quality must be no lower than the preset quality threshold and be able to complete the feature extraction and classifier operation of S31 and S32; this condition indicates that there is a valid detection basis, but does not mean that the corresponding evidence necessarily exists.

[0105] In one specific implementation, the response status is determined as follows:

[0106] S42-1: When candidate inflammatory evidence objects of the same type and corresponding to each other exist before and after the intervention under the same mucosal location marker, and the condition for re-emergence is not met, it is determined to be a maintained state.

[0107] S42-2: If a candidate inflammatory evidence object exists only in the image sequence before intervention, and the corresponding mucosal region meets the extraction conditions for the candidate inflammatory evidence object after intervention, and no candidate inflammatory evidence object of the same type is extracted in the image sequence after intervention, and the determination conditions for the positional movement state are not met, it is determined to be in the disappearance state.

[0108] S42-3: Candidate inflammatory evidence objects are visible before the intervention event; the corresponding positions predicted based on the object trajectory and mucosal background motion are located in fluid-covered areas during the intervention event, or are invisible due to fluid occlusion, saturated reflection, or severe defocusing; when candidate inflammatory evidence objects of the same type and with similar local features are extracted again under the same mucosal location marker in the post-intervention image sequence, they are identified as re-emerging.

[0109] S42-4: For candidate inflammatory evidence objects that can form object trajectories during the intervention event, calculate the actual displacement based on the change in the object's center position in adjacent video frames. Input the object's location on the mucosal background into the corresponding non-rigid deformation field or background motion transformation to obtain the predicted displacement of the object fixed on the mucosal surface as a function of the mucosal background. ; Calculate the remaining displacement .

[0110] When the remaining displacement When the amplitude of the light reaches a preset movement threshold, the cosine similarity between its direction and the average optical flow direction of the corresponding fluid movement region reaches a preset direction consistency threshold, and the candidate inflammatory evidence object moves from the mucosal region corresponding to the original mucosal location marker to the mucosal region corresponding to another mucosal location marker, it is determined to be in a positional movement state. Here, "the mucosal location marker changes" means that the candidate inflammatory evidence object crosses different mucosal regions with the fluid, while the original mucosal location marker itself remains unchanged.

[0111] The system handles state crossover according to the judgment logic of reappearance, position movement, hold, and disappearance states: when the conditions for both reappearance and hold are met simultaneously, the state is prioritized as reappearance; when the conditions for both position movement and disappearance are met simultaneously, the state is prioritized as position movement. After applying the above priority rules, a response state and its supporting video frame are recorded for each object to be judged for authenticity verification.

[0112] S43: Generate authenticity verification results.

[0113] Specifically, the authenticity verification results of candidate inflammatory evidence objects corresponding to the maintained or reappearing states are determined as fixed mucosal sources; the positional movement states and bleeding or exudation types that disappeared after intervention and did not reappear in subsequent frames that meet the extraction conditions are determined as fluid interference; the authenticity verification results of other candidate inflammatory evidence objects that do not meet the fixed mucosal source or fluid interference determination conditions are determined as pending verification. The response state, the corresponding score of the object, the judgment video frame, and the remaining displacement are associated and saved.

[0114] Therefore, S4 outputs the response status and authenticity verification results of the candidate inflammatory evidence objects, which are then used by S5 to determine the object retention status, evidence reliability, and graph structure.

[0115] S5: Construct a co-location map of inflammatory evidence.

[0116] Based on the authenticity verification result generated in S4, S5 is executed, which specifically includes the following sub-steps:

[0117] S51: Identify candidate inflammatory evidence subjects to be retained.

[0118] Specifically, the authenticity verification results are retained for objects with fixed mucosal origin or candidate inflammatory evidence to be verified, while objects with fluid interference are excluded.

[0119] S52: Determine the reliability of the evidence.

[0120] Specifically, the The candidate inflammatory evidence objects to be retained are determined according to their evidence type. Authenticity verification results Initial recognition score and image quality information Determining the reliability of evidence The verification samples can be grouped according to the type of evidence and the results of the authenticity verification, and a logical calibration relationship can be fitted with the fixed mucosal origin label as the target:

[0121]

[0122] in, , , and The calibration parameters corresponding to this type of evidence and the verification results are determined through maximum likelihood estimation or equivalent optimization methods using the verification samples. Different calibration parameters are fitted to the mucosal source and the samples to be verified, respectively; after model training, each calibration parameter is fixed. The larger the value, the more credible the image evidence of the fixed mucosa source.

[0123] S53: Form graph nodes and connecting edges and construct a co-bitmap of inflammatory evidence.

[0124] Specifically, each candidate inflammatory evidence object determined and retained in S51 is used as a graph node, and the evidence type, mucosal location identifier, and evidence reliability of the corresponding candidate inflammatory evidence object are set as graph node attributes; the video frame to which it belongs, timestamp, local image region, response status, and authenticity verification result can also be used as auxiliary graph node attributes. Each candidate inflammatory evidence object corresponds to one graph node, and no connection edges are established between the graph node and itself.

[0125] A non-rigid deformation field is used to map the local image regions corresponding to each graph node to a unified coordinate system. When the mucosal position identifiers of two different graph nodes correspond to the same mucosal region, or when the intersection-union ratio of the mapped local image regions reaches a preset spatial overlap threshold, a spatially co-located connection edge is established between the two graph nodes. The spatially co-located connection edge is used to characterize that two evidence objects are located in the same or overlapping mucosal regions, and in this embodiment, it is stored as an undirected connection edge.

[0126] When two graph nodes come from different video frames and have the same mucosal location identifier and evidence type, a time-duration connection edge is established between the two graph nodes. For cases where there are multiple graph nodes with the same mucosal location identifier, the same evidence type, and within the same video frame, the object correspondence score of S41 is used to establish a one-to-one correspondence across frames, and time-duration connection edges are established only between corresponding graph nodes. The time-duration connection edges can be stored in chronological order to represent the temporal continuity of the evidence objects.

[0127] For cases where the same pair of graph nodes repeatedly satisfies the connection condition, only one connection edge of the corresponding type is saved. An inflammatory evidence co-location graph is constructed based on the graph nodes, spatially co-located connection edges, and temporally persistent connection edges. Candidate inflammatory evidence objects to be verified are retained as graph nodes for updating in subsequent embodiments; candidate inflammatory evidence objects with a fixed mucosal origin, as determined by the authenticity verification results, form the fixed mucosal origin connection subgraph primarily used in S6.

[0128] Thus, S5 outputs a common bitmap of inflammatory evidence, whose nodes and connecting edges can all be traced back to the corresponding candidate inflammatory evidence objects and video frames, which S6 uses to form evidence combinations and perform image classification.

[0129] S6: Perform image classification to obtain the inflammation grading results of endoscopic images.

[0130] S6 is executed based on the inflammatory evidence co-bitmap constructed using S5. Image classification is performed on the mucosal regions corresponding to mucosal location markers, outputting ordered image categories representing the degree of inflammation in the corresponding mucosal region images. Specifically, it includes the following sub-steps:

[0131] S61: Form a preset level of evidence pattern.

[0132] Before processing the target colonoscopy video, samples with image inflammation level annotations are acquired. Image inflammation level annotations are ordered image categories formed by labeling the endoscopic image features. and satisfy The higher the category number, the higher the degree of inflammation presented in the sample image.

[0133] Following the processing methods in S1-S5, a co-location map of inflammatory evidence for each sample is generated, and graph nodes and their connecting edges formed by candidate inflammatory evidence objects originating from fixed mucosa are extracted from it. The samples are grouped according to ordered image categories, and within each category group, the frequency of evidence type occurrence, evidence reliability distribution, frequency of spatial co-location connecting edges, and frequency of temporal persistence connecting edges are statistically analyzed.

[0134] In one specific implementation, evidence types whose frequency reaches a preset pattern support threshold are identified as pattern node types for the ordered image category; the minimum reliability requirement or reliability range of the pattern nodes is determined based on the corresponding evidence reliability distribution; spatial colocation relationships and temporal persistence relationships whose frequency in samples with corresponding pattern nodes reaches a preset connection support threshold are identified as spatial colocation pattern edges and temporal persistence pattern edges, respectively. This forms a preset level of evidence pattern corresponding to the ordered image category.

[0135] Each preset level of evidence pattern includes at least an ordered image category, a required evidence type, a corresponding evidence reliability requirement, a required spatial co-location connection edge, and a required temporal persistence connection edge. The support threshold, reliability requirement, and connection support threshold for each pattern are determined and fixed on independent validation samples; the resulting preset level of evidence patterns are pre-stored for use in the image classification process of the target colonoscopy video.

[0136] S62: Generate a connection subgraph and evidence combination.

[0137] Specifically, for each mucosal location identifier, fixed mucosal origin nodes and their spatial co-location edges and temporal continuity edges are selected from the inflammation evidence co-location map to form a fixed mucosal origin connection sub-map and evidence combination; the nodes to be verified are still retained in the co-location map, but do not meet the necessary node or necessary edge conditions of the preset level evidence pattern.

[0138] S63: Matching evidence combinations and preset level evidence patterns.

[0139] Specifically, firstly, candidate correspondences are established between graph nodes in the fixed mucosal origin connection subgraph and pattern nodes in each preset level of evidence pattern according to the evidence type; then, it is determined whether the corresponding graph nodes maintain the spatial co-location connection edges and temporal continuity connection edges required by the preset level of evidence pattern; after satisfying the corresponding connection structure, the evidence reliability of the graph nodes is compared with the evidence reliability requirements of the pattern nodes.

[0140] In one specific implementation, the node type matching degree is calculated separately. Evidence reliability and matching degree Spatial co-location connection edge matching degree Matching degree of time-dependent connection edges . This represents the proportion of the number of necessary pattern nodes for which corresponding graph nodes have been found to the total number of necessary pattern nodes. The ratio of the number of corresponding graph nodes to the total number of corresponding graph nodes to meet reliability requirements; and These represent the proportions of necessary spatial colocation mode edges and necessary temporal persistence mode edges preserved in the fixed mucosal origin connection subgraph, respectively. The values ​​for each matching degree range from 0 to 1.

[0141] The degree of overall matching between the evidence combination and the l-th preset level of evidence pattern Determined according to the following formula:

[0142] ;

[0143] in, , , and To pre-determine non-negative weights, and The weights on the validation samples labeled with image inflammation levels are determined and fixed based on the predetermined optimization target of achieving the consistency rate of ordered image categories. By setting the spatial colocation edge matching degree and the temporal persistence edge matching degree, even if multiple evidence types have high evidence reliability, they cannot obtain the same complete pattern matching degree when the corresponding spatial colocation relationship or temporal persistence relationship is not satisfied.

[0144] S64: Determine the inflammation grading results of endoscopic images.

[0145] Specifically, the degree of comprehensive matching between the evidence combination and each preset level of evidence pattern is compared, and the ordered image category corresponding to the preset level of evidence pattern with the highest degree of comprehensive matching is determined as the endoscopic image inflammation grading result of the mucosal region corresponding to the mucosal location identifier.

[0146] When two or more preset-level evidence patterns have the same overall matching degree, the preset-level evidence pattern with a higher proportion of necessary connection edges is selected first; if they are still the same, the preset-level evidence pattern with fewer nodes of the necessary unmatched pattern is selected; if they still cannot be distinguished, the pattern priority is selected according to the pattern priority determined and stored in advance by the verification samples. If the highest overall matching degree is lower than the preset acceptance threshold, the ordered image category with the highest overall matching degree is still output, and a low-match confidence mark is attached for subsequent verification, but the ordered image category is not changed as the output of the endoscopic image inflammation grading result.

[0147] The output data includes at least mucosal location identifiers, ordered image categories, and corresponding comprehensive matching degrees. It may also include graph nodes that form the highest matching result, spatial colocation edges, and temporally persistent edges, so that the grading results can be traced back to the corresponding image evidence in the target colonoscopy video.

[0148] As an example of data flow, candidate inflammatory evidence objects of the mucosal erythema type, which exist both before and after intervention at the same mucosal location marker and do not belong to a re-emergent state, are identified as maintaining states and fixed mucosal sources; candidate inflammatory evidence objects of the exudative type, which cross to another mucosal location marker along the direction of fluid movement and whose remaining displacement meets the conditions, are identified as position-moving states and fluid interference and are excluded; the remaining fixed mucosal source objects form graph nodes and connecting edges and participate in pattern matching. The response state is thus used to determine whether and with what level of evidence reliability an object should enter subsequent image classification.

[0149] In summary, this embodiment eliminates fluid interference by identifying the correspondence between objects at the same mucosal site before and after the intervention event, their response status, and verifying authenticity. It constructs a co-location map of inflammatory evidence and uses the matching of a fixed mucosal origin connection sub-map with a preset level of evidence pattern to output ordered image categories. This reduces the impact of the intervention event on the endoscopic image inflammatory grading results while preserving the evidence tracing relationship.

[0150] Example 2, refer to Figures 5 to 6 Based on the target colonoscopy video, corresponding mucosal region, non-rigid deformation field, background motion transformation, mucosal location markers, candidate inflammatory evidence objects, authenticity verification results, evidence reliability, and inflammatory evidence co-location map obtained in Example 1, this embodiment further verifies the candidate inflammatory evidence objects related to the bleeding type associated with lens contact.

[0151] In this embodiment, the preset thresholds or parameters used for lens contact event detection, trajectory mapping, two-way region matching, and object reproduction judgment are determined and fixed according to the method of determining training samples and verification samples in Embodiment 1; wherein, the lens contact event related threshold is determined based on the event detection F1 value, and the two-way region matching and object reproduction related thresholds are determined based on the region matching accuracy and the object correspondence accuracy, respectively.

[0152] Specifically, the intelligent grading method for endoscopic images of ulcerative colitis also includes:

[0153] S7: Determine the lens contact event, lens contact trajectory, and lens release event, specifically including the following sub-steps:

[0154] S71: Extract contact characterization data.

[0155] Specifically, at least two contact characterization data points are extracted from the target colonoscopy video, including changes in local image magnification, radial optical flow, mucosal color changes, changes in reflective areas, and defocusing degree. Each contact characterization data point is stored in video frame increments, associated with the change value, change region, region center, and validity marker. Change values ​​of different dimensions are normalized to 0 to 1 according to the range in the training samples. Specifically, local magnification is obtained from the scale component of local similarity transformation between adjacent frames; radial optical flow is obtained by voting on the remaining optical flow after subtracting background optical flow; color changes are obtained from standardized color statistical differences; reflective changes are obtained from changes in the area, position, or shape of the bright saturated region; and defocusing degree is obtained from gradient energy, Laplacian response, or high-frequency component attenuation.

[0156] S72: Determine synchronous changes and common spatial center.

[0157] Specifically, for each valid contact characterization data point, when its normalized change value reaches the corresponding preset change threshold, the data point is marked as active in the corresponding video frame; otherwise, it is marked as inactive. The active state sequence is then subjected to temporal smoothing with a length no greater than a preset smoothing window to eliminate instantaneous fluctuations in individual video frames.

[0158] When at least two different types of contact characterization data remain active within overlapping consecutive video frame segments, and the interval between their respective starting video frames is no greater than a preset synchronization tolerance, it is determined that at least two contact characterization data are changing synchronously within the same consecutive video frame segment. The length of the consecutive video frame segment is no less than a preset number of contact confirmation frames.

[0159] Using the background motion transformation or non-rigid deformation field of Example 1, the change regions corresponding to at least two synchronously changing contact characterization data are mapped to the same reference video frame coordinate system. For the first... Variation region of contact characterization data The center of the region is calculated by weighting the normalized variation values ​​of the pixels within the region. .

[0160] When the intersection-union ratio of at least two changing regions reaches a preset center region overlap threshold, or the ratio of the distance between the centers of the corresponding regions to the equivalent diameter of the regions is not greater than a preset center distance threshold, these changing regions are determined to have a common spatial center. According to each regional center It is obtained by weighted average of the corresponding normalized change values.

[0161] If there are multiple candidate common spatial centers in the same video frame, the number of support items, the degree of overlap of the change area, and the continuity score relative to the common spatial center of the previous video frame are calculated for each candidate common spatial center. The candidate common spatial center with the highest comprehensive score and reaching the preset center confirmation threshold is retained first. If it cannot be uniquely determined, it is retained as different candidate trajectories for S73 to further associate according to the temporal continuity.

[0162] S73: Determine the lens contact event and lens contact trajectory.

[0163] Specifically, when at least two contact characterization data change synchronously within the same continuous video frame segment, and the corresponding change areas have a common spatial center, the first video frame that continuously satisfies the above conditions is determined as the starting video frame of the lens contact event, and the lens contact event is determined.

[0164] The common spatial center of each video frame is associated in chronological order. For adjacent video frames, the position of the common spatial center of the previous video frame in the next video frame is first predicted using background motion transformation. Then, within a preset search range of the predicted position, the common spatial center with the smallest center distance and overlapping supported item types is selected to establish a correspondence. Center gaps shorter than a preset allowable interval can be filled in by interpolation based on the positions of the corresponding centers before and after, but the interpolated position is not used as contact characterization data alone.

[0165] Lens contact trajectories are formed based on the positional changes of a common spatial center in consecutive video frames. Each lens contact trajectory records at least the lens contact event identifier, the corresponding video frame and timestamp for each trajectory point, the image position of the common spatial center, the joint coverage area of ​​the corresponding changing region, and the data type supporting the contact representation for that trajectory point.

[0166] S74: Determine the video frame corresponding to the lens release event.

[0167] Specifically, the contact characterization data used to determine the lens contact event is detected from the initial video frame after the lens contact event. When at least two contact characterization data cannot be synchronously activated in a series of video frames that are no less than a preset number of release confirmation frames, it is determined that at least two contact characterization data are no longer changing synchronously, and the first valid video frame in the consecutive video frame segment is determined as the video frame corresponding to the lens release event.

[0168] The video time interval corresponding to the lens contact event starts from the initial video frame of the lens contact event and ends at the video frame preceding the video frame corresponding to the lens release event; the video frame corresponding to the lens release event serves as the first time boundary video frame after the end of the lens contact event. If multiple contact characterization data become invalid due to large-area occlusion in consecutive video frames, the invalid state is not directly interpreted as no longer changing synchronously, but is further evaluated in subsequent valid video frames.

[0169] Therefore, S7 outputs the lens contact event, the corresponding video time interval, the lens contact trajectory, and the video frame corresponding to the lens release event, so that S8 can determine the temporal and spatial relationship between the candidate inflammatory evidence object of the bleeding type and the lens contact event.

[0170] Furthermore, this intelligent grading method for endoscopic images of ulcerative colitis also includes:

[0171] S8: Set contact association markers and retain corresponding candidate inflammatory evidence objects.

[0172] S81: Maps the lens contact trajectory to the mucosal location marker.

[0173] Specifically, the lens contact trajectory is initially located in the image coordinate system of the video frame to which each trajectory point belongs. For the annular mucosal region with recognizable texture around the common spatial center, the features of blood vessel bifurcation, fold edges, or local mucosal texture are extracted and matched with the corresponding mucosal region with mucosal location markers in Example 1. The mapping relationship from the video frame to the corresponding mucosal region is determined based on the effective matching points.

[0174] When sufficient matching points cannot be directly obtained near the common space center due to contact occlusion, the mapping relationship in the video frames that can be successfully matched can be propagated to the trajectory point by utilizing the background motion transformation of the video frames before and after the trajectory point; the identifiable mucosal texture in the video frame corresponding to the lens release event can also be used to correct the propagation result.

[0175] Each trajectory point and its combined coverage area are mapped to the corresponding mucosal region. When the mapped combined coverage area overlaps with a corresponding mucosal region, the mucosal position identifier of that corresponding mucosal region is determined as the mucosal position identifier corresponding to the lens contact trajectory; when multiple corresponding mucosal regions are covered, the corresponding multiple mucosal position identifiers and their respective region overlap ratios are recorded. Mucosal position identifiers with an overlap ratio reaching a preset trajectory mapping threshold are used for spatial overlap determination in S82.

[0176] S82: Determine the temporal sequence and spatial overlap.

[0177] Candidate inflammatory evidence objects with hemorrhage as the evidence type are selected from the candidate inflammatory evidence objects generated in Example 1, and hemorrhage object trajectories are formed according to mucosal location identifiers, local image region overlap relationships, and object correspondences in consecutive video frames. For each shot contact event, the hemorrhage object trajectory is retrieved before its starting video frame and starting from the video frame corresponding to the shot release event.

[0178] Within a preset pre-event time window preceding the lens contact event, select a visible mucous membrane area with corresponding mucous membrane location markers. The visible mucous membrane area should not be obscured by fluids, saturated reflections, or severe defocusing, and image quality information should be considered. The quality is not lower than the preset quality threshold of Example 1, and it is able to perform local image region feature extraction and bleeding type classifier operation, that is, it meets the extraction conditions of bleeding type candidate inflammation evidence objects.

[0179] If at least one video frame within the current time window meets the extraction criteria, and no candidate inflammatory evidence object of the bleeding type is extracted under the corresponding mucosal location marker in any of these video frames, but a candidate inflammatory evidence object of the bleeding type with the corresponding mucosal location marker is extracted for the first time starting from the video frame corresponding to the lens release event, then the candidate inflammatory evidence object is determined to meet the time condition of "not extracted before the lens contact event and appearing for the first time after the lens contact event". This condition only indicates that the corresponding object was not extracted in video frames with a valid detection basis, and does not mean that bleeding did not objectively exist before the contact.

[0180] The mucosal location markers of the candidate inflammatory evidence object are compared with those corresponding to the lens contact trajectory. When they are identical, the condition of mucosal location marker overlap is determined. Furthermore, the local image region of the candidate inflammatory evidence object is mapped to the unified coordinate system of the lens contact trajectory; when the intersection-union ratio (IUU) of the joint coverage area of ​​this local image region and the trajectory reaches a preset contact area overlap threshold, the region overlap is used as an additional verification criterion for spatial overlap.

[0181] S83: Set a contact association flag and determine the authenticity verification result as pending verification.

[0182] For bleeding-type candidate inflammation evidence objects that simultaneously meet the following conditions, a contact association marker is set: no bleeding-type candidate inflammation evidence objects with the same mucosal location identifier are extracted in the visible mucosal area before the lens contact event; the object appears for the first time after the lens contact event; the mucosal location identifier of the object coincides with the mucosal location identifier corresponding to the lens contact trajectory.

[0183] Contact association markers should at least be associated with and store the corresponding candidate inflammatory evidence object identifier, lens contact event identifier, lens contact trajectory, video frame corresponding to the lens release event, overlapping mucosal location identifier, and region overlap ratio. The authenticity verification results of candidate inflammatory evidence objects with contact association markers are determined as pending verification, indicating that existing video evidence cannot yet distinguish between fixed mucosal bleeding and mechanical bleeding caused by lens contact.

[0184] S84: Update evidence reliability and graph nodes.

[0185] Candidate inflammatory evidence objects with contact association tags and whose authenticity verification results are "to be verified" are identified as candidate inflammatory evidence objects to be retained. Following the evidence reliability determination method in Example 1, the evidence type is bleeding, the authenticity verification result is "to be verified," and the initial identification score is... and image quality information As input, the reliability of the evidence is re-determined using the reliability mapping relationship corresponding to the one to be verified.

[0186] If the candidate inflammatory evidence object already corresponds to a graph node in the inflammatory evidence co-location graph, its authenticity verification result, evidence reliability, and contact association marker attributes are updated. If no graph node has been formed, the candidate inflammatory evidence object is used as a graph node, and spatial co-location connections and temporal continuity connections are established according to the conditions of Example 1. Graph nodes whose authenticity verification result is to be verified are retained in the inflammatory evidence co-location graph for updating in S10, but are not considered necessary graph nodes for fixing the mucosal origin connection subgraph in Example 1.

[0187] Therefore, S8 outputs candidate inflammatory evidence objects of the bleeding type to be verified, which are marked with contact association and their mucosal location identifiers, for S9 to form a two-way composite image and for S10 to perform authenticity verification.

[0188] Furthermore, this intelligent grading method for endoscopic images of ulcerative colitis also includes: forming two-way composite images in a lens-free state.

[0189] This step, based on the target colonoscopy video, the lens contact event determined in S7, and the contact association markers and mucosal location identifiers obtained in S8, executes S9: forming a two-way composite image without lens contact, specifically including the following steps:

[0190] S91: Divide the on-camera entry stage and the on-camera exit stage.

[0191] Specifically, the sequence of camera movement and the sequence of intestinal segments are determined based on the background motion direction, local mucosal scale changes, and the order in which intestinal segments appear in consecutive video frames of the target colonoscopy video. The camera motion direction is determined by the main motion direction and local scale increase / decrease relationship of mucosal feature points in the background motion transformation of Example 1; the order intestinal segments can be obtained by training an intestinal segment classifier based on image annotation samples of the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, and rectum, or it can be obtained based on pre-identified anatomical landmarks and their temporal order.

[0192] Within a preset time window, the camera movement direction and the order of intestinal segment appearances are assessed using either majority voting or median smoothing. When the camera movement direction continuously reverses, and the order of intestinal segment appearances continuously reverses relative to the preceding video frames, and both durations meet the preset stage transition conditions, the stage transition position is determined. Pauses or partial reverse movements shorter than the preset duration do not change the stage.

[0193] The main video frame segments extending continuously in the direction of the first camera movement and the order in which the first intestinal segment appears before the stage transition position are defined as the entry stage. The main video frame segments extending continuously in the opposite direction of camera movement and the order in which the reverse intestinal segment appears after the stage transition position are defined as the exit stage. When there are multiple local round-trip video frame segments, the video frame segment covering a larger intestinal segment area and meeting the image quality requirements is preferentially retained as the corresponding stage. If two stages cannot be formed, the two-way composite image recorded in S94 is incomplete.

[0194] S92: Extract region matching features.

[0195] At least two region-matching features were extracted from the mucosal regions during the insertion and withdrawal stages, including intestinal segment order, fold topology, vascular bifurcation, and local mucosal texture. Intestinal segment order features represent the intestinal segment to which the mucosal region belongs and its relative order of appearance at the corresponding stage; fold topology features represent the bifurcation, connection, adjacency, and relative direction between fold edges; vascular bifurcation features include the location of vascular bifurcation points, branch angles, and local descriptive information; and local mucosal texture features include local gradient distribution, texture direction, and scale-invariant texture descriptive information.

[0196] For each region matching feature, the similarity between the mucosal region during the insertion and withdrawal stages is calculated and normalized to 0 to 1. A region matching feature is only counted in the region matching if it can be effectively extracted from the corresponding mucosal regions in both stages; each pair of candidate mucosal regions has at least two valid region matching features.

[0197] S93: Match mucosal regions representing the same mucosal site in two phases.

[0198] Specifically, firstly, candidate mucosal regions with the same intestinal segment or corresponding order of appearance are selected from the intestinal segment insertion and withdrawal stages according to the intestinal segment sequence. Then, a comprehensive region matching score is calculated based on at least two effective region matching features. : ;in, Indicates the first Similarity of effective region matching features, This represents the corresponding preset non-negative weights, and the summation only applies to region matching features that are valid in both stages. The weights are determined and fixed based on validation samples labeled with the same mucosal site.

[0199] Further geometric consistency checks are performed on matching points in vascular bifurcation, fold topology, or local mucosal texture, and robust estimation is used to eliminate erroneous matches. When at least two valid region matching features are present, and the region comprehensive matching score is [not specified], [further details are needed]. When the preset region matching threshold is reached and the geometric consistency meets the preset conditions, the two mucosal regions are determined to be mutually matched and represent the same mucosal site.

[0200] When a mucosal region meets the conditions with multiple candidate mucosal regions, the one-to-one correspondence between the mucosal regions in the advancement and withdrawal stages is determined with the goal of maximizing the sum of the comprehensive regional matching scores. Matching mucosal regions are assigned the same mucosal location identifier. If one of the mucosal regions already has a mucosal location identifier assigned in Example 1, that identifier is used without conflicting with other high-confidence correspondences. If two mucosal regions have different existing mucosal location identifiers, they are merged into the same mucosal location identifier and the original identifier correspondence is recorded only when both the comprehensive regional matching score and geometric consistency reach a preset identifier merging threshold; otherwise, they are marked as regional matching conflicts and are not used for determining non-repetition.

[0201] S94: Select video frames from two stages and form a two-way composite image.

[0202] For candidate inflammatory evidence objects with contact-associated markers for bleeding types, one or more video frames are selected from the mucosal regions of the advancement and withdrawal phases that match each other in S93, based on their mucosal location identifiers.

[0203] The selected video frames should be in a non-lens-contact state, meaning they are not located within the video time interval corresponding to any lens contact event, and there should be no lens contact event in the corresponding video frame determined by the synchronous changes of at least two contact characterization data and a common spatial center; the selected mucosal region should also not be obscured by fluids or other content, and should not experience saturated reflection or severe defocusing, and the image quality information should be satisfactory. It is not lower than the preset quality threshold of Example 1, and can complete the extraction of local image region features and the operation of the bleeding type classifier, thereby meeting the extraction conditions of bleeding type candidate inflammation evidence objects.

[0204] When multiple candidate video frames exist at each stage, image quality information is selected first. Video frames with higher resolution, smaller mapping errors to local areas of the contact-related object, and temporal dispersion are selected. Two-way composite images are constructed from video frames selected from the approach and exit phases that have the same mucosal location identifier and meet the above conditions, along with their corresponding mucosal regions. The regional mapping relationship between the two phases is recorded.

[0205] Only when at least one video frame meeting the extraction criteria is obtained in both the approach and exit phases is the corresponding two-way composite image marked as complete and used to make a negative judgment of "not repeated". If a video frame meeting the extraction criteria cannot be obtained from either phase, the corresponding two-way composite image is marked as incomplete and kept for verification in S104.

[0206] Therefore, S9 outputs complete or incomplete two-way composite images corresponding to the contact association markers, identifiers of the same mucosal locations, and inter-stage region mapping relationships, which are then used by S10 to verify whether the candidate inflammatory evidence objects for the bleeding type appear repeatedly.

[0207] S10: Perform two-way diplopia verification and reclassify the image.

[0208] S101: Extract bleeding type candidate inflammation evidence objects from two-way composite images.

[0209] For the local image regions corresponding to candidate inflammatory evidence objects of the bleeding type with contact association markers, the first appearance of the candidate inflammatory evidence object of the bleeding type with contact association markers after the lens contact event is identified as the first-appearance object. Using the region mapping relationship obtained in S93 and S94, the local image region corresponding to the first-appearance object is mapped to each two-way composite image with the same mucosal location identifier.

[0210] Using the local image region extraction, evidence type classification, and candidate inflammation evidence object generation methods described in S31-S33 of Example 1, candidate inflammation evidence objects of the bleeding type are extracted from the corresponding mucosal regions of each two-way composite image. For each object to be compared in other two-way composite images, the intersection-union ratio O, color similarity C, and morphological similarity F of the mapped local image region and the local image region of the object appearing for the first time are calculated.

[0211] Color similarity C can be calculated based on the difference in the mean color channels of two local image regions after color standardization, or the distance of their color histograms. Morphological similarity F can be calculated based on differences in area ratio, aspect ratio, roundness, contour moments, or boundary description information. The values ​​of O, C, and F all range from 0 to 1, with larger values ​​indicating greater similarity. The corresponding thresholds are determined and fixed based on validation samples labeled with the same bleeding object. When a candidate inflammatory evidence object is extracted from other two-way composite images outside the video frame to which the object first appeared, and the evidence type is bleeding, the mucosal location is identical, the cross-union ratio O of the mapped local image regions reaches a preset recurrence overlap threshold, and the color similarity C or morphological similarity F reaches the corresponding preset recurrence threshold, the object that first appeared is determined to have reappeared.

[0212] S102: When it occurs repeatedly, it is identified as a fixed mucosal source.

[0213] When a bleeding-type candidate inflammatory evidence object with a contact association marker appears repeatedly in other two-way paroxysmal images, its authenticity verification result is changed from pending verification to fixed mucosal origin, and it continues to be identified as a retained candidate inflammatory evidence object. The contact association marker can be retained as a verification traceability attribute, but no longer indicates that the authenticity result is still pending.

[0214] Following the evidence reliability determination method in Example 1, the evidence type was bleeding, the authenticity verification result was fixed mucosal origin, and the initial identification score was used. and image quality information As input, the reliability of the evidence is redefined using the reliability mapping relationship corresponding to the fixed mucosal source, and the corresponding graph node attributes are updated.

[0215] S103: When both stages meet the extraction conditions and do not repeat, it is determined to be a mechanical interference related to lens contact.

[0216] When the corresponding mucosal regions in the ingress and regression stages each have at least one video frame that meets the extraction conditions for candidate inflammatory evidence of bleeding type, i.e., S9 forms a complete two-way composite image, and no candidate inflammatory evidence of bleeding type that meets the reproduction conditions of S101 is extracted in the two-way composite images other than the video frame to which the object first appears, it is determined that the object first appearing has not appeared repeatedly.

[0217] Only when both of the above two stages have a valid detection basis and the issue does not recur, is the verification result of the candidate inflammatory evidence object changed from pending verification to lens contact-related mechanical interference, and it is determined to be a candidate inflammatory evidence object that is not retained. This avoids misinterpreting non-detection as mechanical interference due to image occlusion, saturation reflection, or severe defocusing at a certain stage.

[0218] S104: Keep the image pending verification when the two-way composite image is incomplete.

[0219] If at least one video frame that meets the extraction conditions for a candidate inflammatory evidence object of bleeding type cannot be obtained from both the entry and exit phases, no judgment of non-repetition will be made. The authenticity verification result of the corresponding candidate inflammatory evidence object will remain as pending verification and will continue to be retained as a candidate inflammatory evidence object.

[0220] When region matching encounters unavoidable conflicts, or when a local image region of an object cannot be reliably mapped to other two-way composite images for the first time, it is not determined to be a mechanical interference related to lens contact. Instead, it remains pending verification, and a flag indicating insufficient verification data and the corresponding reason are saved. The reliability of the evidence continues to be determined using the reliability mapping relationship corresponding to the pending verification in Example 1.

[0221] S105: Update the co-bitmap of inflammatory evidence based on the revised authenticity verification results.

[0222] For candidate inflammatory evidence objects whose authenticity verification results are modified to be of fixed mucosal origin, the corresponding graph nodes are retained, the authenticity verification results and evidence reliability attributes are updated, and their spatial colocation edges and temporal persistence edges are re-determined according to the establishment conditions of Example 1, so that they can enter the fixed mucosal origin connection subgraph.

[0223] For candidate inflammatory evidence objects whose authenticity verification results are modified to reflect mechanical interference related to lens contact, their graph nodes, spatial co-location edges, and temporal persistence edges connected to these nodes are excluded from the inflammatory evidence co-location graph. For candidate inflammatory evidence objects that remain to be verified, the corresponding graph nodes and contact association markers are retained, but they are not used as necessary graph nodes or necessary edges for fixing the mucosal origin connection subgraph.

[0224] An updated co-location graph of inflammatory evidence is obtained based on the updated graph nodes, spatial co-location edges, and temporal continuity edges; then, according to S62 of Example 1, fixed mucosal origin graph nodes and their connecting edges are selected from the updated co-location graph of inflammatory evidence to form an updated fixed mucosal origin connection subgraph and evidence combination.

[0225] S106: Reclassify the images.

[0226] The updated evidence combination is matched with the preset level evidence patterns formed and fixed in Example 1. The comprehensive matching degree is calculated using the node type matching degree, evidence reliability matching degree, spatial co-location connection edge matching degree, and temporal continuity connection edge matching degree of Example 1. The ordered image category corresponding to the preset level evidence pattern with the highest comprehensive matching degree is determined as the corrected endoscopic image inflammation grading result.

[0227] The reclassified output includes at least the mucosal location identifier, the corrected ordered image category, the corresponding comprehensive matching degree, and the two-way diplopia verification result; it can also record the video frames of the entry and exit stages used to modify the authenticity verification result, the reproduction matching quantities O, C and F, and the graph node status before and after the update, so as to preserve the result traceability relationship.

[0228] In the first data flow scenario, the first bleeding-type candidate inflammation evidence object appearing along the lens contact trajectory after the lens contact event is marked with a contact association and identified as to be verified. When the same mucosal location identifier, overlapping mapped regions, and similar color or shape of the bleeding-type candidate inflammation evidence object are extracted again in other two-way composite images, they are modified to be fixed mucosal sources, the reliability of the evidence is re-determined, and the corresponding graph node is entered into the fixed mucosal source connection subgraph.

[0229] As a second data stream scenario, video frames that meet the extraction conditions are obtained during the camera entry and exit phases. However, if no candidate inflammatory evidence objects that meet the reproduction conditions for bleeding type are extracted in other two-way composite images, they are modified to mechanical interference related to camera contact and the corresponding graph nodes and connecting edges are excluded.

[0230] As a third data stream scenario, if a video frame that meets the extraction conditions is not obtained during the entry or exit phase, its authenticity verification result is kept as pending verification; the corresponding graph nodes are retained but not used as necessary graph nodes for the fixed mucosa source connection subgraph, so as to avoid interpreting insufficient verification data as mechanical interference.

[0231] In summary, this embodiment determines the lens contact event and lens contact trajectory by synchronous changes and common spatial center of at least two contact characterization data, maps the trajectory to mucosal location markers, and performs double-pass diplopia verification on the first contact-related bleeding that occurs after the lens contact event ends. Based on three results—repeated occurrence, effective in both stages but not repeated occurrence, or incomplete double-pass images—the source of fixed mucosa, mechanical interference related to lens contact, or to be verified are determined, respectively. Then, the co-location map of inflammatory evidence is updated and the images are reclassified.

[0232] Example 3: This example provides an intelligent grading system for endoscopic images of ulcerative colitis, including a processor and a memory storing a computer program. When the computer program is executed by the processor, it implements the steps of the intelligent grading method for endoscopic images of ulcerative colitis in Examples 1 and 2.

[0233] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent grading method of endoscopic images of ulcerative colitis, characterized by, This includes the following steps performed by the computing device: Acquire pre-collected target colonoscopy videos, detect intervention events including flushing or suction events, and determine the pre-intervention image sequence and post-intervention image sequence corresponding to the intervention events; Match the pre-intervention image sequence and the post-intervention image sequence to determine the corresponding mucosal regions that characterize the same mucosal site, and assign the same mucosal location identifier to the corresponding mucosal regions that match each other; Local image regions are extracted from the corresponding mucosal regions, and the evidence type is determined based on the color, texture, shape, or edge features of the local image regions. Candidate inflammatory evidence objects with the evidence type and mucosal location identifier are generated. Based on the mucosal location identifier, image matching results, and the image position and local features of the candidate inflammatory evidence objects in consecutive video frames, the candidate inflammatory evidence objects before and after the intervention event are determined to be in a state of retention, disappearance, reappearance, or position movement, and an authenticity verification result is generated. Based on the authenticity verification results, the candidate inflammatory evidence objects to be retained and their evidence reliability are determined. The retained candidate inflammatory evidence objects are used as graph nodes, and their evidence type, mucosal location identifier and evidence reliability are set as graph node attributes. Spatial colocation connection edges and temporal continuity connection edges are established between the graph nodes to construct an inflammatory evidence colocation graph. Based on the evidence combination formed by the attributes of the graph nodes, spatial co-location connections, and temporal persistence connections, the corresponding mucosal regions are classified to obtain ordered image categories that characterize the degree of inflammation in the images, which serve as the endoscopic image inflammation grading results.

2. The method of claim 1, wherein the method is for intelligent grading of endoscopic images of ulcerative colitis. The detection of the intervention event includes: Calculate the optical flow field of adjacent video frames in the target colonoscopy video, and extract the fluid motion region and fluid coverage region; When the optical flow vector in the fluid motion region moves from the diffusion center region to the periphery and the area of ​​the fluid coverage region increases with time, it is determined as a flushing event; When the optical flow vector moves from the periphery to the convergence center region and the area of ​​the fluid-covered region decreases over time, it is determined to be an attraction event; Video frames containing visible mucosal regions are selected from before and after the intervention event to form the pre-intervention image sequence and the post-intervention image sequence.

3. The method of claim 2, wherein the method is for intelligent grading of endoscopic images of ulcerative colitis. Determining the corresponding mucosal region includes: Brightness correction and color standardization were performed on the unobstructed mucosal background area before and after the intervention. At least two feature points were extracted from blood vessel bifurcation, fold edges and local mucosal texture and matched. The non-rigid deformation field is calculated based on the matched feature points. The mucosal region in the post-intervention image sequence is mapped to the pre-intervention image sequence, and the same mucosal location identifier is assigned to the overlapping mucosal regions after mapping, thus obtaining the corresponding mucosal region.

4. The method of claim 3, wherein the method is for intelligent grading of endoscopic images of ulcerative colitis. The types of evidence include one or more of the following: reduced vascularity, mucosal erythema, hemorrhage, exudation, erosion, and ulceration; The determination of the hold state, disappearance state, reappearance state, and position movement state includes: If, under the same mucosal location marker, candidate inflammatory evidence objects of the same type exist both before and after the intervention and do not meet the criteria for reappearance, the state is determined to be "maintained". If, only the corresponding candidate inflammatory evidence objects exist before the intervention, and the corresponding mucosal area meets the extraction criteria for the candidate inflammatory evidence objects after the intervention but does not meet the criteria for positional movement, the state is determined to be "disappeared". If the candidate inflammatory evidence objects are visible before the intervention, are occluded or invisible during the intervention, and reappear under the same mucosal location marker after the intervention, the state is determined to be "reappeared". The predicted displacement of the mucosal background determined by the non-rigid deformation field is subtracted from the actual displacement of the candidate inflammatory evidence object. When the direction of the remaining displacement is consistent with the optical flow direction of the fluid motion region and the mucosal position marker changes, it is determined to be a position movement state. When both the reappearance state and the maintenance state are satisfied, it is preferentially determined to be a reappearance state. When both the position movement state and the disappearance state are satisfied, it is preferentially determined to be a position movement state. The authenticity verification results corresponding to the maintained state or the reappearance state are identified as fixed mucosal origin; the authenticity verification results corresponding to the positional movement state and the bleeding or exudation type that disappeared after intervention and did not reappear are identified as fluid interference; the authenticity verification results of other candidate inflammatory evidence objects are identified as pending verification; candidate inflammatory evidence objects whose authenticity verification results are fixed mucosal origin or pending verification are retained, and candidate inflammatory evidence objects whose authenticity verification results are fluid interference are excluded.

5. The method of claim 4, wherein the method of intelligent grading of endoscopic images for ulcerative colitis is characterized by: The construction of the inflammatory evidence co-bitmap and image classification include: Spatial colocation connection edges are established between graph nodes whose mucosal location identifiers correspond to the same or overlapping mucosal regions, and temporal continuity connection edges are established between graph nodes whose mucosal location identifiers and evidence types are the same in different video frames. From samples labeled with image inflammation levels, determine the evidence type, evidence reliability, and connection edge combination corresponding to each ordered image category to form a preset level evidence pattern; Based on the degree of matching between the connection subgraph formed by candidate inflammatory evidence objects from fixed mucosa and the preset level evidence pattern, the ordered image category corresponding to the preset level evidence pattern with the highest matching degree is determined as the endoscopic image inflammation grading result.

6. The method of claim 1, wherein the method of intelligent grading of endoscopic images for ulcerative colitis is characterized by: Also includes: Extract at least two contact characterization data from the target colonoscopy video, including local image magnification change, radial optical flow, mucosal color change, reflective area change, and degree of defocus; When at least two contact characterization data change synchronously within the same continuous video frame segment and the corresponding change areas have a common spatial center, a lens contact event is determined. A lens contact trajectory is formed based on the position change of the common spatial center, and the video frame corresponding to when the at least two contact characterization data no longer change synchronously is determined as the video frame corresponding to the lens release event.

7. The method of claim 6, wherein the method is for ulcerative colitis endoscopic image intelligent grading. Also includes: Map the lens contact trajectory to the mucosal location marker; For candidate inflammatory evidence objects of bleeding type that were not extracted from the visible mucosal area before the lens contact event, and candidate inflammatory evidence objects of bleeding type that appeared for the first time after the lens contact event and whose mucosal position identifiers coincided with the mucosal position identifiers corresponding to the lens contact trajectory, contact association markers were set, and their authenticity verification results were determined as to be verified and retained as graph nodes.

8. The method of claim 7, wherein the method is for ulcerative colitis endoscopic image intelligent grading. Also includes: The target colonoscopy video is divided into the advance stage and the withdrawal stage based on the direction of camera movement and the order in which intestinal segments appear. Based on at least two region matching features among intestinal segment sequence, fold topology, vascular bifurcation, and local mucosal texture, match mucosal regions representing the same mucosal site in two stages and assign the same mucosal location identifier. Video frames that are not in contact with the lens, are not obstructed, do not have saturated reflection or severe defocus, and meet the extraction conditions for candidate inflammatory evidence objects of bleeding type are selected from the matched mucosal region to form a two-way composite image.

9. The method of claim 8, wherein the method is for ulcerative colitis endoscopic image intelligent grading. Also includes: Two-way diplopia verification was performed on candidate inflammatory evidence objects with contact association markers for bleeding types; When candidate inflammatory evidence objects with the same mucosal location identifier, overlapping local image regions, and similar color or shape are extracted from other bi-path composite images, they are identified as duplicates, their authenticity verification results are modified to fix the mucosal source and retained. When the corresponding mucosal areas in both the advance and retreat phases meet the extraction conditions and do not repeat, the authenticity verification result is modified to mechanical interference related to lens contact and excluded. If a video frame that meets the extraction conditions cannot be obtained from both stages, its authenticity verification result remains unverified. The inflammatory evidence co-bitmap is updated based on the modified authenticity verification results, and the images are reclassified.

10. An intelligent grading system for endoscopic images of ulcerative colitis, characterized in that, The method includes a processor and a memory storing a computer program, which, when executed by the processor, implements the steps of the intelligent grading method for endoscopic images of ulcerative colitis as described in any one of claims 1 to 9.