Endometrial polyp real-time detection system for hysteroscope video

Abstract

The invention relates to the technical field of image analysis, in particular to an endometrial polyp real-time detection system for a hysteroscope video, and the system can realize the following steps through the mutual cooperation of a plurality of modules: obtaining the hysteroscope video of a target patient, and carrying out the region segmentation of each frame of image in the hysteroscope video; a highlight area is screened out; according to texture contour conditions in each highlight area and the adjacent target area, determining a detail loss degree corresponding to each highlight area; according to the matching condition between the to-be-repaired images and the reference image blocks corresponding to the highlight areas in the to-be-repaired images, repairing the highlight areas in the to-be-repaired images; and according to images except the to-be-restored image in the hysteroscope video and the target restored image, realizing endometrial polyp area identification. According to the method, the image in the hysteroscope video is analyzed, the highlight area is self-adaptively repaired, detail loss is reduced to a certain extent, and therefore the polyp recognition accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and more specifically to a real-time endometrial polyp detection system for hysteroscopic video. Background Technology

[0002] In real-time detection of endometrial polyps during actual hysteroscopic video, the high-intensity light reflection caused by strong illumination is often one of the main factors leading to false detection. Because the endoscopic light source shines directly on the mucosal surface, it often causes extremely bright, saturated white spots in local areas, or forms large areas of highlighting, resulting in the loss of some tissue details.

[0003] For areas with high-brightness reflection, existing technologies often repair the highlights by performing single-frame restoration or simply discard them. However, single-frame restoration, such as diffusion-based algorithms and classic image completion, often "borrows" textures from the surrounding areas of the highlight region to fill the center, which may result in the loss of the real tissue structure or the introduction of pseudo-structures. On the other hand, directly discarding the highlight region may miss details related to the polyp, which can easily lead to missed detections or false detections, resulting in poor accuracy in polyp identification. Summary of the Invention

[0004] To address the technical problem of poor accuracy in polyp identification, this invention proposes a real-time endometrial polyp detection system for hysteroscopic video.

[0005] In a first aspect, the present invention provides a real-time detection system for endometrial polyps in hysteroscopic video, the system comprising: The video acquisition and region segmentation module is used to acquire hysteroscopic videos of the target patient and perform region segmentation on each frame of the hysteroscopic video to obtain the target region; The region filtering module is used to filter out highlight areas from all target areas based on the grayscale characteristics of the target area and the grayscale differences between adjacent target areas; The detail loss determination module is used to determine the detail loss degree of each highlight region based on the texture contour of each highlight region and its adjacent target regions; The image block filtering module is used to filter out the image to be repaired and the reference image block corresponding to the highlight area in each frame of the hysteroscopy video based on the degree of detail loss in the highlight area. The region inpainting module is used to repair the highlight regions in each image to be repaired based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight regions within it, thereby obtaining the target repaired image; The region recognition module is used to identify endometrial polyp regions based on images in the hysteroscopic video other than the image to be repaired, as well as the target image to be repaired.

[0006] In conjunction with the first aspect above, in one possible implementation, the step of segmenting each frame of the hysteroscopic video to obtain the target region includes: Any frame of the hysteroscopy video is designated as a marker image, and the marker image is divided into pixel blocks to obtain target pixel blocks, wherein each pixel block is a rectangular area of ​​a preset size; The average grayscale value of all pixels in each target pixel block is used to determine the representative grayscale value for each target pixel block. Based on the grayscale representative values ​​corresponding to all target pixel blocks, the target pixel blocks in the marked image are merged to obtain the target region.

[0007] In conjunction with the first aspect above, in one possible implementation, the step of selecting highlight regions from all target regions based on the grayscale characteristics of the target region and the grayscale differences between adjacent target regions includes: Any frame of the hysteroscopy video is designated as a marker image, any target region in the marker image is designated as a temporary region, and each target region in the marker image other than the temporary region is designated as a candidate region. The union of the preset neighborhoods corresponding to all pixels on the boundary of the temporary region is determined as the overall neighborhood region corresponding to the temporary region; If the intersection of the candidate region and the overall neighborhood region is not empty, then the candidate region is determined as the target region adjacent to the temporary region, and is denoted as the neighboring target region corresponding to the temporary region; Based on the average grayscale value of all pixels within the temporary region and the grayscale difference between the temporary region and its corresponding neighboring target regions, the possible highlight value of the temporary region is determined. If the highlight intensity value corresponding to the temporary area is greater than the preset highlight threshold, then the temporary area is recorded as a highlight area.

[0008] In conjunction with the first aspect above, in one possible implementation, determining the degree of detail loss corresponding to each highlight region based on the texture contour of each highlight region and its adjacent target regions includes: Any one highlight region is designated as a marked highlight region, and the image to which the marked highlight region belongs is designated as a marked highlight image. Each target region in the marked highlight image other than the marked highlight region is designated as a region to be determined. The union of the preset neighborhoods corresponding to all pixels on the boundary of the marked highlight region is determined as the neighborhood representative region corresponding to the marked highlight region; If the intersection of the region to be determined and the neighboring representative region is not empty, then the region to be determined is determined as the target region adjacent to the marked highlight region, and is recorded as the neighboring target region corresponding to the marked highlight region; The union of all neighboring target regions corresponding to the marked highlight region is determined as the overall surrounding region corresponding to the marked highlight region; The union of the marked highlight area and its corresponding surrounding area is determined as the overall expanded area corresponding to the marked highlight area; The degree of detail loss corresponding to the marked highlight area is determined based on the texture contour of the marked highlight area and its corresponding surrounding area and the overall expanded area.

[0009] In conjunction with the first aspect above, in one possible implementation, determining the degree of detail loss corresponding to the marked highlight area based on the texture contour of the marked highlight area and its corresponding overall surrounding area and overall expanded area includes: Edge detection is performed inside the overall expanded region corresponding to the marked highlight region to obtain the expanded edge corresponding to the marked highlight region; Edge detection is performed on the interior of the overall surrounding area to obtain the surrounding edges corresponding to the marked highlight area; Edge detection is performed inside the marked highlight area to obtain the base edge corresponding to the marked highlight area; From all expanded edges, select those that contain at least one surrounding edge as candidate edges; From all candidate edges, select the expanded edges that contain at least one basic edge as the target edges; The degree of detail loss corresponding to the marked highlight area is determined based on the number of target edges, the number of surrounding edges, the gradient difference between the surrounding edges contained in the target edge and the base edge, and the area of ​​the marked highlight area.

[0010] In conjunction with the first aspect above, in one possible implementation, the step of selecting the image to be repaired and the reference image block corresponding to the highlight area within it from the hysteroscopic video based on the degree of detail loss corresponding to the highlight area in each frame of the hysteroscopic video includes: The target confidence level of each frame of the hysteroscopic video is determined based on the degree of detail loss corresponding to the highlight areas in each frame. If the target confidence level corresponding to the image is less than or equal to the preset confidence threshold, then the image is identified as the image to be repaired. Reference image blocks corresponding to the highlight areas within each image to be repaired are selected from the hysteroscopic video.

[0011] In conjunction with the first aspect above, in one possible implementation, determining the target credibility corresponding to each frame of the hysteroscopic video based on the detail loss degree corresponding to the highlight area in each frame of the hysteroscopic video includes: Any frame of the hysteroscopy video is designated as a marker image, and the target confidence level corresponding to the marker image is determined based on the maximum value of the loss of detail corresponding to all highlight areas in the marker image.

[0012] In conjunction with the first aspect above, in one possible implementation, the step of selecting reference image blocks corresponding to the highlight regions within each image to be repaired from the hysteroscopic video includes: Any image to be repaired in the hysteroscopy video is identified as a marked image to be repaired. A predetermined number of frames closest to the marked image to be repaired are selected from the hysteroscopy video and recorded as temporary images. From all temporary images, temporary images with a target confidence level greater than a predetermined confidence threshold are selected and recorded as pending images. Using a feature point matching algorithm, regions that match each highlight region in the marked image to be repaired are selected from each undetermined image, and these regions are denoted as the matching regions corresponding to each highlight region in the marked image to be repaired. Based on the grayscale difference between each highlight region in the marked image to be repaired and its corresponding matching regions, reference image blocks corresponding to each highlight region in the marked image to be repaired are selected.

[0013] In conjunction with the first aspect above, in one possible implementation, the step of selecting reference image blocks corresponding to each highlight region in the marked image to be repaired based on the grayscale difference between each highlight region in the marked image to be repaired and its corresponding matching regions includes: Any highlight region in the marked image to be repaired is identified as the region to be repaired, and the overall expansion region corresponding to the region to be repaired and the overall expansion region corresponding to each of its matching regions are obtained; The feature point matching algorithm is used to determine the feature points in the overall expanded region corresponding to the region to be repaired, and the feature points in the overall expanded region corresponding to each of the corresponding matching regions. Based on the grayscale difference between the feature points in the overall expanded region corresponding to the region to be repaired and the matching feature points in the overall expanded region corresponding to each of its corresponding matching regions, the target similarity between the region to be repaired and each of its corresponding matching regions is determined. If the target similarity between the region to be repaired and its corresponding matching region is greater than a preset similarity threshold, then the matching region is determined as a reference image block corresponding to the region to be repaired.

[0014] In conjunction with the first aspect above, in one possible implementation, the step of repairing the highlight regions in each image to be repaired based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight regions within it, to obtain the target repaired image, includes: Any image to be repaired in the hysteroscopic video is designated as the marked image to be repaired, any highlight area in the marked image to be repaired is designated as the area to be repaired, and any pixel in the area to be repaired is designated as the pixel to be repaired. Pixels that match the pixel to be repaired are selected from each reference image block corresponding to the region to be repaired, and used as reference points to obtain the set of reference points corresponding to the pixel to be repaired, thereby obtaining the set of reference points corresponding to each pixel in each highlight region of the marked image to be repaired; The grayscale values ​​of all reference points in the reference point set corresponding to each pixel in each highlight region of the marked image to be repaired are weighted and averaged to obtain the target representative index corresponding to each pixel in each highlight region of the marked image to be repaired; The grayscale values ​​of all pixels in all highlight areas of the marked image to be repaired are updated to their corresponding target representative indexes to obtain the target repair image corresponding to the marked image to be repaired.

[0015] Secondly, the present invention provides a method for real-time detection of endometrial polyps in hysteroscopic video, implemented by a system for real-time detection of endometrial polyps in hysteroscopic video, the method comprising: Acquire hysteroscopic videos of the target patient and perform region segmentation on each frame of the hysteroscopic video to obtain the target region; Based on the grayscale characteristics of the target area and the grayscale differences between adjacent target areas, highlight areas are selected from all target areas. Based on the texture contours of each highlight region and its adjacent target regions, determine the degree of detail loss for each highlight region; Based on the degree of detail loss corresponding to the highlight areas in each frame of the hysteroscopic video, the image to be repaired and the reference image blocks corresponding to the highlight areas within it are selected from the hysteroscopic video. Based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight regions within it, the highlight regions in each image to be repaired are repaired to obtain the target repaired image; Based on the images in the hysteroscopic video other than the image to be repaired, as well as the target image to be repaired, the endometrial polyp region can be identified.

[0016] Thirdly, a server is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the device to perform the aforementioned method for real-time detection of endometrial polyps using hysteroscopic video.

[0017] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to execute the above-described method for real-time detection of endometrial polyps using hysteroscopic video.

[0018] Fifthly, a computer-readable storage medium is provided, which stores computer program code that, when executed on a computer, causes the computer to perform the above-described method for real-time detection of endometrial polyps using hysteroscopic video.

[0019] The present invention has the following beneficial effects: The present invention provides a real-time endometrial polyp detection system for hysteroscopic videos. By analyzing images in hysteroscopic videos, the system adaptively repairs highlight areas, reducing detail loss to a certain extent and solving the technical problem of poor polyp identification accuracy, thereby improving the accuracy of polyp identification. Specifically, the present invention identifies highlight areas in images by analyzing the grayscale of the images in hysteroscopic videos. Based on the texture contours of the highlight areas and their adjacent target areas, the system quantifies the degree of detail loss corresponding to the highlight areas, thereby selecting the images to be repaired and the reference image blocks corresponding to the highlight areas within them. Based on the matching between the images to be repaired and the reference image blocks corresponding to the highlight areas within them, the system repairs the highlight areas in each image to be repaired, reducing detail loss to a certain extent. Finally, based on the repaired images, the system achieves endometrial polyp identification, thereby improving the accuracy of polyp identification. Attached Figure Description

[0020] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of a real-time endometrial polyp detection system for hysteroscopic video according to the present invention; Figure 2 This is a flowchart of a method for real-time detection of endometrial polyps using hysteroscopic video according to the present invention; Figure 3 This is a schematic diagram of the structure of a computer device according to the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the specific implementation methods, structures, features, and effects of the technical solution proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] refer to Figure 1 This diagram illustrates a real-time endometrial polyp detection system for hysteroscopic video according to the present invention. The system includes: The video acquisition and region segmentation module 101 is used to acquire the hysteroscopic video of the target patient and perform region segmentation on each frame of the hysteroscopic video to obtain the target region.

[0025] The target patients may be those undergoing endometrial polyp detection. Hysteroscopic videos are dynamic images recorded in real time during examinations or surgeries using hysteroscopic equipment.

[0026] As an example, the video acquisition and region segmentation module 101 can specifically implement the following steps: The first step is to obtain hysteroscopic videos of the target patient.

[0027] For example, patient preparation involves, specifically, completing the patient's medical history assessment and obtaining informed consent before anesthesia; hysteroscopic procedure involves, specifically, preparing and disinfecting the hysteroscope and related equipment, injecting irrigation fluid (such as saline) to dilate the uterine cavity, slowly inserting the hysteroscope, and observing the endometrium and lesion areas; video acquisition involves, specifically, adjusting the light source and focus of the hysteroscope to ensure a clear image, recording the entire procedure, and ensuring that key areas are covered; video storage and management involves, specifically, saving the video to the detection system, where the video can be used as input data for subsequent polyp detection and analysis.

[0028] The second step is to identify any frame from the hysteroscopy video as the marker image, and then divide the marker image into pixel blocks to obtain the target pixel block.

[0029] Here, a pixel block can be a rectangular area of ​​a preset size. The preset size can be a relatively small size pre-set based on the actual scene, such as a preset size of 5×5. The target pixel block can be an image block obtained by equally dividing the marked image. If the preset size is 5×5, then the target pixel block can be a 5×5 pixel block.

[0030] The third step is to determine the average grayscale value of all pixels in each target pixel block as the representative grayscale value for each target pixel block.

[0031] The fourth step is to merge the target pixel blocks in the above-mentioned marked image according to the grayscale representative values ​​corresponding to all target pixel blocks to obtain the target region.

[0032] The target region can be composed of target pixel blocks with similar and adjacent grayscale values.

[0033] For example, the target region can be obtained by: using the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to cluster the target pixel blocks in the above-mentioned labeled image based on the grayscale representative values ​​corresponding to all target pixel blocks, and recording the continuous regions in each cluster as the target region, where the continuous regions are the regions that are adjacent in position.

[0034] Optionally, the method for obtaining the target region can also be: based on the gray-level representative values ​​corresponding to all target pixel blocks, perform region growing on all target pixel blocks in the above-mentioned marked image, and record the region obtained by region growing as the target region. In this case, the region growing criterion can be the gray-level representative value similarity criterion, that is, grow adjacent target pixel blocks with similar gray-level representative values ​​into the same region.

[0035] It should be noted that the target area is often a region composed of pixels with similar gray values.

[0036] The region filtering module 102 is used to filter out highlight regions from all target regions based on the grayscale of the target region and the grayscale differences between adjacent target regions.

[0037] It should be noted that during real-time detection of endometrial polyps in hysteroscopic video, the endoscopic light source often shines directly onto the mucosal surface, frequently resulting in extremely bright, color-saturated spots or large areas of highlights. After grayscale processing of the video, the grayscale values ​​of these highlighted areas are significantly higher, and because sudden strong light often only affects a localized area, their grayscale values ​​often differ significantly from the surrounding tissue. Therefore, highlighted areas can be identified and determined by their grayscale values ​​and their differences from the surrounding tissue.

[0038] As an example, filtering out highlight areas from all target areas may include the following steps: The first step is to identify any frame of the hysteroscopy video as a marker image, identify any target region in the marker image as a temporary region, and identify each target region in the marker image other than the temporary region as a candidate region.

[0039] The second step is to determine the union of the preset neighborhoods corresponding to all pixels on the boundary of the temporary region as the overall neighborhood region corresponding to the temporary region.

[0040] The preset neighborhood can be a neighborhood pre-set according to the actual scene, and it can be a 3×3 neighborhood.

[0041] The third step is to determine the candidate region as the target region adjacent to the temporary region if the intersection of the candidate region and the above-mentioned overall neighboring region is not empty. This candidate region is then recorded as the neighboring target region corresponding to the temporary region.

[0042] It should be noted that a temporary area may correspond to multiple adjacent target areas.

[0043] The fourth step is to determine the possible value of the highlight intensity of the temporary region based on the average gray value of all pixels in the temporary region and the gray value difference between the temporary region and its corresponding neighboring target regions.

[0044] For example, the formula for determining the possible specular value of a temporary region can be: ; ; ; Where S is the possible value of the highlight intensity corresponding to the temporary region. The weights of W are pre-set. The weights of A are pre-set and can be adjusted according to the actual scenario, and can be made... For example, it can be set W represents the possible value of the highlight intensity corresponding to the temporary region. A represents the degree of brightness difference corresponding to the temporary region. and These are pre-set adjustment factors for the corresponding variables, mainly used to adjust the function range. They can be set according to the actual scenario, such as... and Both can be set to 1. It is the largest gray value in the image. Since the range of gray values ​​is often [0, 255], therefore... It can be set to 255. It is the average grayscale value of all pixels within the temporary region. is the average grayscale value of all pixels in the a-th neighboring target region corresponding to the temporary region. N is the number of neighboring target regions corresponding to the temporary region. a is the index of the neighboring target region corresponding to the temporary region. and This represents the normalized value.

[0045] It should be noted that all variables in the embodiments of the present invention, especially the denominator, can be set with corresponding adjustment factors according to the actual situation to adjust their value range or prevent the denominator from being 0.

[0046] It should be noted that the stronger the light, the greater the likelihood that it will appear white in the video, and its grayscale value will tend to be closer to... That is, 255. Therefore, the larger W is, the more likely the temporary area is to be affected by strong light. The larger A is, the more likely the gray level of the temporary area is to be higher than that of the surrounding area, and the more likely the temporary area is to be affected by strong light. Therefore, the larger S is, the more likely the temporary area is to be affected by strong light, and the more likely the temporary area is to be a highlight area.

[0047] Fifth step: If the potential value of the highlight level corresponding to the above temporary area is greater than the preset highlight threshold, then the above temporary area is recorded as the highlight area.

[0048] The preset highlight threshold can be a pre-set threshold, which can be set according to the actual situation. For example, it can be 0.6.

[0049] The detail loss determination module 103 is used to determine the detail loss degree corresponding to each highlight area based on the texture contour of each highlight area and its adjacent target areas.

[0050] It should be noted that in hysteroscopic video frames, when highlights cause a significant increase in local brightness and blur or lose the actual tissue details of the endometrium, the system often cannot accurately identify the tissue structure in the highlighted area. In this case, it can be considered that the actual tissue of the video frame is obscured by the highlights, the frame has a high degree of detail loss, and the reliability is low.

[0051] Because different highlight conditions affect the endometrial region differently, some highlighted areas, although affected by light, often retain identifiable tissue details due to low intensity, and their internal structures typically maintain good continuity with surrounding unaffected tissue. However, in areas with excessively high highlight intensity, most of the tissue details may be obscured, making them unrecognizable and showing significant differences from the surrounding tissue. For areas where the tissue is identifiable, the video frame's reliability is generally high; conversely, for areas where details are obscured, the frame shows a higher degree of detail loss and thus lower reliability.

[0052] If a video frame has a large proportion of clearly defined highlight areas, it often indicates that the frame is significantly affected by light interference, reducing its overall usability. This suggests that the video frame has a high degree of detail loss and low reliability.

[0053] As an example, determining the level of detail loss for each highlight area may include the following steps: The first step is to identify any one highlight area as a marked highlight area, and to identify the image to which the marked highlight area belongs as a marked highlight image. Then, identify each target area in the marked highlight image other than the marked highlight area as a region to be identified.

[0054] The second step is to determine the union of the preset neighborhoods corresponding to all pixels on the boundary of the marked highlight area as the neighborhood representative area corresponding to the marked highlight area.

[0055] Third, if the intersection of the undetermined region and the above-mentioned neighboring representative region is not empty, then the undetermined region is determined as the target region adjacent to the above-mentioned marked highlight region, and is recorded as the neighboring target region corresponding to the above-mentioned marked highlight region.

[0056] The fourth step is to determine the union of all neighboring target areas corresponding to the marked highlight area as the overall surrounding area corresponding to the marked highlight area.

[0057] The fifth step is to determine the overall expansion region corresponding to the marked highlight area by combining the above-mentioned marked highlight area with the corresponding overall surrounding area.

[0058] Step 6, determining the level of detail loss corresponding to the marked highlight areas and their corresponding surrounding and expanded areas based on the texture contours of the marked highlight areas may include the following sub-steps: The first sub-step involves using the Canny edge detection algorithm to perform edge detection on the interior of the overall expanded region corresponding to the marked highlight region, and recording the detected edge as the expanded edge corresponding to the marked highlight region.

[0059] The second sub-step involves using the Canny edge detection algorithm to perform edge detection on the interior of the entire surrounding area, and recording the detected edges as the surrounding edges corresponding to the marked highlight areas.

[0060] The third sub-step involves using the Canny edge detection algorithm to perform edge detection inside the marked highlight area, and recording the detected edges as the base edges corresponding to the marked highlight area.

[0061] It should be noted that a base edge is often the outline edge of the highlight area or the edge of the internal residual texture.

[0062] The fourth sub-step involves selecting from all expanded edges that contain at least one surrounding edge as candidate edges.

[0063] The fifth sub-step involves selecting expansion edges from all candidate edges that contain at least one base edge as target edges.

[0064] The sixth sub-step determines the degree of detail loss corresponding to the marked highlight area based on the number of target edges, the number of surrounding edges, the gradient difference between the surrounding edges contained in the target edge and the base edge, and the area of ​​the marked highlight area.

[0065] For example, the formula for determining the degree of detail loss corresponding to the marked highlight area can be: ; ; Where E represents the degree of detail loss corresponding to the marked highlight area. , , and The weights of the corresponding variables are pre-set and can be adjusted according to the actual scenario. For example, it can be set , , , XS represents the possible specular intensity value corresponding to the marked specular area. b is the number of target edges. B is the number of surrounding edges. c is the index of the target edge. H is the texture gradient difference between the marked specular area and the surrounding area. It is an absolute value function. It is the average of the gradient values ​​corresponding to all pixels on all surrounding edges included in the c-th target edge. is the mean of the gradient values ​​corresponding to all pixels on all base edges contained in the c-th target edge. G is the maximum value of the gradient values ​​corresponding to all pixels in the marked specular image. m is the area of ​​the marked specular region, which can be represented by the number of pixels in the marked specular region. M is the area of ​​the largest target region in the marked specular image, which can be represented by the number of pixels in the largest target region in the marked specular image. , and This represents the normalized value.

[0066] It should be noted that a larger XS value generally indicates that the marked highlight area is more likely to be affected by strong light, more likely to be an area where the highlight brightness is significantly increased due to the highlight, and more likely to suffer detail loss due to the strong light. In reality, highlight areas often represent local locations of tissue affected by strong light. If not affected by strong light, their texture often shows a certain transition with the surrounding texture. That is, if not affected by strong light or by relatively weak light, the transitional texture will often be displayed in the image, showing that part of the texture of the highlight area is connected to the texture of the surrounding area. When The smaller the value of H, the more texture there is around the highlighted area, and the fewer textures connect the highlighted area to the surrounding area. This often indicates a stronger light influence causing texture loss in the highlighted area, resulting in a relatively smaller proportion of connected textures. Consequently, the highlighted area is more likely to lose detail due to strong light. In reality, if the texture within the highlighted area is not significantly affected by lighting, the difference in gradient between it and the surrounding connected textures is usually small. A larger value of H generally indicates a greater difference in texture gradient between the highlighted area and the surrounding area, suggesting a higher likelihood of detail loss due to strong light. A larger value for E usually indicates a higher proportion of the marked highlight area, suggesting it's more likely to be affected by a wider range of lighting. Therefore, a larger E value often indicates that the marked highlight area is more likely to lose detail due to strong light, and thus requires more restoration.

[0067] The formula for determining the level of detail loss corresponding to the marked highlight area can also be: ; in, , and The weights of the corresponding variables are pre-set and can be adjusted according to the actual scenario. For example, it can be set , , .

[0068] The image block filtering module 104 is used to filter out the image to be repaired and the reference image block corresponding to the highlight area in each frame of the hysteroscopic video based on the degree of detail loss in the highlight area.

[0069] As an example, selecting the image to be repaired and the corresponding reference image block for its highlight area from a hysteroscopy video may include the following steps: The first step is to determine the target credibility of each frame of the hysteroscopy video based on the degree of detail loss corresponding to the highlight areas in each frame.

[0070] For example, any frame of the hysteroscopy video can be designated as a marker image, and the target confidence level corresponding to the marker image can be determined based on the maximum value of the detail loss degree corresponding to all highlight areas in the marker image.

[0071] For example, the formula for determining the target confidence level corresponding to different frames in a hysteroscopy video can be: ; in, This represents the target confidence level corresponding to the i-th frame of the hysteroscopy video. i is the frame number of the image in the hysteroscopy video. It is the maximum value of detail loss corresponding to all highlight areas in the i-th frame of the hysteroscopy video.

[0072] It should be noted that when A larger value often indicates that the i-th frame of the hysteroscopy video is more likely to contain areas of detail loss due to strong light, and thus indicates that the i-th frame of the hysteroscopy video requires more restoration. Therefore, when A larger value usually indicates that there are less likely areas in the i-th frame of the hysteroscopy video to be lost due to strong light, meaning that the i-th frame of the hysteroscopy video does not need to be repaired and can be used as a reference image for repairing lost details.

[0073] The second step is to identify the image as the image to be repaired if the target credibility of the image is less than or equal to the preset credibility threshold.

[0074] The preset confidence threshold can be a pre-set threshold, which can be set according to the actual situation. For example, it can be 0.5.

[0075] The third step, selecting the reference image block corresponding to the highlight area in each image to be repaired from the above hysteroscopic videos, may include the following sub-steps: The first sub-step involves identifying any image to be repaired in the aforementioned hysteroscopic video as a marked image to be repaired, and selecting a preset number of frames from the aforementioned hysteroscopic video that are closest to the marked image to be repaired, which are then recorded as temporary images. From all temporary images, temporary images with a target confidence level greater than a preset confidence threshold are selected and recorded as pending images.

[0076] The preset quantity can be a pre-set quantity, which can be set according to the actual scenario. It can be greater than or equal to 5, for example, it can be 10.

[0077] It should be noted that the temporary image can be used as a reference image to mark the image to be repaired and restore lost details. The acquisition interval between the temporary image and the marked image to be repaired is often extremely short, for example, within 0.5 seconds, at which point soft tissue deformation can often be ignored.

[0078] The second sub-step involves using a feature point matching algorithm to select regions from each undetermined image that match each highlight region in the marked image to be repaired, and these regions are denoted as the matching regions corresponding to each highlight region in the marked image to be repaired.

[0079] The feature point matching algorithm can be SIFT (Scale-Invariant Feature Transform). Highlight regions and their corresponding matching regions often represent the same structure.

[0080] It should be noted that when obtaining the matching region corresponding to the highlight region, feature matching can be performed on each undetermined image and the marked image to be repaired to obtain the region that matches each undetermined image and the marked image to be repaired. Thus, the region in each undetermined image that matches the highlight region in the marked image to be repaired can be obtained.

[0081] The third sub-step, which involves selecting reference image blocks corresponding to each highlight region in the marked image to be repaired based on the grayscale difference between each highlight region and its corresponding matching regions, may include the following steps: First, any highlight region in the marked image to be repaired is identified as the region to be repaired, and the overall expansion region corresponding to the region to be repaired and the overall expansion region corresponding to each of its matching regions are obtained.

[0082] The methods for obtaining the overall expansion area corresponding to the area to be repaired and the overall expansion area corresponding to the matching area are the same as those for obtaining the overall expansion area corresponding to the marked highlight area, and will not be repeated here.

[0083] Next, using a feature point matching algorithm, the feature points in the overall expanded region corresponding to the region to be repaired are determined to be the matching feature points in the overall expanded region corresponding to each of the matching regions.

[0084] Then, based on the grayscale difference between the feature points in the overall expanded region corresponding to the region to be repaired and the matching feature points in the overall expanded region corresponding to each of the matching regions, the target similarity between the region to be repaired and each of the matching regions is determined.

[0085] Among them, a feature point and its matching feature point often represent the same position in the same structure.

[0086] For example, the formula for determining the target similarity between the region to be repaired and its corresponding matching region can be: ; in, is the target similarity between the region to be repaired and its corresponding j-th matching region. j is the index of the matching region corresponding to the region to be repaired. Y is the number of feature points detected by the feature point matching algorithm in the overall expanded region corresponding to the region to be repaired. x is the index of the feature points detected by the feature point matching algorithm in the overall expanded region corresponding to the region to be repaired. It is an absolute value function. It is the gray value of the x-th feature point in the overall expanded region corresponding to the region to be repaired. It is the gray value of the feature point that matches the x-th feature point within the overall expanded area corresponding to the j-th matching area of ​​the area to be repaired. It is the largest gray value in the image. Since the range of gray values ​​is often [0, 255], therefore... It can be set to 255. and This represents the normalized value.

[0087] It should be noted that in practice, details of the area to be repaired are often lost. Therefore, when quantifying the similarity between the area to be repaired and its corresponding matching area, it is often necessary to introduce its surrounding area. The more similar the grayscale distribution between the overall expanded area corresponding to the area to be repaired and the overall expanded area corresponding to its corresponding matching area, the more likely the tissue locations represented by the two areas are to be the same. The larger the value, the more similar the grayscale distribution between the overall expanded region corresponding to the region to be repaired and the overall expanded region corresponding to the j-th matching region is. This often indicates that the detailed information that the region to be repaired and its corresponding j-th matching region should contain is more likely to be consistent, and that the j-th matching region is more suitable for updating the region to be repaired.

[0088] Finally, if the target similarity between the region to be repaired and its corresponding matching region is greater than a preset similarity threshold, then the matching region is determined as a reference image block corresponding to the region to be repaired.

[0089] The preset similarity threshold can be a pre-set threshold, which can be set according to the actual situation. For example, it can be 0.8.

[0090] It should be noted that when performing highlight region completion on low-confidence video frames, a reference frame with higher confidence can be selected from adjacent frames. Feature point extraction and matching are then performed between the current frame and the reference frame to achieve inter-frame geometric alignment. Since hysteroscopy videos often involve movement, it is frequently necessary to use a reference frame with high scene similarity from the current low-confidence video frame as the true reference frame for highlight region completion. Because the endometrial region typically remains stable during acquisition, its corresponding features often change little between adjacent frames. Therefore, the corresponding regions of adjacent blocks of the current highlight region in adjacent frames can be determined, and their similarity can be calculated. When the similarity between adjacent blocks of the highlight region is high, weighted fusion can be used to map the reliable details of the highlight region in adjacent frames to the current frame, thereby completing the illuminated area and ensuring a natural and coherent completion result.

[0091] The region repair module 105 is used to repair the highlight regions in each image to be repaired based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight regions within it, thereby obtaining the target repaired image.

[0092] It should be noted that the embodiments of the present invention mainly target the repair of highlight regions where there is a corresponding reference image block. If there is no corresponding reference image block for the highlight region, the highlight region can remain as is or be repaired using other existing methods during the repair process. For example, a single-frame spatial inpainting algorithm (such as diffusion inpainting or the Telea algorithm) can be used to fill in the highlight region.

[0093] As an example, repairing the highlight areas in each image to be repaired to obtain the target repaired image may include the following steps: The first step is to identify any image to be repaired in the hysteroscopy video as the marked image to be repaired, identify any highlight area in the marked image as the area to be repaired, and identify any pixel in the area to be repaired as the pixel to be repaired.

[0094] The second step involves using a feature point matching algorithm to select pixels that match the pixels to be repaired from each reference image block corresponding to the region to be repaired, and using these pixels as reference points to obtain a set of reference points corresponding to the pixels to be repaired. This results in a set of reference points corresponding to each pixel in each highlight region of the marked image to be repaired.

[0095] The set of reference points corresponding to the pixel to be repaired may include: pixels in all reference image blocks corresponding to the region to be repaired that match the pixel to be repaired.

[0096] The third step is to perform a weighted average of the gray values ​​of all reference points in the reference point set corresponding to each pixel in each highlight region of the above-mentioned marked image to be repaired, so as to obtain the target representative index corresponding to each pixel in each highlight region of the above-mentioned marked image to be repaired. At this time, the weight of the weighted average can be set according to the actual scene, and the cumulative value of all weights of the weighted average can be 1.

[0097] Optionally, the method for obtaining the target representative index corresponding to each pixel in each highlight region of the image to be repaired can also be: the mean of the gray values ​​corresponding to all reference points in the reference point set corresponding to each pixel in each highlight region of the image to be repaired is determined as the target representative index corresponding to each pixel in each highlight region of the image to be repaired.

[0098] The fourth step is to update the grayscale values ​​of all pixels in all highlight areas of the marked image to be repaired to their corresponding target representative index, thereby obtaining the target repair image corresponding to the marked image to be repaired.

[0099] The region recognition module 106 is used to recognize the endometrial polyp region based on the images in the hysteroscopic video other than the image to be repaired, as well as the target repair image.

[0100] As an example, the time sequence consisting of all images in the hysteroscopic video except for the image to be repaired, as well as all the target repair images, can be denoted as the target image sequence. Then, the endometrial polyp region can be identified in each frame of the target image sequence using neural network technology or threshold segmentation.

[0101] Optionally, the method for identifying endometrial polyp regions can also be as follows: update the image to be repaired in the hysteroscopic video to its corresponding target repair image to obtain the repaired hysteroscopic video; use U-Net++ (Nested U-Net) segmentation technology to perform tissue structure segmentation on the repaired video and extract the structural information of the endometrial region; and further identify possible endometrial polyp regions based on the segmentation results, thereby achieving accurate detection and localization of polyp parts.

[0102] It should be noted that when repairing highlight areas, highlight area detection can be performed on each frame of video first. Brightness analysis identifies areas with extremely high brightness and significant grayscale differences from surrounding tissues, thus defining these areas as highlights. Subsequently, the reliability of each frame is assessed based on the clarity of its details, identifying low-reliability frames as targets requiring repair. During compensation, the system uses highly reliable and detailed areas in adjacent frames as references to perform cross-frame completion of tissue structures obscured by highlights or blur, thereby improving the overall usability of the frames. Polyp identification on frames after highlight repair and cross-frame completion effectively restores obscured tissue details, significantly reduces the probability of missed and false detections, and improves the robustness and clinical usability of the detection.

[0103] refer to Figure 2 Based on the same inventive concept as the above-described method embodiments, this invention provides a method for real-time detection of endometrial polyps using hysteroscopic video, comprising the following steps: Step S1: Acquire the hysteroscopic video of the target patient and perform region segmentation on each frame of the hysteroscopic video to obtain the target region.

[0104] Step S2: Based on the grayscale of the target area and the grayscale difference between adjacent target areas, select the highlight area from all target areas.

[0105] Step S3: Determine the degree of detail loss for each highlight region based on the texture contour of each highlight region and its adjacent target regions.

[0106] Step S4: Based on the degree of detail loss corresponding to the highlight areas in each frame of the hysteroscopic video, select the image to be repaired and the reference image block corresponding to the highlight areas within it from the hysteroscopic video.

[0107] Step S5: Based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight areas within it, the highlight areas in each image to be repaired are repaired to obtain the target repaired image.

[0108] Step S6: Based on the images in the hysteroscopic video other than the image to be repaired, and the target image to be repaired, the endometrial polyp region is identified.

[0109] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. For example, as shown... Figure 3 As shown, the computer device 300 includes: a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and running on the processor 302. When the processor 302 executes the computer program 303, the computer device can execute the aforementioned method for real-time detection of endometrial polyps using hysteroscopic video.

[0110] Based on the same inventive concept as the above-described method embodiments, the present invention provides a server, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the device to execute the above-described method for real-time detection of endometrial polyps using hysteroscopic video.

[0111] Based on the same inventive concept as the above-described method embodiments, the present invention provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to execute the above-described method for real-time detection of endometrial polyps using hysteroscopic video.

[0112] Based on the same inventive concept as the above-described method embodiments, the present invention provides a computer-readable storage medium storing computer program code, which, when executed on a computer, causes the computer to perform the above-described method for real-time detection of endometrial polyps using hysteroscopic video.

[0113] In summary, this invention analyzes the grayscale of images in hysteroscopic videos to identify highlight regions. Based on the texture contours of the highlight regions and their adjacent target regions, it quantifies the degree of detail loss corresponding to the highlight regions, thereby selecting the images to be repaired and the reference image blocks corresponding to their highlight regions. Based on the matching between the images to be repaired and the reference image blocks corresponding to their highlight regions, the highlight regions in each image to be repaired are repaired, which reduces the loss of detail to a certain extent. Finally, based on the repaired images, the identification of endometrial polyps is achieved, thereby improving the accuracy of polyp identification.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A real-time detection system for endometrial polyps using hysteroscopic video, characterized in that... The system includes: The video acquisition and region segmentation module is used to acquire hysteroscopic videos of the target patient and perform region segmentation on each frame of the hysteroscopic video to obtain the target region; The region filtering module is used to filter out highlight areas from all target areas based on the grayscale characteristics of the target area and the grayscale differences between adjacent target areas; The detail loss determination module is used to determine the detail loss degree of each highlight region based on the texture contour of each highlight region and its adjacent target regions; The image block filtering module is used to filter out the image to be repaired and the reference image block corresponding to the highlight area in each frame of the hysteroscopy video based on the degree of detail loss in the highlight area. The region inpainting module is used to repair the highlight regions in each image to be repaired based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight regions within it, thereby obtaining the target repaired image; The region recognition module is used to identify endometrial polyp regions based on images in the hysteroscopic video other than the image to be repaired, as well as the target image to be repaired.

2. The real-time endometrial polyp detection system for hysteroscopic video as described in claim 1, characterized in that... The process of segmenting each frame of the hysteroscopic video to obtain the target region includes: Any frame of the hysteroscopy video is designated as a marker image, and the marker image is divided into pixel blocks to obtain target pixel blocks, wherein each pixel block is a rectangular area of ​​a preset size; The average grayscale value of all pixels in each target pixel block is used to determine the representative grayscale value for each target pixel block. Based on the grayscale representative values ​​corresponding to all target pixel blocks, the target pixel blocks in the marked image are merged to obtain the target region.

3. The real-time endometrial polyp detection system for hysteroscopic video as described in claim 1, characterized in that... The step of selecting highlight regions from all target regions based on the grayscale of the target region and the grayscale differences between adjacent target regions includes: Any frame of the hysteroscopy video is designated as a marker image, any target region in the marker image is designated as a temporary region, and each target region in the marker image other than the temporary region is designated as a candidate region. The union of the preset neighborhoods corresponding to all pixels on the boundary of the temporary region is determined as the overall neighborhood region corresponding to the temporary region; If the intersection of the candidate region and the overall neighborhood region is not empty, then the candidate region is determined as the target region adjacent to the temporary region, and is denoted as the neighboring target region corresponding to the temporary region; Based on the average grayscale value of all pixels within the temporary region and the grayscale difference between the temporary region and its corresponding neighboring target regions, the possible highlight value of the temporary region is determined. If the highlight intensity value corresponding to the temporary area is greater than the preset highlight threshold, then the temporary area is recorded as a highlight area.

4. The real-time endometrial polyp detection system for hysteroscopic video as described in claim 1, characterized in that... The step of determining the degree of detail loss for each highlight region based on the texture contour of each highlight region and its adjacent target regions includes: Any one highlight region is designated as a marked highlight region, and the image to which the marked highlight region belongs is designated as a marked highlight image. Each target region in the marked highlight image other than the marked highlight region is designated as a region to be determined. The union of the preset neighborhoods corresponding to all pixels on the boundary of the marked highlight region is determined as the neighborhood representative region corresponding to the marked highlight region; If the intersection of the region to be determined and the neighboring representative region is not empty, then the region to be determined is determined as the target region adjacent to the marked highlight region, and is recorded as the neighboring target region corresponding to the marked highlight region; The union of all neighboring target regions corresponding to the marked highlight region is determined as the overall surrounding region corresponding to the marked highlight region; The union of the marked highlight area and its corresponding surrounding area is determined as the overall expanded area corresponding to the marked highlight area; The degree of detail loss corresponding to the marked highlight area is determined based on the texture contour of the marked highlight area and its corresponding surrounding area and the overall expanded area.

5. A real-time endometrial polyp detection system for hysteroscopic video as described in claim 4, characterized in that... The step of determining the degree of detail loss corresponding to the marked highlight area based on the texture contour of the marked highlight area and its corresponding overall surrounding area and overall expanded area includes: Edge detection is performed inside the overall expanded region corresponding to the marked highlight region to obtain the expanded edge corresponding to the marked highlight region; Edge detection is performed on the interior of the overall surrounding area to obtain the surrounding edges corresponding to the marked highlight area; Edge detection is performed inside the marked highlight area to obtain the base edge corresponding to the marked highlight area; From all expanded edges, select those that contain at least one surrounding edge as candidate edges; From all candidate edges, select the expanded edges that contain at least one basic edge as the target edges; The degree of detail loss corresponding to the marked highlight area is determined based on the number of target edges, the number of surrounding edges, the gradient difference between the surrounding edges contained in the target edge and the base edge, and the area of ​​the marked highlight area.

6. The real-time endometrial polyp detection system for hysteroscopic video as described in claim 1, characterized in that... The step of selecting the image to be repaired and the reference image block corresponding to the highlight area in each frame of the hysteroscopic video based on the degree of detail loss in the highlight area includes: The target confidence level of each frame of the hysteroscopic video is determined based on the degree of detail loss corresponding to the highlight areas in each frame. If the target confidence level corresponding to the image is less than or equal to the preset confidence threshold, then the image is identified as the image to be repaired. Reference image blocks corresponding to the highlight areas within each image to be repaired are selected from the hysteroscopic video.

7. A real-time endometrial polyp detection system for hysteroscopic video as described in claim 6, characterized in that... The step of determining the target credibility of each frame of the hysteroscopic video based on the detail loss degree corresponding to the highlight area in each frame of the video includes: Any frame of the hysteroscopy video is designated as a marker image, and the target confidence level corresponding to the marker image is determined based on the maximum value of the loss of detail corresponding to all highlight areas in the marker image.

8. A real-time endometrial polyp detection system for hysteroscopic video as described in claim 6, characterized in that... The step of selecting reference image blocks corresponding to the highlight areas within each image to be repaired from the hysteroscopic video includes: Any image to be repaired in the hysteroscopy video is identified as a marked image to be repaired. A predetermined number of frames closest to the marked image to be repaired are selected from the hysteroscopy video and recorded as temporary images. From all temporary images, temporary images with a target confidence level greater than a predetermined confidence threshold are selected and recorded as pending images. Using a feature point matching algorithm, regions that match each highlight region in the marked image to be repaired are selected from each undetermined image, and these regions are denoted as the matching regions corresponding to each highlight region in the marked image to be repaired. Based on the grayscale difference between each highlight region in the marked image to be repaired and its corresponding matching regions, reference image blocks corresponding to each highlight region in the marked image to be repaired are selected.

9. A real-time endometrial polyp detection system for hysteroscopic video as described in claim 8, characterized in that... The step of selecting reference image blocks corresponding to each highlight region in the marked image to be repaired based on the grayscale difference between each highlight region and its corresponding matching regions includes: Any highlight region in the marked image to be repaired is identified as the region to be repaired, and the overall expansion region corresponding to the region to be repaired and the overall expansion region corresponding to each of its matching regions are obtained; The feature point matching algorithm is used to determine the feature points in the overall expanded region corresponding to the region to be repaired, and the feature points in the overall expanded region corresponding to each of the corresponding matching regions. Based on the grayscale difference between the feature points in the overall expanded region corresponding to the region to be repaired and the matching feature points in the overall expanded region corresponding to each of its corresponding matching regions, the target similarity between the region to be repaired and each of its corresponding matching regions is determined. If the target similarity between the region to be repaired and its corresponding matching region is greater than a preset similarity threshold, then the matching region is determined as a reference image block corresponding to the region to be repaired.

10. A real-time endometrial polyp detection system for hysteroscopic video as described in claim 1, characterized in that... The step of repairing the highlight regions in each image to be repaired based on the matching between the image to be repaired and the reference image blocks corresponding to the highlight regions within it, to obtain the target repaired image, includes: Any image to be repaired in the hysteroscopic video is designated as the marked image to be repaired, any highlight area in the marked image to be repaired is designated as the area to be repaired, and any pixel in the area to be repaired is designated as the pixel to be repaired. Pixels that match the pixel to be repaired are selected from each reference image block corresponding to the region to be repaired, and used as reference points to obtain the set of reference points corresponding to the pixel to be repaired, thereby obtaining the set of reference points corresponding to each pixel in each highlight region of the marked image to be repaired; The grayscale values ​​of all reference points in the reference point set corresponding to each pixel in each highlight region of the marked image to be repaired are weighted and averaged to obtain the target representative index corresponding to each pixel in each highlight region of the marked image to be repaired; The grayscale values ​​of all pixels in all highlight areas of the marked image to be repaired are updated to their corresponding target representative indexes to obtain the target repair image corresponding to the marked image to be repaired.

Images