Method and system for extracting features of dyed regions in vaginal endoscope images
By adjusting the light source and lens position, combining RGB screening and optical flow vector analysis, multiple identification block sets are obtained and intersections are obtained, which solves the problem of indistinguishable light spots from the lesion area in vaginal endoscopy detection, and improves the abnormal staining area Extraction accuracy and detection accuracy.
Patent Information
- Application Number
- CN202411346483.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-26
AI Technical Summary
In vaginal endoscopic detection, it is difficult for the prior art to accurately extract the characteristics of the stained area, especially in early cancer screening, where the reflective points are close to the color gamut of the lesion area, resulting in low accuracy of automated image extraction.
By adjusting the light source and lens position, combining RGB screening conditions and optical flow vector analysis, multiple recognition block sets are obtained and intersections are obtained to form a preliminary feature area, and a feature extraction model is input to extract abnormal stained areas to avoid error deletion and noise interference.
It significantly improves the extraction accuracy of abnormally stained areas in vaginal endoscopic images, reduces the training complexity and processing time of feature extraction models, and enhances the accuracy of detection.
Smart Images

Figure CN118887241B_ABST
Abstract
Description
Technical Field
[0001] The image processing technology of the present invention relates to a field, and in particular, to a method and system for extracting features of a stained area of a vaginal endoscope image. Background Art
[0002] During vaginal endoscopic examination, doctors will use dyeing to determine whether there is a lesion. Doctors will inject some dyeing liquid into the guide tube, and then use an endoscope to observe the color of the dyed part, and determine whether there is a lesion based on the color difference. Currently, commonly used dyes generally dye the lesion area white. During endoscopic examination, it can be clearly seen that the lesion area presents a particularly thick white shape, which can be used to determine whether there is a lesion.
[0003] In actual examinations, especially in early cancer screening, most patients are healthy patients without deterioration, and only a small number of patients have very small cancerous changes. Therefore, most endoscopic images are healthy images, and only a small number of patients have a little white spot the size of a grain of rice (1mm~2mm) in their endoscopic images. Therefore, in early cancer screening, it is difficult to accurately determine whether there is a lesion area based on the endoscope.
[0004] To this end, some endoscope systems use automated image extraction technology, which is based on a neural network model. It learns and filters the RGB three-channel information of the lesion area to obtain the RGB three-channel color gamut range of the lesion area, thereby automatically outlining possible lesion areas.
[0005] When collecting endoscopic images, in addition to the camera, a light source is also needed for the camera. The area collected by the endoscope has a lot of liquid after being stained, so when the light source shines on it, many reflective spots will be formed. These reflective spots are also very small, and the color gamut of the three primary colors in the image is very close to the color gamut of the lesion area. It is impossible to accurately distinguish between reflective spots and lesion areas based solely on color features, so in practice, the accuracy of automatically extracting lesion areas using neural network models is not high. Summary of the invention
[0006] The content of the present invention is used to introduce the concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of the present invention is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.
[0007] In order to solve the technical problem that the image features of the stained area cannot be accurately extracted, the present invention provides a method for extracting the features of the stained area of a vaginal endoscope image, comprising the following steps:
[0008] Step 1: Acquire an endoscopic image of the target area;
[0009] Step 2: Based on the RGB screening condition, a number of recognition blocks are screened out from the endoscopic image to obtain a first recognition block set;
[0010] Step 3: Filter out several feature points that are irrelevant to the light source change from the endoscopic image and use the feature points as reference points;
[0011] Step 4: Keep the lens fixed and change the position of the light source; re-collect the endoscopic image of the target area, and then filter out a number of recognition blocks from the endoscopic image based on the RGB screening condition to obtain a second recognition block set;
[0012] Step 5: restore the light source to its initial position, keep the light source fixed, and translate the lens; recollect the endoscopic image of the target area, and then filter out a number of recognition blocks from the endoscopic image based on the RGB screening condition to obtain a third recognition block set;
[0013] Step 6: Intersecting the first recognition block set, the second recognition block set and the third recognition block set to obtain a preliminary feature region;
[0014] The preliminary feature regions are input into the feature extraction model, and the feature extraction model extracts abnormal staining regions therefrom.
[0015] In the technical solution proposed by the present invention, in order to accurately identify and extract abnormal staining areas, this solution first uses the RGB algorithm to perform three extraction operations in steps, obtains multiple recognition blocks respectively, and then forms a first recognition block set, a second recognition block set and a third recognition block set. Subsequently, by obtaining the intersection of these three recognition block sets, a preliminary feature area can be obtained, which is regarded as the key basis for the subsequent extraction of abnormal staining areas. This method cleverly utilizes the characteristics that the reflective area in the endoscopic image changes with the position of the light source and the camera position, so that the reflective area affected by the light source in the image can be accurately screened out. In addition, unlike directly eliminating the part that may be a reflective point, this solution uses the intersection of three recognition block sets for screening, which effectively avoids the loss of recognition blocks containing abnormal staining areas due to accidental deletion. In this way, when the subsequent feature extraction model is used to extract abnormal staining areas, its input image information is more concise and rich in effective information, which not only reduces the training complexity of the feature extraction model, but also significantly improves the processing efficiency of the model.
[0016] Furthermore, step 1 includes the following steps:
[0017] Step 11: Move the lens to a predetermined position so that the lens is aimed at the target area;
[0018] Step 12: Adjust the focal length of the lens to make the target area clear and place the target area in the center of the lens's field of view; record the position of the lens and the focal length of the lens at this time to obtain the position information of the lens;
[0019] Step 13: Acquire at least two endoscopic images A and B of the current target area.
[0020] In the technical solution described in the present invention, by recording the position information and focal length setting of the lens, it is ensured that even after the lens is moved or the focal length is adjusted, the actual size of the current target area can still be accurately determined. This design effectively avoids the problem of lack of necessary correspondence between the first two photos taken due to changes in the lens position or focal length, thereby ensuring that these photos can be smoothly used in the subsequent image screening process.
[0021] In the process of screening potential lesion areas based on tissue staining conditions, the key is to identify areas that may indicate lesions even if they are tiny, so as to issue early warnings to patients in a timely manner and promote the goal of early detection and early treatment. However, if too much focus is placed on extremely tiny lesion areas, the screening results are susceptible to noise interference, which in turn has an adverse effect on the accuracy of the model in the subsequent feature extraction model training or prediction stages.
[0022] Furthermore, step 2 includes the following steps:
[0023] Step 21: pre-configure RGB screening conditions, where the RGB screening conditions include a range of pixel values in a red channel, a range of pixel values in a green channel, and a range of pixel values in a blue channel;
[0024] Step 22: Identify the pixel value of each channel in each pixel point in the endoscope image A, filter out the pixel points that meet the RGB screening conditions, and obtain a number of initial points;
[0025] Step 23: pre-set the minimum connection distance r, generate a circular area outward from each initial point with a radius of r, and if the circular areas of two or more initial points intersect, merge the circular areas of the initial points to obtain several initial areas;
[0026] Step 24: Filter out initial regions whose areas are smaller than a preset area threshold to obtain a number of recognition blocks, thereby obtaining a first recognition block set.
[0027] The technical solution proposed in the present invention first uses the RGB screening standard to accurately screen out qualified pixels in the identification block extraction stage, thereby locking the potential area where the lesion exists at the pixel level. Subsequently, in order to effectively resist noise interference, this solution adopts a pixel diffusion strategy so that adjacent and close pixels can be aggregated into a larger initial area. Furthermore, if the area of the initial area exceeds the preset threshold, it is regarded as a possible outlier; conversely, if it does not exceed the preset range, it is judged as noise and directly excluded. This scheme can not only effectively filter out noise, but also prevent the mistaken deletion of many small suspected lesion areas during the noise removal process, thereby significantly improving the accuracy of abnormal staining area extraction.
[0028] When using dyes to stain diseased tissues, the diseased area appears white, which can easily be confused with abnormal reflective spots. These reflective spots, which are only the size of rice grains, have a low visual similarity with the staining results. For machine learning algorithms, it is quite challenging to distinguish the differences between such small samples. If the machine algorithm is forced to recognize these tiny differences, it may cause the model to overfit in actual applications, which in turn damages the prediction accuracy. In response to this problem, the present invention proposes the following technical solutions:
[0029] A further step 3 comprises the following steps:
[0030] Step 31: Acquire endoscopic image A and endoscopic image B;
[0031] Step 32: Based on the identification block obtained in step 2, the area where the identification block is located is marked on the endoscopic image A and the endoscopic image B respectively;
[0032] Step 33: Randomly select feature points with the same position on the endoscopic image A and the endoscopic image B from the area of the non-recognition block to perform similarity detection;
[0033] If the similarity exceeds the threshold, the feature point is recorded as a reference point;
[0034] If the threshold is not exceeded, the feature point is recorded as a non-reference point; until 10 reference points at least 1 mm apart are screened out.
[0035] The technical solution proposed in the present invention aims to effectively distinguish between the lesion area and the reflective points. The core strategy is to eliminate the interference of the reflective points by adjusting the lens position and the light source position. Whether adjusting the light source or the lens position, it is necessary to ensure that the corresponding images can be accurately aligned so as to accurately screen out the reflective points. To this end, the primary task of this solution is to determine two reference points that are independent of the light source position and the lens position, so as to achieve accurate overlap of the endoscopic images before and after the lens position changes. In the process of screening the reference points, the present invention first excludes the identification blocks containing the highlight area, and then ensures that the reference points are kept at a distance of 1mm when selecting the reference points. This design not only takes into account the possible slight rotation of the lens, but also ensures that even in this case, the two images can be accurately aligned according to the reference points. Through this method, the present solution can accurately lock the reference points of the endoscopic image, thereby greatly improving the accuracy of removing reflective points.
[0036] Due to the time interval between the two collected endoscopic images, part of the patient's tissue may be displaced during the detection process due to factors such as muscle shaking. If these positions prone to displacement are selected as reference points, it is very easy to cause the reference points to be lost, thereby affecting the accuracy of the overlap of the endoscopic images in subsequent steps 4 and 5. In response to this problem, the present invention proposes the following technical solutions:
[0037] Step 33 includes the following steps:
[0038] Step 331: Arrange the endoscopic image A and the endoscopic image B in the order of acquisition, and overlap the endoscopic image A and the endoscopic image B. When the endoscopic image A and the endoscopic image B overlap, the corresponding position of each pixel in the target area is the same;
[0039] Step 332: remove the area where the identification block in the endoscopic image A and the endoscopic image B is located;
[0040] Step 333: grayscale processing the endoscopic image A and the endoscopic image B;
[0041] Step 334: for each pixel point (x, y) at the corresponding position in the endoscopic image A and the endoscopic image B, the following minimization energy function P (u, v) is constructed;
[0042] ;
[0043] ;
[0044] ;
[0045] Among them, Q1 and Q2 are the first intermediate parameter and the second intermediate parameter, u and v are the horizontal and vertical components of the optical flow vector, which represent the pixel point from IA To I B distance, σ is the smoothing weight, I A is the endoscopic image A, I B is the endoscopic image B, x and y are the coordinates of the pixel points;
[0046] , Respectively represent the partial derivatives of u in the x direction and y direction, which are used to measure the rate of change of u;
[0047] , Represent the partial derivatives of v in the x direction and y direction, respectively, which are used to measure the rate of change of v;
[0048] dx, dy: represent small changes in the x-direction and y-direction respectively, used for integral calculation;
[0049] ∂ represents the operator of partial derivatives;
[0050] is the double integral symbol;
[0051] Step 335: For each pixel, calculate the size U of its optical flow vector;
[0052] , for each pixel point, if U is greater than α, the pixel point is recorded as 1, and if it is not greater than α, the pixel point is recorded as 0, so as to obtain a binary image representing the changes of the endoscopic image A and the endoscopic image B;
[0053] Step 336: The area where the pixels in the binary image are 0 on the endoscopic image A and the endoscopic image B is used as the comparison area, feature points are extracted from the comparison area, and similarity detection is performed on the feature points of the two endoscopic images. If the similarity exceeds a threshold, the feature point is recorded as a reference point; if it does not exceed the threshold, the feature point is recorded as a non-reference point; until 10 reference points at least 1 mm apart are screened out.
[0054] In the scheme proposed by the present invention, before selecting the reference point, an optical flow vector analysis is first performed. This step can identify those areas that have not undergone obvious changes but have a trend of change. For example, some points may be about to be displaced due to muscle shaking, but in endoscopic images A and B, since the muscles have not yet completely shaken, the similarity of these points is still very high. If these points are used as reference points, they may disappear due to muscle movement in subsequent steps 4 and 5, thereby affecting the accuracy of image alignment. Therefore, the present invention effectively excludes many areas that may change due to muscle shaking through optical flow vector monitoring. In this way, the present scheme can more accurately find the reference points of the endoscopic image, thereby improving the accuracy of removing reflective points.
[0055] Step 4 includes the following steps:
[0056] Step 41: Move the light source horizontally relative to the lens and keep the lens fixed;
[0057] Step 42: Keep the focal length of the lens unchanged and collect an endoscopic image C;
[0058] Step 43: For the endoscopic image C, a plurality of recognition blocks are screened out in the manner of steps 22 to 24 to obtain a second recognition block set of the endoscopic image C.
[0059] In this scheme, by moving the light source horizontally, the bright spots in the endoscope image due to reflection will disappear or generate new reflection spots as the position of the light source changes. Using this feature, this scheme can effectively distinguish whether the white spots in the identification block are caused by reflection spots or abnormal spots caused by local tissue lesions.
[0060] Furthermore, step 5 includes the following steps:
[0061] Step 51: Move the light source to the initial position and move the lens horizontally;
[0062] Step 52: Keep the focal length of the lens unchanged and collect an endoscopic image D;
[0063] Step 53: For the endoscopic image D, a plurality of recognition blocks are screened out in the manner of steps 22 to 24 to obtain a third recognition block set of the endoscopic image D.
[0064] In this scheme, the light source is kept at a different moving lens, and the abnormal points caused by local tissue lesions will change relative to the center of the endoscopic image as the lens moves, while the abnormal water vapor on the lens moves synchronously with the movement of the lens, and the corresponding reflective points will also change due to the change in lens displacement. Therefore, in this scheme, by slightly moving the lens, the difference between the water vapor and reflective points on the lens and the abnormal points caused by local tissue lesions can be found.
[0065] Further, step 6 includes the following steps:
[0066] Step 61: Obtain reference point r in endoscopic image A i , i represents the index of the reference point in the endoscopic image;
[0067] Step 62: Align the endoscopic image C with the reference point r i The corresponding pixel C i Extract it and calculate the reference point r based on the SIFT algorithm i With pixel C iIf the similarity is greater than 90%, it means that the reference point is matched successfully;
[0068] Calculate all reference points r in turn i With pixel C i If more than 70% of the reference points are matched successfully, the endoscopic image A and the endoscopic image C are overlapped with each other based on the positions of the reference points; if more than 70% of the reference points are matched successfully, step 1 to step 5 are repeated;
[0069] Step 63: Align the endoscopic image D with the reference point r i The corresponding comparison point D i Extract it and calculate the reference point r based on the SIFT algorithm i Compared with point D i If the similarity is greater than 90%, it means that the reference point is matched successfully;
[0070] Calculate all reference points r in turn i Compared with point D i If more than 70% of the reference points are matched successfully, the endoscopic image A and the endoscopic image D are overlapped with each other based on the positions of the reference points; if more than 70% of the reference points are matched successfully, steps 1 to 5 are executed again;
[0071] Step 64: marking the recognition blocks in the first recognition block set, the second recognition block set and the third recognition block set on the endoscope A, and selecting the area included in the first recognition block set, the second recognition block set and the third recognition block set as the preliminary feature area;
[0072] Step 65: Input the preliminary feature region into the feature extraction model, and the feature extraction model extracts the abnormal staining region therefrom.
[0073] In the technical solution provided by the present invention, corresponding feature points are extracted based on the SIFT algorithm, and then the alignment of the endoscopic images is completed based on the similarity of the feature points. The feature points extracted by the SIFT algorithm are insensitive to lighting. When the position of the light source changes relative to the lens position, the feature points can still be matched, thereby accurately aligning the endoscopic images.
[0074] As a second aspect of the present invention, the present invention provides a system for extracting features of stained regions of vaginal endoscope images, comprising:
[0075] A lens, used for acquiring endoscopic images;
[0076] A light source, used to provide illumination;
[0077] The information processing device extracts abnormal staining areas using the aforementioned method for extracting features of stained areas in vaginal endoscope images.
[0078] The beneficial effects of the present invention are:
[0079] The technical solution provided by the present invention can accurately extract the abnormal staining area caused by the lesion of the patient, thereby increasing the accuracy of endoscopic examination. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, so that other features, purposes and advantages of the present invention become more obvious. The accompanying drawings and descriptions of the exemplary embodiments of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.
[0081] In addition, throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the components and elements are not necessarily drawn to scale.
[0082] Figure 1 Endoscopic image.
[0083] Figure 2 This is an endoscopic image where it is difficult to distinguish between reflective spots and abnormally stained areas.
[0084] Figure 3 Flow chart of the method for extracting features of stained regions in vaginal endoscopy images. DETAILED DESCRIPTION
[0085] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.
[0086] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.
[0087] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0088] The present invention proposes a method for extracting features of stained regions in endoscopic images, aiming to accurately identify abnormal stained regions in endoscopic images. The following is a brief introduction to the background and principle of the method: In endoscopic examination, especially vaginal endoscopy, dyeing liquid is injected into the patient's body to enhance the contrast between diseased tissue and non-lesioned tissue, thereby facilitating the location of the diseased part. Figure 1 As shown, the lesion area is clearly marked (in actual cases, the lesion area is usually smaller, Figure 1 Shown is an advanced case with overt symptoms).
[0089] However, in most cases, the lesions in the endoscopic images are very small or even non-existent, which requires the inspection process to accurately capture these subtle abnormalities. In order to improve the accuracy of the inspection, artificial intelligence algorithms are introduced to assist medical staff in extracting abnormal staining areas. However, the abnormal staining areas in the endoscopic images (usually white) are highly similar to the reflective spots (also white and bright) in color gamut. Figure 2 As shown, the irregular edge shape of the abnormally stained area poses a challenge to the artificial intelligence algorithm based on pixel analysis. Figure 2 In the figure, the circled parts are from top to bottom: the first circle is the reflective point, the second circle is the lesion area, and the third circle is the reflective point.
[0090] In response to this problem, the present invention proposes an innovative solution. This solution can effectively distinguish abnormal staining areas from reflective points by comprehensively using image processing technology and machine learning algorithms. Specific methods include (but are not limited to) adjusting the light source and lens position to eliminate reflective interference, using optical flow vector analysis to screen stable reference points, and using advanced image segmentation and feature extraction technology to accurately identify abnormal staining areas. Through these measures, the method of the present invention can significantly improve the accuracy of extracting abnormal staining areas in endoscopic images, providing strong support for the inspection work of medical staff.
[0091] refer to Figure 3 Embodiment 1: A method for extracting features of a stained region of a vaginal endoscope image, comprising the following steps:
[0092] Step 1: Acquire an endoscopic image of the target area.
[0093] Step 1 includes the following steps:
[0094] Step 11: Move the lens to a predetermined position so that the lens is aimed at the target area;
[0095] Step 12: Adjust the focal length of the lens to make the target area clear and place the target area in the center of the lens's field of view; record the position of the lens and the focal length of the lens at this time to obtain the position information of the lens;
[0096] Step 13: Acquire at least two endoscopic images A and B of the current target area, where there is a time interval between the endoscopic images A and B.
[0097] Step 1 mainly requires sending the endoscope into the patient's body to complete the acquisition of the required endoscopic images. When acquiring endoscopic image 1 and endoscopic image B, the endoscope needs to remain still to obtain image information of the same area. The target area in this solution is the area that needs to be detected. In actual inspections, the endoscope is generally not necessarily inserted after it is sent to a predetermined position, so this is also the case with this solution. After the endoscope enters the patient's body and acquires endoscopic image A and endoscopic image B, the endoscope remains still and then executes subsequent steps.
[0098] The focal length of the lens and the position information of the lens are used to restore the lens to its initial position. Therefore, the lens needs to have a certain degree of mobility. In this solution, the lens can be set in a movable manner, and then after the endoscope enters the patient's body, the lens can be moved slightly. The moving structure of this part can be completely restored by technicians in this field under known principles, so the specific structure of how to move the lens will not be further explained. The form of the corresponding lens position information will not be further explained. However, it is certain that the position information of the lens in this solution is required to be set relative to the center of the target area. If the position information of the lens is the same for two shots, it must be that the photos are taken of the same object.
[0099] Step 2: Based on the RGB screening condition, a number of recognition blocks are screened out from the endoscopic image to obtain a first recognition block set.
[0100] Step 2 includes the following steps:
[0101] Step 21: pre-configure RGB screening conditions, where the RGB screening conditions include a range of pixel values in a red channel, a range of pixel values in a green channel, and a range of pixel values in a blue channel.
[0102] refer to Figure 1 and Figure 2 In the stained endoscopic image, the biggest difference between normal tissue and abnormal stained tissue is the difference in color. Therefore, the range of pixel values in the red channel, the range of pixel values in the green channel, and the range of pixel values in the blue channel can be pre-set to complete the screening of normal tissue and abnormal stained tissue. The range of pixel values in the red channel, the range of pixel values in the green channel, and the range of pixel values in the blue channel are the RGB screening conditions. In this solution, because there is no need to distinguish between reflective points and abnormal lesion areas, the RGB screening conditions are very loose and cover the color gamut range of the two areas.
[0103] Step 22: Identify the pixel value of each channel in each pixel point in the endoscope image A, filter out the pixel points that meet the RGB screening conditions, and obtain a number of initial points;
[0104] Because each pixel has three corresponding channel values, this solution uses the previous RGB screening conditions for each pixel at the pixel level, so that all pixels that meet the RGB screening conditions can be obtained. The pixel here is the smallest area unit in the endoscope image.
[0105] Step 23: pre-set the minimum connection distance r, generate a circular area outward from each initial point with a radius of r, and if the circular areas of two or more initial points intersect, merge the circular areas of the initial points to obtain several initial areas;
[0106] Step 24: Filter out initial regions whose areas are smaller than a preset area threshold to obtain a number of recognition blocks, thereby obtaining a first recognition block set.
[0107] Because there may be many abnormal noise points in the endoscopic image, these noise points may represent the characteristics of reflective points or abnormal staining areas. For this reason, in this scheme, each initial point is expanded by r pixel grid radius. Therefore, in practice, if it is an isolated pixel point, it cannot be connected together after expansion, and it is smaller than the area threshold, and then it is lost. The pixels that are clustered together will be diffused by r pixel grids because the radius is r, so the circular area formed by these pixels will merge to form a larger initial area. Therefore, this scheme can filter out some relatively small abnormal points.
[0108] Step 3: Filter out several feature points that are irrelevant to light source changes from the endoscopic image and use the feature points as reference points.
[0109] Step 31: Acquire endoscopic image A and endoscopic image B;
[0110] Step 32: Based on the identification block obtained in step 2, the area where the identification block is located is marked on the endoscopic image A and the endoscopic image B respectively;
[0111] Step 33: Randomly select feature points with the same position on the endoscopic image A and the endoscopic image B from the area of the non-recognition block to perform similarity detection;
[0112] If the similarity exceeds the threshold, the feature point is recorded as a reference point;
[0113] If the threshold is not exceeded, the feature point is recorded as a non-reference point; until 10 reference points at least 1 mm apart are screened out.
[0114] Step 3 is mainly used to find the same feature points of endoscopic image A and endoscopic image B. These feature points are actually the points in the endoscopic image that will not be affected by reflective points and will not be affected by the movement of the patient's internal tissue. They generally belong to the middle area of the muscle tissue. After finding these feature points, these feature points can be used to overlap and correspond the endoscopic images obtained in different time periods. For this purpose, the method in step 33 is used to calculate the same feature points of endoscopic image A and endoscopic image B. Of course, in order to avoid the influence of reflective points, the reference points will be directly extracted from the non-identification block area in this solution.
[0115] The specific method of extracting reference points includes the following steps:
[0116] Step 33 includes the following steps:
[0117] Step 331: Arrange the endoscopic image A and the endoscopic image B in the order of acquisition, and overlap the endoscopic image A and the endoscopic image B. When the endoscopic image A and the endoscopic image B overlap, the corresponding position of each pixel in the target area is the same;
[0118] Step 332: remove the area where the identification block in the endoscopic image A and the endoscopic image B is located;
[0119] Step 333: grayscale processing the endoscopic image A and the endoscopic image B;
[0120] Step 334: for each pixel point (x, y) at the corresponding position in the endoscopic image A and the endoscopic image B, the following minimization energy function P (u, v) is constructed;
[0121] ;
[0122] ;
[0123] ;
[0124] Among them, Q1 and Q2 are the first intermediate parameter and the second intermediate parameter, u and v are the horizontal and vertical components of the optical flow vector, which represent the pixel point from I A To I B distance, σ is the smoothing weight, I A is the endoscopic image A, I B is the endoscopic image B, x and y are the coordinates of the pixel points;
[0125] , Respectively represent the partial derivatives of u in the x direction and y direction, which are used to measure the rate of change of u;
[0126] , Represent the partial derivatives of v in the x direction and y direction, respectively, which are used to measure the rate of change of v;
[0127] dx, dy: represent small changes in the x-direction and y-direction respectively, used for integral calculation;
[0128] ∂ represents the operator of partial derivatives;
[0129] is the double integral symbol;
[0130] Step 335: For each pixel, calculate the size U of its optical flow vector;
[0131] ; For each pixel point, if U is greater than α, the pixel point is recorded as 1, and if it is not greater than α, the pixel point is recorded as 0, so as to obtain a binary image representing the changes in the endoscopic image A and the endoscopic image B;
[0132] Step 336: The area where the pixels in the binary image are 0 on the endoscopic image A and the endoscopic image B is used as the comparison area, feature points are extracted from the comparison area, and similarity detection is performed on the feature points of the two endoscopic images. If the similarity exceeds a threshold, the feature point is recorded as a reference point; if it does not exceed the threshold, the feature point is recorded as a non-reference point; until 10 reference points at least 1 mm apart are screened out.
[0133] In this solution, when extracting reference points, the comparison is mainly based on the optical flow vector, and the optical flow vector is used to determine whether there is a difference between the two areas. If the optical flow vector of a certain area in the endoscopic image A and the endoscopic image B is small, the movement trend of the area is small, and it is more conducive to being a reference point. On the contrary, if the optical flow vector of a certain area is large, the area is not suitable for being a reference point. The reason why the present invention uses the optical flow vector to judge is mainly to solve the reason why only feature points cannot be accurately matched. In practice, when the internal tissue is active, it often moves the area corresponding to the feature point, but the feature point will not be lost. For example, the central area of a certain active muscle is in a state of motion when the muscle contracts and expands, so the position of the central area is different when the muscle expands and contracts, but the characteristics of the central area are the same. In this way, it is easy to overlap the central area of the image when contracting and the middle area of the image when expanding. This is not the case in this solution, but the existing optical flow vector is used to judge the displacement of the object, so many areas whose positions are easily offset during breathing can be screened out.
[0134] Furthermore, in step 336, the feature points are extracted based on the SIFT algorithm. In step 335, a binary image has been obtained, so it is only necessary to mask the area with a pixel of 1 in endoscopic image A and endoscopic image B, and extract feature points from the same position in the remaining area for comparison.
[0135] The feature points and reference points here are not pixels, but points extracted based on the SIFT algorithm that are not prone to change. In fact, they can be understood as a smaller area.
[0136] Step 4: Keep the lens fixed and change the position of the light source; re-collect the endoscopic image of the target area, and then filter out a number of recognition blocks from the endoscopic image based on the RGB screening condition to obtain a second recognition block set.
[0137] Step 4 includes the following steps:
[0138] Step 41: Move the light source horizontally relative to the lens and keep the lens fixed;
[0139] Step 42: Keep the focal length of the lens unchanged and collect an endoscopic image C;
[0140] Step 43: For the endoscopic image C, a plurality of recognition blocks are screened out in the manner of steps 22 to 24 to obtain a second recognition block set of the endoscopic image C.
[0141] Step 4 is to move the light source to obtain the second recognition block set, so the probability that the recognition block contained in both the first recognition block and the second recognition block is the lesion area is greater, while the probability that only one of the first recognition block and the second recognition block is the lesion area is smaller. Correspondingly, step 4 is only to obtain endoscopic image C of the same object as endoscopic image A and endoscopic image B, but with a different light source. Because the lens has not changed, the area in endoscopic image C is consistent with the part corresponding to the same area on endoscopic image A.
[0142] Step 5: restore the light source to its initial position, keep the light source fixed, and translate the lens; re-collect the endoscopic image of the target area, and then filter out several recognition blocks from the endoscopic image based on the RGB filtering condition to obtain a third recognition block set.
[0143] Step 5 includes the following steps:
[0144] Step 51: Move the light source to the initial position and move the lens horizontally.
[0145] In this step, the moving distance of the lens is less than 2 mm. In this solution, the moving distance is set to less than 2 mm, so that very small reflective points can be removed. The edge features of large reflective areas are obvious and can be directly judged manually during screening.
[0146] Step 52: Keep the focal length of the lens unchanged and collect an endoscopic image D;
[0147] Step 53: For the endoscopic image D, a plurality of recognition blocks are screened out in the manner of steps 22 to 24 to obtain a third recognition block set of the endoscopic image D.
[0148] In step 5 , the moving distance of the lens is small, but after the lens is moved, the endoscope image D and the endoscope images A to C need to be aligned with each other. The specific alignment step is performed in step 62 .
[0149] Step 6: Intersecting the first recognition block set, the second recognition block set and the third recognition block set to obtain a preliminary feature region;
[0150] The preliminary feature regions are input into the feature extraction model, and the feature extraction model extracts abnormal staining regions therefrom.
[0151] The feature extraction model in this solution is a common neural network model, which determines whether it is an abnormally stained area based on color features and texture features. Specifically, the recognition algorithm of the neural network model can use existing target recognition technology. The key point of this solution is to remove the influence of the noise feature of the reflective point, so the training requirements for the feature extraction model are lower.
[0152] Step 6 includes the following steps:
[0153] Step 61: Obtain reference point r in endoscopic image A i , i represents the index of the reference point in the endoscopic image;
[0154] Step 62: Align the endoscopic image C with the reference point r i The corresponding pixel C i Extract it and calculate the reference point r based on the SIFT algorithm i With pixel C i If the similarity is greater than 90%, it means that the reference point is matched successfully;
[0155] Calculate all reference points r in turn i With pixel C i If more than 70% of the reference points are matched successfully, the endoscopic image A and the endoscopic image C are overlapped with each other based on the positions of the reference points; if more than 70% of the reference points are matched successfully, step 1 to step 5 are repeated;
[0156] In step 62, the reference point r in the endoscopic image A is i It has been obtained in the previous solution and can be obtained directly. iLocated on the endoscopic image C, contrast point C i The position on the endoscopic image C and the reference point r i .
[0157] Step 63: Align the endoscopic image D with the reference point r i The corresponding comparison point D i Extract it and calculate the reference point r based on the SIFT algorithm i Compared with point D i If the similarity is greater than 90%, it means that the reference point is matched successfully;
[0158] Calculate all reference points r in turn i Compared with point D i If more than 70% of the reference points are matched successfully, the endoscopic image A and the endoscopic image D are overlapped with each other based on the positions of the reference points; if less than 70% of the reference points are matched successfully, steps 1 to 5 are executed again.
[0159] The similarity ratio of 90% and the reference point ratio of 70% in this solution are set according to the actual situation and can be adjusted according to needs in practice. This solution selects the above two values based on the results of a large number of experiments.
[0160] In step 63, the reference point r in the endoscopic image A is i It has been obtained in the previous solution and can be obtained directly. i Located on the endoscope image D, the endoscope image D is obtained by lens translation, so the comparison point D i The corresponding position can be found according to the translation relationship, and then the comparison point D can be obtained. i .
[0161] Step 62 and step 63 are the process of judging whether the images overlap. In practice, because the lens does not move when acquiring the endoscopic image C, the endoscopic images A, B, and C should not be aligned. In this solution, step 62 is actually set to further judge whether there is a large activity in the tissue. If there is a large activity, the endoscopic image A and the endoscopic image C cannot be aligned. At this time, steps 1 to 5 need to be re-executed. The same is true for step 63.
[0162] Step 64: marking the recognition blocks in the first recognition block set, the second recognition block set and the third recognition block set on the endoscope A, and selecting the area included in the first recognition block set, the second recognition block set and the third recognition block set as the preliminary feature area;
[0163] Step 65: Input the preliminary feature region into the feature extraction model, and the feature extraction model extracts the abnormal staining region therefrom.
[0164] The above descriptions are only some preferred embodiments of the present invention and the explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the embodiments of the present invention (but not limited to) by each other.
Claims
1. A method for extracting features of stained regions of vaginal endoscope images, characterized in that: include: Step 1: Acquire an endoscopic image of the target area; Step 2: Based on the RGB screening condition, a number of recognition blocks are screened out from the endoscopic image to obtain a first recognition block set; Step 3: Filter out several feature points that are irrelevant to the light source change from the endoscopic image and use the feature points as reference points; Step 4: Keep the lens fixed and change the position of the light source; re-collect the endoscopic image of the target area, and then filter out a number of recognition blocks from the endoscopic image based on the RGB screening condition to obtain a second recognition block set; Step 5: restore the light source to its initial position, keep the light source fixed, and translate the lens; recollect the endoscopic image of the target area, and then filter out a number of recognition blocks from the endoscopic image based on the RGB screening condition to obtain a third recognition block set; Step 6: Intersecting the first recognition block set, the second recognition block set and the third recognition block set to obtain a preliminary feature region; The preliminary feature regions are input into a feature extraction model, and the feature extraction model extracts abnormal staining regions therefrom; Step 1 includes the following steps: Step 11: Move the lens to a predetermined position so that the lens is aimed at the target area; Step 12: Adjust the focus of the lens to make the target area clear and in the center of the lens's field of view; Record the position of the lens and the focal length of the lens at this time to obtain the position information of the lens; Step 13: acquiring at least two endoscopic images A and B of the current target area, where there is a time interval between the endoscopic images A and B; Step 3 includes the following steps: Step 31: Acquire endoscopic image A and endoscopic image B; Step 32: Based on the identification block obtained in step 2, the area where the identification block is located is marked on the endoscopic image A and the endoscopic image B respectively; Step 33: Randomly select feature points with the same position on the endoscopic image A and the endoscopic image B from the area of the non-recognition block to perform similarity detection; If the similarity exceeds the threshold, the feature point is recorded as a reference point; If the threshold is not exceeded, the feature point is recorded as a non-reference point until 10 reference points at least 1 mm apart are selected; Step 33 includes the following steps: Step 331: Arrange the endoscopic image A and the endoscopic image B in the order of acquisition, and overlap the endoscopic image A and the endoscopic image B. When the endoscopic image A and the endoscopic image B overlap, the corresponding position of each pixel in the target area is the same; Step 332: remove the area where the identification block in the endoscopic image A and the endoscopic image B is located; Step 333: grayscale processing the endoscopic image A and the endoscopic image B; Step 334: for each pixel point (x, y) at the corresponding position in the endoscopic image A and the endoscopic image B, the following minimization energy function P (u, v) is constructed; ; ; ; Among them, Q1 and Q2 are the first intermediate parameter and the second intermediate parameter, u and v are the horizontal and vertical components of the optical flow vector, which represent the pixel point from I A To I B distance, σ is the smoothing weight, I A is the endoscopic image A, I B is the endoscopic image B, x and y are the coordinates of the pixel points; , Respectively represent the partial derivatives of u in the x direction and y direction, which are used to measure the rate of change of u; , Represent the partial derivatives of v in the x direction and y direction, respectively, which are used to measure the rate of change of v; dx, dy: represent small changes in the x-direction and y-direction respectively, used for integral calculation; ∂ represents the operator of partial derivatives; is the double integral symbol; Step 335: For each pixel, calculate the size U of its optical flow vector; , for each pixel point, if U is greater than α, the pixel point is recorded as 1, and if it is not greater than α, the pixel point is recorded as 0, so as to obtain a binary image representing the changes of the endoscopic image A and the endoscopic image B; Step 336: The area where the pixels in the binary image are 0 on the endoscopic image A and the endoscopic image B is used as the comparison area. Feature points are extracted from the comparison area, and the similarity of the feature points of the two endoscopic images is detected. If the similarity exceeds the threshold, the feature point is recorded as a reference point. If the threshold is not exceeded, the feature point is recorded as a non-reference point until 10 reference points at least 1 mm apart are selected; Step 2 includes the following steps: Step 21: pre-configure RGB screening conditions, where the RGB screening conditions include a range of pixel values in a red channel, a range of pixel values in a green channel, and a range of pixel values in a blue channel; Step 22: Identify the pixel value of each channel in each pixel point in the endoscope image A, filter out the pixel points that meet the RGB screening conditions, and obtain a number of initial points; Step 23: pre-set the minimum connection distance r, generate a circular area outward from each initial point with a radius of r, and if the circular areas of two or more initial points intersect, merge the circular areas of the initial points to obtain several initial areas; Step 24: Filter out initial regions whose areas are smaller than a preset area threshold to obtain a number of recognition blocks, thereby obtaining a first recognition block set.
2. The method for extracting the features of the stained area of a vaginal endoscope image according to claim 1, characterized in that: Step 4 includes the following steps: Step 41: Move the light source horizontally relative to the lens and keep the lens fixed; Step 42: Keep the focal length of the lens unchanged and collect an endoscopic image C; Step 43: For the endoscopic image C, a plurality of recognition blocks are screened out in the manner of steps 22 to 24 to obtain a second recognition block set of the endoscopic image C.
3. The method for extracting the features of the stained area of a vaginal endoscope image according to claim 2, characterized in that: Step 5 includes the following steps: Step 51: Move the light source to the initial position and move the lens horizontally; Step 52: Keep the focal length of the lens unchanged and collect an endoscopic image D; Step 53: For the endoscopic image D, a plurality of recognition blocks are screened out in the manner of steps 22 to 24 to obtain a third recognition block set of the endoscopic image D.
4. The method for extracting features of stained regions of vaginal endoscope images according to claim 3, characterized in that: Step 6 includes the following steps: Step 61: Obtain reference point r in endoscopic image A i , i represents the index of the reference point in the endoscopic image; Step 62: Align the endoscopic image C with the reference point r i The corresponding pixel C i Extract it and calculate the reference point r based on the SIFT algorithm i With pixel C i If the similarity is greater than 90%, it means that the reference point is matched successfully; Calculate all reference points r in turn i With pixel C i If more than 70% of the reference points are matched successfully, the endoscopic image A and the endoscopic image C are overlapped with each other based on the positions of the reference points; if more than 70% of the reference points are matched successfully, step 1 to step 5 are repeated; Step 63: Align the endoscopic image D with the reference point r i The corresponding comparison point D i Extract it and calculate the reference point r based on the SIFT algorithm i Compared with point D i If the similarity is greater than 90%, it means that the reference point is matched successfully; Calculate all reference points r in turn i Compared with point D i If more than 70% of the reference points are matched successfully, the endoscopic image A and the endoscopic image D are overlapped with each other based on the positions of the reference points; if more than 70% of the reference points are matched successfully, steps 1 to 5 are executed again; Step 64: marking the recognition blocks in the first recognition block set, the second recognition block set and the third recognition block set on the endoscope A, and selecting the area included in the first recognition block set, the second recognition block set and the third recognition block set as the preliminary feature area; Step 65: Input the preliminary feature region into the feature extraction model, and the feature extraction model extracts the abnormal staining region therefrom.
5. A system for extracting features of dyed regions of vaginal endoscope images: characterized in that: include A lens, used for acquiring endoscopic images; A light source, used to provide illumination; An information processing device extracts abnormal stained areas using the method for extracting features of stained areas in vaginal endoscope images according to any one of claims 1 to 4.
