Deep learning based dynamic mura removal method for fiberscope

Abstract

The application discloses a fiber endoscope dynamic moire elimination method based on deep learning, and the method comprises the following steps: inputting acquired detection video data into a preset scene construction model to construct a detection scene three-dimensional model containing an endoscope model; inputting acquired video frame images A i and simulation video frame images B i into a preset pixel point pixel value calculation model to determine standard pixel values of each pixel point; extracting a to-be-processed video frame image A j from the detection video data and inputting the to-be-processed video frame image A j into a preset moire elimination model to calculate a first pixel difference value of a to-be-processed pixel point m, determine an abnormal sub-region, perform pixel value compensation on abnormal pixel points in the abnormal sub-region, and update the to-be-processed video frame image A j according to the compensated pixel values. The application can accurately and quickly identify moire through a deep learning method, effectively and quickly eliminate moire in dynamic video frames, and improve video image display effects.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning. Background Technology

[0002] Fiber optic endoscopes are commonly used imaging devices in medical examinations such as breast and gastric examinations. Fiber optic endoscopes utilize tightly packed bundles of optical fibers to transmit images. Each fiber transmits the light information of one pixel, forming an image composed of numerous discrete light points (fiber cores) at the eyepiece or image sensor end. The cladding regions between the fiber cores form a dark grid.

[0003] When the regular mesh structure of a fiber bundle is close to or meets specific conditions of the pixel array spatial frequency of an image sensor (such as a CCD or CMOS), aliasing can cause interfering artifacts, typically wavy or mesh-like, on the image, known as moiré patterns. These artifacts are particularly noticeable when the object being imaged has fine textures or regular structures. Especially during dynamic endoscopic procedures (such as lens movement or organ peristalsis), the relative motion between the scene and the endoscope / sensor can cause dynamic changes in the moiré pattern (flickering, movement, deformation), resulting in color shifts and distortions in the video image, thus impairing its quality and clarity.

[0004] Traditional static image filtering methods (such as frequency domain filtering and spatial domain smoothing) are difficult to effectively handle such dynamically changing artifacts. For example, filters with fixed parameters cannot adapt to rapid changes in the position, intensity, and frequency of moiré patterns; over-filtering can lead to severe loss of image details (such as tissue texture, blood vessel edges, and lesion boundaries), affecting diagnosis.

[0005] Therefore, the existing technology has defects and urgently needs improvement. Summary of the Invention

[0006] In view of the above problems, the purpose of this invention is to provide a deep learning-based method for eliminating dynamic moiré patterns in fiber optic endoscopes, which can effectively and quickly eliminate moiré patterns and improve the display effect of video images.

[0007] The first aspect of this invention provides a deep learning-based method for eliminating dynamic moiré patterns in fiber optic endoscopes, comprising:

[0008] Acquire detection video data;

[0009] The detected video data is input into a preset scene construction model for analysis, and a three-dimensional model of the detection scene including the endoscope model is constructed.

[0010] Extract video frame image A from the detected video data. iThe video frame image A is obtained by detecting the 3D model of the scene. i Corresponding analog video frame image B i ;

[0011] The video frame image A i and the simulated video frame image B i The input is analyzed by a preset pixel value calculation model to determine the standard pixel value of each pixel.

[0012] Extract the video frame image A to be processed from the detected video data. j The video frame image A to be processed j The image A of the video frame to be processed is input into a preset moiré removal model. j The first pixel difference of the pixel point m to be processed is used to determine the abnormal sub-regions affected by moiré patterns. Pixel value compensation is performed on the abnormal pixels within the abnormal sub-regions. The video frame image A to be processed is then processed based on the compensated pixel values. j Update.

[0013] In this solution, the step of inputting the detected video data into a preset scene construction model for analysis, and constructing a three-dimensional model of the detection scene including an endoscope model, includes:

[0014] The detection scenario is determined by analyzing the detected video data.

[0015] Based on the detection scenario, retrieve the corresponding sample 3D model from the database to determine the 3D model of the detection scenario;

[0016] An endoscope model is generated in the 3D model of the detection scene.

[0017] In this solution, the step of extracting video frame image A from the detected video data i The video frame image A is obtained by detecting the 3D model of the scene. i Corresponding analog video frame image B i ,include:

[0018] According to the video frame image A i Update the 3D model of the detection scene;

[0019] According to the video frame image A i Determine the position and state of the endoscope model in the 3D model of the detection scene, and determine the video frame image A. i The three-dimensional coordinates of each pixel in the three-dimensional model of the detection scene;

[0020] Bind the display coordinates of each pixel to its corresponding 3D coordinates;

[0021] The video frame image A is obtained based on the position of the endoscope model in the 3D model of the detection scene. i Corresponding analog video frame image B i .

[0022] In this solution, the video frame image A is... i and the simulated video frame image B i The input is fed into a preset pixel value calculation model for analysis to determine the standard pixel value for each pixel, including:

[0023] Based on video frame image A i The display coordinates of pixel n are calculated in video frame image A. i The first pixel value a i(n) and in the simulated video frame image B i The second pixel value b i(n) The second pixel difference P i(n) ;

[0024]

[0025] Among them, Ra i(n) Ga i(n) and Ba i(n) The first pixel value a i(n) The pixel values ​​of the red, green, and blue channels, Rb i(n) , Gb i(n) and Bb i(n) The second pixel value b i(n) The pixel values ​​of the red, green, and blue channels;

[0026] When the second pixel difference P i(n) When the difference is less than the corresponding first preset pixel difference, for the video frame image A i Mark;

[0027] Based on a preset time interval t, statistical analysis of video frame images A i-t To video frame image A i The number of video frame image tags between;

[0028] When the number of video frame image markers exceeds a preset threshold, the weighted average pixel value of pixel n in each marked video frame image is calculated to determine the standard pixel value q of pixel n. i(n) ;

[0029] Conversely, based on a preset time interval t, the next set of video frame images is traversed until the standard pixel value q of pixel n is determined. i(n) ;

[0030] The standard pixel value q of pixel n i(n) Bind to the corresponding three-dimensional coordinates.

[0031] In this solution, the step of extracting the video frame image A to be processed from the detected video data is described. j The video frame image A to be processed j The image A of the video frame to be processed is input into a preset moiré removal model. j The first pixel difference of the pixel point m to be processed, and the abnormal sub-regions with moiré patterns are determined based on the first pixel difference, including:

[0032] The video frame image A to be processed is divided according to the preset partitioning specifications. j Divide into multiple sub-regions;

[0033] The region level of each sub-region is determined based on the display coordinates of the center pixel of the sub-region, and the number of pixels to be processed in each sub-region is determined based on the number of pixels selected corresponding to the region level.

[0034] Obtain the three-dimensional coordinates of the pixel m to be processed;

[0035] The standard pixel value q of the pixel to be processed is determined based on the three-dimensional coordinates of the pixel to be processed m. j(m) ;

[0036] Calculate the standard pixel value q of the pixel to be processed m. j(m) and the corresponding adjustment coefficient k m The product of these factors determines the third pixel value c of the pixel m to be processed. i(m) ;

[0037] Calculate the first pixel value a of the pixel m to be processed. j(m) and the third pixel value c i(m) First pixel difference Q j(m) ;

[0038]

[0039] Among them, Ra j(m) Ga j(m) and Ba j(m) The first pixel value a of pixel m is respectively j(m) The pixel values ​​of the red, green, and blue channels, Rc j(m) ,Gc j(m) and Bc j(m) The third pixel value c of pixel m i(m) The pixel values ​​of the red, green, and blue channels;

[0040] When the first pixel difference Qj(m) When the difference exceeds the second preset pixel difference threshold, the sub-region where the display coordinates of the pixel m to be processed are located is marked as an abnormal sub-region.

[0041] This plan also includes:

[0042] When the pixel to be processed m does not have a standard pixel value q j(m) At that time, calculate the second pixel difference P of the pixel point m to be processed. j(m) ;

[0043] When the second pixel difference P j(m) When the difference exceeds the third preset pixel difference threshold, the sub-region where the display coordinates of the pixel m to be processed are located is marked as an abnormal sub-region.

[0044] This plan also includes:

[0045] According to the pixel m in the video frame image A to be processed j The displayed coordinates determine the adjustment coefficient k m .

[0046] In this solution, the pixel value compensation for abnormal pixels within the abnormal sub-region includes:

[0047] Each pixel in the abnormal sub-region is analyzed sequentially, and pixels with a first pixel difference greater than a second preset pixel difference threshold are marked as abnormal pixels.

[0048] Obtain the neighboring pixels of the abnormal pixel and filter the neighboring abnormal pixels;

[0049] Calculate the weighted average pixel difference of each color channel between the first pixel value and the corresponding third pixel value of the remaining adjacent pixels, and determine the compensation coefficient of each color channel of the abnormal pixel.

[0050] The pixel value of each color channel in the second pixel value of the abnormal pixel is multiplied by the corresponding compensation coefficient of the color channel to determine the compensated pixel value of the abnormal pixel.

[0051] This plan also includes:

[0052] The abnormal pixels are sorted in descending order based on the number of adjacent pixels, and the pixel value of each abnormal pixel is compensated in descending order of the number of adjacent pixels.

[0053] This plan also includes:

[0054] After determining the pixel value after compensation for the abnormal pixel, the abnormal pixel is identified as the neighboring pixel of the abnormal pixel, and the number of neighboring pixels of the abnormal pixel is updated.

[0055] This invention discloses a deep learning-based method for eliminating dynamic moiré patterns in fiber optic endoscopes. The method includes: inputting acquired detection video data into a preset scene construction model to construct a 3D model of the detection scene containing an endoscope model; and inputting the acquired video frame image A... i and simulated video frame image B i Input the data into a preset pixel value calculation model to determine the standard pixel value of each pixel; extract the video frame image A to be processed from the detected video data. j Input the image into the preset moiré removal model to calculate the video frame image A to be processed. j The first pixel difference of the pixel m to be processed is used to determine the abnormal sub-region. Pixel value compensation is performed on the abnormal pixels within the abnormal sub-region. The video frame image A to be processed is then determined based on the compensated pixel values. j This invention utilizes deep learning methods to accurately and quickly identify moiré patterns and effectively eliminate them in dynamic video frames, thereby improving video image display quality. Attached Figure Description

[0056] Figure 1 A flowchart of the deep learning-based dynamic moiré pattern elimination method for fiber optic endoscopes provided by the present invention is shown.

[0057] Figure 2 The simulated video frame image B provided by the present invention is shown. i Flowchart for determining the method;

[0058] Figure 3 A flowchart of the standard pixel value calculation method provided by the present invention is shown. Detailed Implementation

[0059] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0060] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0061] Figure 1The flowchart of the deep learning-based method for eliminating dynamic moiré patterns in fiber optic endoscopes provided by this invention is shown.

[0062] like Figure 1 As shown, this invention discloses a deep learning-based method for eliminating dynamic moiré patterns in fiber optic endoscopes, comprising:

[0063] S102, acquire detection video data;

[0064] S104, Input the detected video data into the preset scene construction model for analysis, and construct a three-dimensional model of the detection scene including the endoscope model;

[0065] S106, Extract video frame image A from the detected video data. i Video frame image A is obtained by detecting the 3D model of the scene. i Corresponding analog video frame image B i ;

[0066] S108, transfer video frame image A i and simulated video frame image B i The input is analyzed by a preset pixel value calculation model to determine the standard pixel value of each pixel.

[0067] S110, Extract the video frame image A to be processed from the detected video data. j The video frame image A to be processed j Input the image into the preset moiré removal model to calculate the video frame image A to be processed. j The first pixel difference of the pixel m to be processed is used to determine the abnormal sub-regions affected by moiré patterns. Pixel value compensation is performed on the abnormal pixels within the abnormal sub-regions. The video frame image A to be processed is then determined based on the compensated pixel values. j Update.

[0068] According to an embodiment of the present invention, historical detection data (including historical detection image data and historical detection video data) during the endoscopic detection process is first collected. The collected historical detection data is then analyzed to construct a preset dynamic moiré pattern elimination model for fiber optic endoscopes. The preset dynamic moiré pattern elimination model for fiber optic endoscopes includes three sub-models: a preset scene construction model, a preset pixel value calculation model, and a preset moiré pattern elimination model.

[0069] First, the detection video data acquired by the endoscope is analyzed by constructing a model based on a pre-defined scene. Image features of the video frames in the detection video data are extracted, and a corresponding detection scene is selected to construct a 3D model of the detection scene. An endoscope model with the same parameters as the endoscope used in the detection scene is then generated within the 3D model of the detection scene. The endoscope model moves synchronously according to the movement parameters of the endoscope. Based on video frame image A in the detection video data... i The endoscope model is controlled to acquire images of the 3D model of the detection scene, obtaining images A from the video frame. i Corresponding analog video frame image B i .

[0070] Then, the detected video data is analyzed frame by frame using a preset pixel value calculation model, and the results are compared with video frame image A. i and simulated video frame image B i Determine video frame image A i There exists a standard pixel value for pixel n. The standard pixel value is q. i(n) The standard pixel value represents the actual pixel value unaffected by lighting conditions, shooting angle, etc. It includes standard pixel values ​​for the red, green, and blue channels, with each color channel's standard pixel value calculated separately. The standard pixel value of pixel n is then bound to its corresponding 3D coordinates.

[0071] Finally, the detected video data is analyzed frame by frame using a preset moiré removal model, and the video frame image A to be processed is... j The system is divided into multiple identical sub-regions. The region level of each sub-region is determined based on the pixel distance between the display coordinates of the center pixel and the origin of the two-dimensional coordinate system. A corresponding number of pixels to be selected is set for each region level. The pixel to be processed, *m*, is randomly selected within each sub-region based on this selection number. The standard pixel value *q* of the pixel to be processed is then retrieved from the database using the three-dimensional coordinates of *m*. j(m) The standard pixel values ​​of each color channel are multiplied by the corresponding adjustment coefficients to determine the third pixel value c of the pixel m to be processed. i(m) The first pixel value a of the pixel to be processed m. j(m) and the third pixel value c i(m) Calculate the first pixel difference Q j(m) When the first pixel difference Q of the pixel to be processed m j(m)When the pixel difference exceeds the second preset threshold set by the system, the sub-region containing the pixel m to be processed is identified as an abnormal sub-region. All pixels within the abnormal sub-region are analyzed, and pixels with a first pixel difference exceeding the second preset threshold are marked as abnormal pixels. Pixel value compensation is then performed on the abnormal pixel using its adjacent pixels (including the eight directions: top left, top, top right, right, bottom right, bottom, bottom left, and left). Furthermore, the system can process multiple sub-regions simultaneously using multi-threading to improve moiré pattern removal efficiency.

[0072] Before determining the standard pixel value for each pixel, a preset dynamic moiré pattern elimination model for fiber optic endoscopy is used to determine whether moiré pattern-affected areas exist in the video frame image by comparing the video frame image with the simulated video frame image. During the adjustment process, the standard pixel value for each pixel is gradually determined to improve the moiré pattern elimination effect.

[0073] According to an embodiment of the present invention, the detection video data is input into a preset scene construction model for analysis to construct a three-dimensional model of the detection scene including an endoscope model, including:

[0074] The detection scenario is determined by analyzing the detection video data.

[0075] Based on the detection scenario, retrieve the corresponding sample 3D model from the database to determine the 3D model of the detection scenario;

[0076] Generate an endoscope model from the 3D model of the detection scene.

[0077] It should be noted that the detection scenarios include those of the breast, esophagus, stomach, and bladder. The input detection video data is analyzed by building a model based on the preset scenarios, and image features of the video frames in the detection video data are extracted. The detection scenarios of the detection video data are determined by the extracted image features, and a corresponding three-dimensional model of the detection scenario is established. Based on the endoscope parameters used in the detection process, an endoscope model with the same endoscope parameters is generated in the three-dimensional model of the detection scenario.

[0078] Simultaneously, the movement parameters of the endoscope are collected, and the endoscope model is controlled to move synchronously in the 3D model of the detection scene based on the movement parameters of the endoscope.

[0079] Figure 2 The simulated video frame image B provided by the present invention is shown. i A flowchart for determining the method.

[0080] like Figure 2 As shown, according to an embodiment of the present invention, video frame image A is extracted from the detected video data. i Video frame image A is obtained by detecting the 3D model of the scene. iCorresponding analog video frame image B i ,include:

[0081] S202, based on video frame image A i Update the 3D model of the detection scene;

[0082] S204, based on video frame image A i Determine the position and state of the endoscope model in the 3D model of the detection scene, and determine the video frame image A. i The 3D coordinates of each pixel in the 3D model of the detection scene;

[0083] S206, bind the display coordinates of each pixel to its corresponding three-dimensional coordinates;

[0084] S208, Obtain video frame image A based on the position and state of the endoscope model in the 3D model of the detection scene. i Corresponding analog video frame image B i .

[0085] It should be noted that the 3D model of the detection scene is trained using sample data and typically represents a scene in a healthy state. However, in actual detection scenes, lesions such as inflammation, polyps, and nodules are often present, leading to differences between the actual detection scene and the 3D model. By extracting image features from video frame images and analyzing these features, the location of lesions is determined, and the 3D model of the detection scene is updated. Each pixel has unique 3D coordinates. Using the initial coordinates of the endoscope model in the 3D model of the detection scene as the origin, a 3D coordinate system is constructed. The 3D coordinates (including x, y, and z axes) of each pixel are determined based on its position in the 3D model of the detection scene. The display coordinates of the pixels are determined based on the video frame images in the display interface. The center point of the display interface is used as the origin to construct a 2D coordinate system. The display coordinates of each pixel are determined based on its position in the video frame images. Simultaneously, the 3D model of the detection scene is simulated and acquired based on the endoscope model to determine the video frame image A. i Corresponding analog video frame image B i .

[0086] Figure 3 A flowchart of the standard pixel value calculation method provided by the present invention is shown.

[0087] like Figure 3 As shown, according to an embodiment of the present invention, video frame image A i and simulated video frame image B i The input is fed into a preset pixel value calculation model for analysis to determine the standard pixel value for each pixel, including:

[0088] S302, based on video frame image A i The display coordinates of pixel n are calculated in video frame image A. i The first pixel value a i(n) and in the simulated video frame image B i The second pixel value b i(n) The second pixel difference P i(n) ;

[0089]

[0090] Among them, Ra i(n) Ga i(n) and Ba i(n) The first pixel value a i(n) The pixel values ​​of the red, green, and blue channels, Rb i(n) , Gb i(n) and Bb i(n) The second pixel value b i(n) The pixel values ​​of the red, green, and blue channels;

[0091] S304, when the second pixel difference P i(n) When the difference is less than the corresponding first preset pixel difference, for video frame image A i Mark;

[0092] S306, Based on a preset time interval t, statistically analyze video frame images A. i-t To video frame image A i The number of video frame image tags between;

[0093] S308, when the number of video frame image markers exceeds a preset threshold, calculate the weighted average pixel value of pixel n in each marked video frame image, and determine the standard pixel value q of pixel n. i(n) ;

[0094] S310, Conversely, based on the preset time interval t, the next set of video frame images is traversed until the standard pixel value q of pixel point n is determined. i(n) ;

[0095] S312, the standard pixel value q of pixel n i(n) Bind to the corresponding three-dimensional coordinates.

[0096] It should be noted that, because video frame image A i and simulated video frame image B i Corresponding to each other, pixel n in video frame image A i and simulated video frame image B i The display coordinates are the same. This is achieved by using pixel n in video frame image A. iThe first pixel value a i(n) and in the simulated video frame image B i The second pixel value b i(n) Calculate the second pixel difference P of pixel n i(n) The pixel value of a pixel includes the pixel values ​​of the red, green, and blue channels.

[0097] For pixel n, video frame images are selected based on a preset time interval t, and video frame image A is... i The corresponding time is determined as the end time of the selected time interval, and the start time of the selected time interval is determined by combining it with the preset time interval t. Video frame images A are then statistically analyzed. i-t To video frame image A i The number of video frame image markers is calculated and compared with a preset threshold. When the number of video frame image markers exceeds the preset threshold, a corresponding first weighting weight is determined based on the display coordinates of pixel n in the video frame image. The closer the display coordinates are to the origin of the two-dimensional coordinate system, the greater the corresponding first weighting weight. Pixel n is then placed in video frame image A... i The first pixel value a i(n) Multiply by the corresponding first weighted weight to determine the first pixel value a. i(n) The first weighted pixel value is used to calculate all first pixel values ​​a. i(n) The average of the first weighted pixel values ​​is used to determine the standard pixel value q for pixel n. i(n) The standard pixel value q of pixel n i(n) The standard pixel value q of pixel n is the actual pixel value unaffected by lighting conditions, shooting angle, etc. i(n) Standard pixel values ​​Rq including red, green, and blue channels i(n) Gq i(n) and Bq i(n) The standard pixel value for each color channel is calculated separately.

[0098] During the traversal, video frame image A is sequentially... i+1 To video frame image A i+z The corresponding time is determined by iterating through the selected time interval, counting the number of video frame image markers within the selected time interval, and then calculating the standard pixel value q of pixel n. i(n) Where z is an integer greater than 1. Furthermore, the iteration over pixel n also ends when pixel n disappears from the video frame image.

[0099] The first preset pixel difference, the preset time interval t, and the preset quantity threshold are all set by those skilled in the art according to actual needs.

[0100] According to an embodiment of the present invention, a video frame image A to be processed is extracted from the detected video data. j The video frame image A to be processed j Input the image into the preset moiré removal model to calculate the video frame image A to be processed. j The first pixel difference of the pixel m to be processed is used to determine the abnormal sub-regions affected by moiré patterns, including:

[0101] According to the preset partitioning specifications, the video frame image A to be processed is... j Divide into multiple sub-regions;

[0102] The region level of each sub-region is determined based on the display coordinates of the center pixel of the sub-region, and the number of pixels to be processed in each sub-region is determined based on the number of pixels selected corresponding to the region level.

[0103] Obtain the three-dimensional coordinates of the pixel m to be processed;

[0104] Determine the standard pixel value q of the pixel to be processed m based on its three-dimensional coordinates. j(m) ;

[0105] Calculate the standard pixel value q of the pixel to be processed m. j(m) and the corresponding adjustment coefficient k m The product of these factors determines the third pixel value c of the pixel m to be processed. i(m) ;

[0106] Calculate the first pixel value a of the pixel m to be processed. j(m) and the third pixel value c i(m) First pixel difference Q j(m) ;

[0107]

[0108] Among them, Ra j(m) Ga j(m) and Ba j(m) The first pixel value a of pixel m is respectively j(m) The pixel values ​​of the red, green, and blue channels, Rc j(m) ,Gc j(m) and Bc j(m) The third pixel value c of pixel m i(m) The pixel values ​​of the red, green, and blue channels;

[0109] When the first pixel difference Q j(m) When the difference exceeds the second preset pixel difference threshold, the sub-region where the display coordinates of the pixel m to be processed are located is marked as an abnormal sub-region.

[0110] It should be noted that the preset partitioning specifications are set by the system, and the video frame image A to be processed is partitioned according to the preset partitioning specifications. j The system divides the area into multiple sub-regions of identical size. The closer the display coordinates of the center pixel of a sub-region are to the origin of the two-dimensional coordinate system, the higher the region's level. The system sets a corresponding number of pixels to select for each region level; the higher the region level, the more pixels are selected. During pixel selection, pixels with standard pixel values ​​are preferred.

[0111] Through the video frame image A to be processed j Determine the corresponding analog video frame image B j By simulating video frame image B j Convert the display coordinates of the pixel m to be processed into three-dimensional coordinates, and determine the standard pixel value q bound to these three-dimensional coordinates. j(m) (Rq j(n) Gq j(n) Bq j(n) Due to factors such as lighting and shooting angle, there is a certain deviation between the theoretical pixel value and the standard pixel value of a pixel. The system analyzes historical detection data to determine the adjustment coefficient k for pixel m. m (Rk m Gk m Bk m ). (The standard pixel value q) j(m) The pixel values ​​of each channel are multiplied by their corresponding adjustment coefficients to determine the third pixel value c of pixel m. i(m) (Rc j(m) ,Gc j(m) Bc j(m) ).

[0112] Rc j(m) =Rk m ×Rq j(m) ;

[0113] Gc j(m) =Gk m ×Gq j(m) ;

[0114] Bc j(m) =Bk m ×Bq j(m) ;

[0115] The second preset pixel difference threshold is set by those skilled in the art according to actual needs.

[0116] According to an embodiment of the present invention, it further includes:

[0117] When the pixel to be processed m does not have a standard pixel value q j(m)At that time, calculate the second pixel difference P of the pixel to be processed m. j(m) ;

[0118] When the second pixel difference P j(m) When the difference exceeds the third preset pixel difference threshold, the sub-region where the display coordinates of the pixel m to be processed are located is marked as an abnormal sub-region.

[0119] It should be noted that the second pixel difference P of the pixel to be processed m j(m) The calculation method is expressed by the formula:

[0120]

[0121] Among them, Ra j(m) Ga j(m) and Ba j(m) The first pixel value a of the pixel m to be processed is respectively j(m) The pixel values ​​of the red, green, and blue channels, Rb j(m) , Gb j(m) and Bb j(m) The second pixel value b of the pixel to be processed m is respectively j(m) The pixel values ​​of the red, green, and blue channels.

[0122] The third preset pixel difference threshold is set by those skilled in the art according to actual needs.

[0123] According to an embodiment of the present invention, it further includes:

[0124] Based on pixel m in the video frame image A to be processed j The displayed coordinates determine the adjustment coefficient k m .

[0125] It should be noted that the adjustment coefficient k m Including the red channel adjustment factor Rk m Green channel adjustment coefficient Gk m And the blue channel adjustment factor Bk m The adjustment factor k of pixel m at different display coordinates is affected by factors such as lighting, shooting angle, and actual pixel value. m They are not necessarily the same. The system analyzes historical detection data and compares the first and third pixel values ​​of sample pixels under different display coordinates to determine the adjustment coefficient of the pixel under different display coordinates.

[0126] According to an embodiment of the present invention, pixel value compensation is performed on abnormal pixels within an abnormal sub-region, including:

[0127] Each pixel in the abnormal sub-region is analyzed sequentially, and pixels whose first pixel difference is greater than the second preset pixel difference threshold are marked as abnormal pixels.

[0128] Obtain the neighboring pixels of the abnormal pixel and filter the neighboring abnormal pixels;

[0129] Calculate the weighted average pixel difference of each color channel between the first pixel value and the corresponding third pixel value of the remaining adjacent pixels, and determine the compensation coefficient of each color channel of the abnormal pixel.

[0130] The pixel values ​​of each color channel in the second pixel value of the abnormal pixel are multiplied by the corresponding compensation coefficient of the color channel to determine the compensated pixel value of the abnormal pixel.

[0131] It should be noted that the compensation coefficients include red channel compensation coefficients, green channel compensation coefficients, and blue channel compensation coefficients. Each channel's compensation coefficient is calculated separately during the calculation process. Taking the red channel compensation coefficient calculation as an example, the pixel difference between the first pixel value and the corresponding third pixel value of an adjacent pixel in the red channel is multiplied by the corresponding second weighting weight to determine the red channel weighted pixel value of that adjacent pixel. The average of the red channel weighted pixel values ​​of all adjacent pixels is then calculated to determine the red channel compensation coefficient. Each channel corresponds to a different second weighting weight, which is set by the system based on the pixel values ​​of each channel in the third pixel value of the abnormal pixel; the larger the pixel value, the larger the corresponding second weighting weight.

[0132] The second preset pixel difference threshold is set by those skilled in the art according to actual needs.

[0133] According to an embodiment of the present invention, it further includes:

[0134] The abnormal pixels are sorted in descending order based on the number of adjacent pixels, and the pixel value of each abnormal pixel is compensated in descending order of the number of adjacent pixels.

[0135] It should be noted that during the pixel value compensation process for abnormal pixels, priority is given to selecting abnormal pixels with a larger number of adjacent pixels for pixel value compensation. When the number of adjacent pixels is the same, priority is given to processing the abnormal pixel closest to the origin of the two-dimensional coordinate system.

[0136] According to an embodiment of the present invention, it further includes:

[0137] Once the pixel value after compensation for the abnormal pixel is determined, the abnormal pixel is identified as the neighboring pixel of the abnormal pixel, and the number of neighboring pixels of the abnormal pixel is updated.

[0138] It should be noted that after determining the pixel value of the abnormal pixel after compensation, the abnormal pixel is regarded as the neighboring pixel of the abnormal pixel, and the pixel value of the neighboring abnormal pixel is compensated by the pixel value of the abnormal pixel after compensation.

[0139] All information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the "detection video data" and "sample 3D model" involved in this disclosure were obtained with full authorization.

[0140] This invention discloses a deep learning-based method for eliminating dynamic moiré patterns in fiber optic endoscopes. The method includes: inputting acquired detection video data into a preset scene construction model to construct a 3D model of the detection scene containing an endoscope model; and inputting the acquired video frame image A... i and simulated video frame image B i Input the data into a preset pixel value calculation model to determine the standard pixel value of each pixel; extract the video frame image A to be processed from the detected video data. j Input the image into the preset moiré removal model to calculate the video frame image A to be processed. j The first pixel difference of the pixel m to be processed is used to determine the abnormal sub-region. Pixel value compensation is performed on the abnormal pixels within the abnormal sub-region. The video frame image A to be processed is then determined based on the compensated pixel values. j This invention utilizes deep learning methods to accurately and quickly identify moiré patterns and effectively eliminate them in dynamic video frames, thereby improving video image display quality.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0142] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0144] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning, characterized in that, include: Acquire detection video data; The detected video data is input into a preset scene construction model for analysis, and a three-dimensional model of the detection scene including the endoscope model is constructed. Extract video frame image A from the detected video data. i The video frame image A is obtained by detecting the 3D model of the scene. i Corresponding analog video frame image B i ; The video frame image A i and the simulated video frame image B i The input is analyzed by a preset pixel value calculation model to determine the standard pixel value of each pixel. Extract the video frame image A to be processed from the detected video data. j The video frame image A to be processed j The image A of the video frame to be processed is input into a preset moiré removal model. j The first pixel difference of the pixel point m to be processed is used to determine the abnormal sub-regions affected by moiré patterns. Pixel value compensation is performed on the abnormal pixels within the abnormal sub-regions. The video frame image A to be processed is then processed based on the compensated pixel values. j Update; The step of inputting the detected video data into a preset scene construction model for analysis, and constructing a three-dimensional model of the detection scene including the endoscope model, includes: The detection scenario is determined by analyzing the detected video data. Based on the detection scenario, retrieve the corresponding sample 3D model from the database to determine the 3D model of the detection scenario; An endoscope model is generated in the three-dimensional model of the detection scene; The video frame image A is extracted from the detected video data. i The video frame image A is obtained by detecting the 3D model of the scene. i Corresponding analog video frame image B i ,include: According to the video frame image A i Update the 3D model of the detection scene; According to the video frame image A i Determine the position and state of the endoscope model in the 3D model of the detection scene, and determine video frame image A. i The three-dimensional coordinates of each pixel in the three-dimensional model of the detection scene; Bind the display coordinates of each pixel to its corresponding 3D coordinates; The video frame image A is obtained based on the position of the endoscope model in the 3D model of the detection scene. i Corresponding analog video frame image B i .

2. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 1, characterized in that, The video frame image A i and the simulated video frame image B i The input is fed into a preset pixel value calculation model for analysis to determine the standard pixel value for each pixel, including: Based on video frame image A i The display coordinates of pixel n are calculated in video frame image A. i The first pixel value a i(n) and in the simulated video frame image B i The second pixel value b i(n) The second pixel difference P i(n) ; ； Among them, Ra i(n) Ga i(n) and Ba i(n) The first pixel value a i(n) The pixel values ​​of the red, green, and blue channels, Rb i(n) , Gb i(n) and Bb i(n) The second pixel value b i(n) The pixel values ​​of the red, green, and blue channels; When the second pixel difference P i(n) When the difference is less than the corresponding first preset pixel difference, for the video frame image A i Mark; Based on a preset time interval t, statistical analysis of video frame images A i-t To video frame image A i The number of video frame image tags between; When the number of video frame image markers exceeds a preset threshold, the weighted average pixel value of pixel n in each marked video frame image is calculated to determine the standard pixel value q of pixel n. i(n) ; Conversely, based on a preset time interval t, the next set of video frame images is traversed until the standard pixel value q of pixel n is determined. i(n) ; The standard pixel value q of pixel n i(n) Bind to the corresponding three-dimensional coordinates.

3. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 1, characterized in that, The process of extracting video frame image A from the detected video data j The video frame image A to be processed j The image A of the video frame to be processed is input into a preset moiré removal model. j The first pixel difference of the pixel point m to be processed, and the abnormal sub-regions with moiré patterns are determined based on the first pixel difference, including: The video frame image A to be processed is divided according to the preset partitioning specifications. j Divide into multiple sub-regions; The region level of each sub-region is determined based on the display coordinates of the center pixel of the sub-region, and the number of pixels to be processed in each sub-region is determined based on the number of pixels selected corresponding to the region level. Obtain the three-dimensional coordinates of the pixel m to be processed; The standard pixel value q of the pixel to be processed is determined based on the three-dimensional coordinates of the pixel to be processed m. j(m) ; Calculate the standard pixel value q of the pixel to be processed m. j(m) and the corresponding adjustment coefficient k m The product of these factors determines the third pixel value c of the pixel m to be processed. i(m) ; Calculate the first pixel value a of the pixel m to be processed. j(m) and the third pixel value c i(m) First pixel difference Q j(m) ; ； Among them, Ra j(m) Ga j(m) and Ba j(m) The first pixel value a of pixel m is respectively j(m) The pixel values ​​of the red, green, and blue channels, Rc j(m) ,Gc j(m) and Bc j(m) The third pixel value c of pixel m i(m) The pixel values ​​of the red, green, and blue channels; When the first pixel difference Q j(m) When the difference exceeds the second preset pixel difference threshold, the sub-region where the display coordinates of the pixel m to be processed are located is marked as an abnormal sub-region.

4. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 3, characterized in that, Also includes: When the pixel to be processed m does not have a standard pixel value q j(m) At that time, calculate the second pixel difference P of the pixel point m to be processed. j(m) ; When the second pixel difference P j(m) When the difference exceeds the third preset pixel difference threshold, the sub-region where the display coordinates of the pixel m to be processed are located is marked as an abnormal sub-region.

5. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 3, characterized in that, Also includes: According to the pixel m in the video frame image A to be processed j The displayed coordinates determine the adjustment coefficient k m .

6. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 3, characterized in that, The step of compensating for abnormal pixels within the abnormal sub-region includes: Each pixel in the abnormal sub-region is analyzed sequentially, and pixels with a first pixel difference greater than a second preset pixel difference threshold are marked as abnormal pixels. Obtain the neighboring pixels of the abnormal pixel and filter the neighboring abnormal pixels; Calculate the weighted average pixel difference of each color channel between the first pixel value and the corresponding third pixel value of the remaining adjacent pixels, and determine the compensation coefficient of each color channel of the abnormal pixel. The pixel value of each color channel in the second pixel value of the abnormal pixel is multiplied by the corresponding compensation coefficient of the color channel to determine the compensated pixel value of the abnormal pixel.

7. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 6, characterized in that, Also includes: The abnormal pixels are sorted in descending order based on the number of adjacent pixels, and the pixel value of each abnormal pixel is compensated in descending order of the number of adjacent pixels.

8. The method for eliminating dynamic moiré patterns in fiber optic endoscopes based on deep learning according to claim 7, characterized in that, Also includes: After determining the pixel value after compensation for the abnormal pixel, the abnormal pixel is identified as the neighboring pixel of the abnormal pixel, and the number of neighboring pixels of the abnormal pixel is updated.

Images