Real-time positioning method, fusion display method and device for endoscope image area

Abstract

The invention discloses a real-time positioning method, a fusion display method and a device for an endoscope image area. The method comprises the following steps of: calculating a comprehensive standard deviation of the same pixel point of multiple frames of endoscope screen images in an endoscope screen image set in RGB (Red, Green, Blue) three channels; using a preset clustering method to determine the pixel point with the comprehensive standard deviation greater than a preset threshold as a target pixel point; image pixel points in the target pixel points are marked to the blank image, an initial reconstruction image is generated, a target connected domain with the largest area is obtained, and a bounding rectangle of the target connected domain is extracted to generate a corresponding endoscope area image. The method is suitable for various endoscopes of different types and different models in the market, pure endoscope area images can be accurately positioned by processing the endoscope screen images, and splicing fusion display is performed, so that interference information is eliminated for subsequent further processing or analysis, and the method is suitable for popularization and application. For example, an artificial intelligence auxiliary diagnosis module is used for disease diagnosis, and a more accurate diagnosis result is obtained.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging, and in particular to a real-time positioning method, a fusion display method and a device for an endoscopic image area. Background Art

[0002] Optical endoscopy plays an important role in the diagnosis of various types of cancer. Taking gastric cancer as an example, the current diagnosis rate of early gastric cancer in my country is low. Improving the ability to detect suspicious lesions of early gastric cancer and accurate biopsy is crucial for the diagnosis of early gastric cancer. With the rapid development of endoscopic diagnostic technology, it helps to better discover and evaluate gastric mucosal lesions and early gastric cancer, and the application of a variety of new endoscopic diagnostic technologies in clinical practice has significantly improved the detection rate of early gastric cancer and gastric mucosal lesions. At present, the clinical diagnosis of early gastric cancer has formed a relatively fixed endoscopic diagnostic model, that is, first use white light endoscopy to initially screen high-risk patients for gastric cancer, and then use relevant endoscopic diagnostic techniques to further diagnose suspicious lesions based on this, taking into account the size, depth, location and other factors of the lesions. Related endoscopes include chromoendoscopy, magnifying endoscopy, confocal laser endoscopy, cytoendoscopy, ultrasound endoscopy, etc. The working principles and applicable scenarios of these endoscopes are different: based on white light endoscopes, chromoendoscopy sprays coloring pigment dyes onto the local gastric mucosal surface to enhance tissue features, which can accurately determine the lesion margins and range of canceration of early gastric cancer; magnifying endoscopy adds a zoom lens to white light endoscopes to magnify the endoscopic image of mucosal tissue to hundreds of times, and observe the microvessels and glandular duct structures on the surface of the gastric mucosa; confocal laser microendoscopy uses laser irradiation of mucosal tissue to detect reflected fluorescence for high-resolution endoscopic histological evaluation, which can observe the microstructure of gastric mucosal cells, glands and microcirculation blood vessels; cell endoscopy can provide ultra-high magnification microscopic imaging, combined with dyes to perform in vivo staining examinations of tissue mucosa, and increase the contrast between mucosal tissue cells; ultrasound endoscopy can explore the tissue layers of lesions in real time, so that the surface image of the digestive tract and the ultrasound scanning image below the mucosa in the digestive tract can be presented at the same time.

[0003] At the same time, AI-assisted diagnosis technology has also been applied in clinical diagnosis. AI-assisted diagnosis technology identifies endoscopic image features based on default disease lesion features and outputs lesion diagnosis results. This diagnostic technology reduces the subjectivity of endoscopists' naked-eye judgment of endoscopic images and greatly improves the diagnosis rate of early gastric cancer. It can also greatly reduce the endoscopic operation time. There are multiple manufacturers of the same endoscope on the market, and the same manufacturer has different models of the same product. The screen image resolution, information displayed on the screen, and interface layout of these endoscopic products are different. The AI-assisted diagnosis module is usually located on a separate computer and receives these different types of endoscopic screen images through video signals. If the entire endoscopic screen image is used as the input for AI-assisted diagnosis, it may contain additional interference information, resulting in the output of incorrect diagnostic results and even medical accidents. Summary of the invention

[0004] The present invention provides a real-time positioning method, a fusion display method and a device for an endoscopic image area, which solve the above-mentioned technical problems.

[0005] A first aspect of an embodiment of the present invention provides a real-time positioning method for an endoscopic image area, comprising the following steps:

[0006] Step 1, obtaining an endoscopic screen image set, wherein the endoscopic screen image set includes a plurality of continuous frames of endoscopic screen images;

[0007] Step 2, calculating the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images;

[0008] Step 3, using a preset clustering method to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points;

[0009] Step 4: classify all target pixels to generate image pixels and information pixels, and mark the image pixels to a blank image to generate an initial reconstructed image;

[0010] Step 5, obtaining the target connected domain with the largest area in the initial reconstructed image, extracting the circumscribed rectangle of the target connected domain, and generating an endoscopic region image corresponding to the endoscopic screen image.

[0011] A second aspect of an embodiment of the present invention provides a method for fusion display of an endoscopic image region, further comprising a fusion display step, specifically:

[0012] Step 601, obtaining a plurality of endoscopic screen image sets of different types;

[0013] Step 602, executing steps 1 to 5 for each endoscopic screen image set to generate an endoscopic region image corresponding to each endoscopic screen image set;

[0014] Step 603: stitch and fuse the multiple endoscopic area images, and display the fusion result on the target device.

[0015] A third aspect of an embodiment of the present invention provides a real-time positioning device for an endoscopic image region, comprising an acquisition module, a calculation module, a clustering module, a marking module and a positioning module.

[0016] The acquisition module is used to acquire an endoscopic screen image set, wherein the endoscopic screen image set includes a plurality of continuous frames of endoscopic screen images;

[0017] The calculation module is used to calculate the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images;

[0018] The clustering module is used to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points using a preset clustering method;

[0019] The marking module is used to classify all target pixels, generate image pixels and information pixels, and mark the image pixels to a blank image to generate an initial reconstructed image;

[0020] The positioning module is used to obtain the target connected domain with the largest area in the initial reconstructed image, extract the circumscribed rectangle of the target connected domain, and generate an endoscopic area image corresponding to the endoscopic screen image.

[0021] A fourth aspect of an embodiment of the present invention provides a fusion display device for an endoscopic image area, further comprising a fusion display module.

[0022] The fusion display module is used to obtain multiple endoscopic screen image sets of different types, and sequentially drive the calculation module, the clustering module, the marking module and the positioning module to generate an endoscopic area image corresponding to each endoscopic screen image set, and display the fusion result on the target device after splicing and fusing the multiple endoscopic area images.

[0023] A fifth aspect of an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned real-time positioning method for the endoscopic image area.

[0024] A sixth aspect of an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for fusion display of endoscopic image areas.

[0025] A seventh aspect of an embodiment of the present invention provides a real-time positioning device for an endoscopic image area, comprising a computer-readable storage medium and a processor, wherein the processor implements the steps of the above-mentioned real-time positioning method for the endoscopic image area when executing a computer program on the computer-readable storage medium.

[0026] An eighth aspect of an embodiment of the present invention provides a fusion display device for an endoscopic image area, comprising a computer-readable storage medium and a processor, wherein the processor implements the steps of the above-mentioned fusion display method for the endoscopic image area when executing a computer program on the computer-readable storage medium.

[0027] The beneficial effects of the present invention are as follows: the present invention provides a real-time positioning method, a fusion display method and a device for an endoscopic image area, which are applicable to various endoscopic products of different types and models on the market. By processing the endoscopic screen images of these endoscopic products, pure endoscopic area images can be accurately positioned, and multiple endoscopic area images can be spliced ​​and fused to eliminate interference information and then displayed on a target device for subsequent further processing or analysis, such as using an artificial intelligence-assisted diagnosis module to diagnose diseases and obtain more accurate diagnosis results.

[0028] In order to make the above-mentioned objects, features and advantages of the invention more obvious and easy to understand, the preferred embodiments of the present invention are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a method for real-time positioning of an endoscopic image area provided by an embodiment;

[0031] Figure 2 is a flowchart of step 2 in a method for real-time positioning of an endoscopic image area provided by an embodiment;

[0032] Figure 3 is a flowchart of step 4 in a method for real-time positioning of an endoscopic image area provided in an embodiment;

[0033] Figure 4 is a flowchart of step 4 in a method for real-time positioning of an endoscopic image area provided in another embodiment;

[0034] Figure 5 is a flowchart of step 5 in a method for real-time positioning of an endoscopic image area provided in an embodiment;

[0035] Figure 6 is a flowchart of step 6 in a method for fusion display of an endoscopic image area provided in an embodiment;

[0036] Figure 7 is a schematic diagram of the effect of a method for fusion display of an endoscopic image area provided by an embodiment;

[0037] Figure 8 is a schematic structural diagram of a real-time positioning device for an endoscopic image area provided by an embodiment;

[0038] Fig. 9 is a schematic diagram of the structure of a fusion display device for an endoscopic image area provided by an embodiment;

[0039] Fig.10 is a schematic structural diagram of a real-time positioning device for an endoscopic image area provided by an embodiment;

[0040] Fig.11 The present invention is a schematic diagram of the structure of a fusion display device for an endoscopic image area provided in an embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the scope of protection of the present invention. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in the present invention do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0043] Figure 1 FIG. 1 is a flow chart of a real-time positioning method for an endoscopic image region provided in Example 1. Figure 1 As shown, the real-time positioning method includes the following steps:

[0044] Step 1, obtaining an endoscopic screen image set, wherein the endoscopic screen image set includes a plurality of continuous frames of endoscopic screen images;

[0045] Step 2, calculating the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images;

[0046] Step 3, using a preset clustering method to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points;

[0047] Step 4: classify all target pixels to generate image pixels and information pixels, and mark the image pixels to a blank image to generate an initial reconstructed image;

[0048] Step 5, obtaining the target connected domain with the largest area in the initial reconstructed image, extracting the circumscribed rectangle of the target connected domain, and generating an endoscopic region image corresponding to the endoscopic screen image.

[0049] The real-time positioning method of the endoscopic image area in the above embodiment is applicable to various endoscopes of different types and models on the market. By processing the endoscopic screen image, the pure endoscopic area image can be accurately positioned, thereby eliminating interference information for subsequent further processing or analysis, such as using an artificial intelligence-assisted diagnosis module to diagnose diseases and obtain more accurate diagnostic results.

[0050] Each step will be described below through specific examples.

[0051] It should be noted that the endoscope of the technical solution of the present invention is not limited to the gastric tumor inspection endoscope referred to in the background technology, but also includes endoscopes used in the inspection of other parts, such as the esophagus, stomach, intestines, bladder ureter, bronchi and lungs, and the endoscopic screen images generated by these endoscopes are all processing objects of the method of the present invention, that is, the endoscopic image areas of these endoscopic screen images can be located by the method of the present invention.

[0052] Specifically, clinical video data of the endoscope during the examination process is first collected, and then the clinical video data is analyzed, that is, an endoscope screen image set corresponding to the endoscope can be established, and each endoscope screen image set includes multiple frames of continuous endoscope screen images.

[0053] It is understandable that each frame of the endoscopic screen image is composed of a large number of pixels. The height of the endoscopic screen image can be recorded as H pixels and the width as W pixels. Because the endoscopic screen image is in color, each pixel contains color information. In digital image processing, color is usually represented by the RGB model. RGB represents the three color channels of red, green, and blue respectively. The value range of each channel is 0-255, where 0 means that the color is not included and 255 means that the color is the brightest. The technical solution of the present invention needs to calculate the standard deviation of the pixel value of each pixel (i, j) in the three RGB channels. Here, the standard deviation is a statistic that measures the degree of discreteness of the image color distribution, which is used to reflect the degree of deviation of the color value of each pixel in each frame of the image from the mean. For example, the standard deviation of the R channel pixel value of the pixel point (i, j) in the kth frame of the endoscopic screen image refers to the degree of deviation of the red channel value of the pixel point (i, j) in the kth frame of the image from the red channel average value of the pixel point (i, j).

[0054] In a specific embodiment, Figure 2 As shown, step 2 calculates the comprehensive standard deviation of the same pixel in the multi-frame endoscopic screen image in the RGB three channels, which specifically includes the following steps:

[0055] Step 201, respectively collect the R channel pixel value of each pixel point in each frame of the endoscope screen image G channel pixel value and the B channel pixel value k represents the kth frame of the endoscopic screen image, and the endoscopic screen image includes an image area and a text area. Here, the three-channel pixel value of each pixel point can be obtained using an image processing library (such as OpenCV), and the specific method is not described in detail here.

[0056] Step 202, calculate the standard deviation S of the R channel pixel value of the same pixel point in multiple frames of endoscopic screen images R (i, j), G channel pixel value standard deviation S G (i, j) and the standard deviation S of the B channel pixel value B (i,j) is:

[0057]

[0058]

[0059]

[0060] std{·} is a function for calculating the standard deviation, K represents the number of image frames of the endoscopic screen image set, (i, j) represents the pixel points of the endoscopic screen image, and k represents the kth frame image.

[0061] Finally, step 203 is performed to calculate the comprehensive standard deviation S(i, j) of each pixel in the RGB three channels as follows:

[0062] S(i,j)=max(S R (i,j),S G (i,j),S B (i,j)), that is, the maximum value of the standard deviations of the pixel values ​​of the three channels is used to represent the comprehensive standard deviation of the pixel.

[0063] In a specific embodiment, step 3 uses a clustering method, such as a K-means clustering method, a density peak clustering method, etc., to cluster all pixels into two categories according to the comprehensive standard deviation of all pixels, namely, a category with a higher comprehensive standard deviation and a category with a lower comprehensive standard deviation, the former indicating that there is a large change in the pixel value on the endoscopic screen image, and the latter indicating that there is a small change in the pixel value on the endoscopic screen image. Specifically, step 3 uses a clustering method such as a K-means clustering method, a density peak clustering method, etc. to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points, that is, the pixel points whose pixel values ​​have a large change are used as target pixel points that may be images or texts in this technical solution.

[0064] Then, step 4 is performed to classify the target pixel points. Figure 3 As shown, all target pixels are classified to generate image pixels and information pixels, specifically:

[0065] Step 401: grayscale K frames of endoscopic screen images respectively, and calculate the pixel grayscale value G of each target pixel point (m, n) in each frame of endoscopic screen image. k (m,n);

[0066] Step 402, generating a corresponding grayscale value histogram H(m,n) according to the grayscale value of each target pixel point (m,n) on all grayscale images;

[0067] Step 403, using a preset window function to process the gray value histogram, the preset window function Win = ε (p-31) - ε (p-224), where ε is a step function and p is a gray level;

[0068] Step 404, calculate the gray level probability distribution of the target pixel point (m, n) according to the processed gray value histogram, and calculate the information entropy E(m, n) of each target pixel point (m, n) as follows:

[0069]

[0070] in p l (m,n) represents the probability that the gray level of the target pixel (m,n) is l, H l (m,n) indicates the number of times the gray level of the target pixel (m,n) is l;

[0071] Step 405, determine whether the information entropy is greater than a first preset value, if so, determine the corresponding target pixel as an image pixel, otherwise, determine it as an information pixel. In a specific embodiment, the first preset value can be 3, which can obtain better image and text classification results.

[0072] The above embodiment determines whether the target pixel is an image pixel by calculating the information entropy of the target pixel. In image processing, the information entropy of a pixel can be understood as a measure of the amount of information contained in the grayscale value of the pixel. For an image, the higher the information entropy, the greater the amount of information contained in the image, and the richer the details of the image. Therefore, in this embodiment, when the information entropy of the target pixel is greater than the first preset value, the target pixel can be determined as an image pixel.

[0073] It should be noted that, before calculating the information entropy of the target pixel, the present embodiment first uses a window function based on a step function to process the gray value histogram of the target pixel, wherein the preset window function Win=ε(p-31)-ε(p-224), wherein ε is a unit step function, ε(p-31) indicates that when the gray level is greater than 31, the step function value is 1, and when the gray level is less than or equal to 31, the step function value is 0; and ε(p-224) indicates that when the gray level is greater than 224, the function value is 1, and when the gray level is less than or equal to 224, the function value is 0. Thus, by applying the above window function to the gray value histogram, a clipping region can be defined, and the positions of the outer circle 1 / 8*256=32 are not used, and only the positions of the middle 224 are retained.

[0074] In another embodiment, if Figure 4 As shown, all target pixels are classified to generate image pixels and information pixels, specifically:

[0075] Step 406, obtaining the R channel pixel value of each target pixel point (m, n) in each frame of the endoscopic screen image G channel pixel value and the B channel pixel value k represents the kth frame image;

[0076] Step 407, respectively calculate the difference of the pixel values ​​of each channel corresponding to each target pixel point (m, n) in each frame of the image, and obtain the maximum difference D k (m,n) is:

[0077]

[0078] Step 408, obtaining the maximum difference of the target pixel point (m, n) of each frame image, and calculating the average value D(m, n) of the maximum difference corresponding to all frames:

[0079] D(m,n)=mean{D k (m,n),k=1,…,K}where mean{·} is the function for calculating the mean value;

[0080] Step 409, determine whether the average value is not greater than a second preset value, if so, determine the corresponding target pixel point as an information pixel point, otherwise, determine it as an image pixel point. In one embodiment, the second preset value can be 8, so that the target pixel value can be classified and a better classification result can be obtained.

[0081] It can be understood that after the image pixels are classified, the image pixels can be marked to a blank image to generate an initial reconstructed image, such as Figure 5 As shown, specifically:

[0082] Step 501, creating a blank image of the same size as the endoscope screen image, and marking all image pixels on the blank image to generate an initial reconstructed image;

[0083] Step 502: Perform a preset opening operation on the initial reconstructed image, and select a 5*5 square convolution kernel to eliminate noise of the initial reconstructed image.

[0084] Finally, the connected domains of the initial reconstructed image after noise elimination are counted, the target connected domain with the largest area is found, and its bounding rectangle is extracted. The bounding rectangle is the image of the endoscope area. For an endoscope with a constant screen layout, this method only needs to be run once to locate the image area. If the screen layout changes, this method is continuously run at regular intervals to locate the image area in real time.

[0085] Another embodiment of the present invention also provides a fusion display method for an endoscopic image area, which adds a fusion display step on the basis of the aforementioned real-time positioning method for the endoscopic image area. The fusion display method can place the captured images of various types of endoscopic areas on a target device, such as a display, thereby eliminating interference information for subsequent further processing or analysis, such as using an artificial intelligence-assisted diagnosis module to diagnose diseases and obtain more accurate diagnosis results.

[0086] like Figure 6 As shown, the added fusion display steps specifically include:

[0087] Step 601, obtaining a plurality of endoscopic screen image sets of different types;

[0088] Step 602, executing steps 1 to 5 for each endoscopic screen image set to generate an endoscopic region image corresponding to each endoscopic screen image set;

[0089] Step 603: stitch and fuse the multiple endoscopic area images, and display the fusion result on the target device.

[0090] The different types of multiple endoscopic screen image sets described in the above embodiments include endoscopic screen image sets corresponding to white light endoscopy, staining endoscopy, magnifying endoscopy, confocal endoscopy, cytoendoscopy and / or ultrasonic endoscopy, etc. Figure 7 is a schematic diagram of the effect of the fusion display method provided by an embodiment, such as Figure 7As shown, after locating the endoscopic area of ​​the white light endoscope screen image and the confocal endoscope screen image respectively, the two endoscopic area images are spliced ​​and fused. During the splicing and fusion process, the image combination, arrangement, and size display method are adjusted to keep the display effects of the two endoscopic area images consistent, so that two endoscopic examination images (for example, for the same patient part) can be displayed simultaneously on the display screen of the target device to obtain more accurate diagnostic results.

[0091] It should be noted that the above-mentioned endoscope types are only examples, and other endoscope types that are not mentioned in this application document but will be used in the inspection are all within the protection scope of this application.

[0092] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0093] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned real-time positioning method or fusion display method of the endoscopic image area.

[0094] Figure 8 FIG. 1 is a schematic diagram of the structure of a real-time positioning device for an endoscopic image region (hereinafter referred to as a real-time positioning device) provided by an embodiment. Figure 8 As shown, it includes an acquisition module 100, a calculation module 200, a clustering module 300, a marking module 400 and a positioning module 500.

[0095] The acquisition module 100 is used to acquire an endoscopic screen image set, wherein the endoscopic screen image set includes a plurality of continuous frames of endoscopic screen images;

[0096] The calculation module 200 is used to calculate the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images;

[0097] The clustering module 300 is used to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points using a preset clustering method;

[0098] The marking module 400 is used to classify all target pixels, generate image pixels and information pixels, and mark the image pixels to a blank image to generate an initial reconstructed image;

[0099] The positioning module 500 is used to obtain the target connected domain with the largest area in the initial reconstructed image, extract the circumscribed rectangle of the target connected domain, and generate an endoscopic area image corresponding to the endoscopic screen image.

[0100] The real-time positioning device of the endoscopic image area in the above embodiment is applicable to various endoscopes of different types and models on the market. By processing the endoscopic screen image, the pure endoscopic area image can be accurately positioned, thereby eliminating interference information for subsequent further processing or analysis, such as using an artificial intelligence-assisted diagnosis module to diagnose diseases and obtain more accurate diagnostic results.

[0101] In a preferred embodiment, the calculation module 200 specifically includes:

[0102] The first acquisition unit is used to respectively acquire the R channel pixel value of each pixel point in each frame of the endoscope screen image G channel pixel value and the B channel pixel value The endoscope screen image includes an image area and a text area;

[0103] The first calculation unit is used to calculate the standard deviation S of the R channel pixel value of the same pixel point in multiple frames of endoscopic screen images. R (i, j), G channel pixel value standard deviation S G (i, j) and the standard deviation S of the B channel pixel value B (i,j);

[0104] The second calculation unit is used to calculate the comprehensive standard deviation S(i, j) of each pixel in the RGB three channels as follows:

[0105] S(i,j)=max(S R (i,j),S G (i,j),S B (i,j));

[0106] Wherein, std{·} is a function for calculating the standard deviation, max() is a function for calculating the maximum value, K represents the number of image frames of the endoscopic screen image set, (i, j) represents the pixel points of the endoscopic screen image, and k represents the kth frame image.

[0107] In a preferred embodiment, the clustering module 300 includes:

[0108] The grayscale unit is used to grayscale K frames of endoscopic screen images respectively and calculate the pixel grayscale value G of each target pixel point (m, n) in each frame of endoscopic screen image. k (m,n);

[0109] A histogram generating unit, used for generating a corresponding grayscale value histogram H(m,n) according to the grayscale value of each target pixel point (m,n) on all grayscale images;

[0110] A first image processing unit, configured to process the gray value histogram using a preset window function, wherein the preset window function is Win=ε(p-31)-ε(p-224), wherein ε is a step function and p is a gray level;

[0111] The third calculation unit is used to calculate the gray level probability distribution of the target pixel point (m, n) according to the processed gray value histogram, and calculate the information entropy E(m, n) of each target pixel point (m, n) as follows:

[0112]

[0113] in p l (m,n) represents the probability that the gray level of the target pixel (m,n) is l, H l (m,n) indicates the number of times the gray level of the target pixel (m,n) is l;

[0114] The first judgment unit is used to judge whether the information entropy is greater than a first preset value. If so, the corresponding target pixel point is judged as an image pixel point, otherwise, it is judged as an information pixel point.

[0115] In another preferred embodiment, the clustering module 300 may further include:

[0116] The second acquisition unit is used to obtain the R channel pixel value of each target pixel point (m, n) in each frame of the endoscopic screen image G channel pixel value and the B channel pixel value k represents the kth frame image;

[0117] The fourth calculation unit is used to calculate the difference of the pixel values ​​of each channel corresponding to each target pixel point (m, n) in each frame of the image, and obtain the maximum difference D k (m,n) is:

[0118]

[0119] The fifth calculation unit is used to calculate the average value D(m,n) of the maximum difference corresponding to each target pixel point (m,n):

[0120] D(m,n)=mean{D k (m,n),k=1,…,K}where mean{·} is the function for calculating the mean value;

[0121] The second judgment unit is used to judge whether the average value is not greater than a second preset value. If so, the corresponding target pixel point is judged as an information pixel point, otherwise, it is judged as an image pixel point.

[0122] In a preferred embodiment, the marking module 400 specifically includes:

[0123] A blank image establishing unit, used for establishing a blank image of the same size as the endoscope screen image, and marking all image pixels to the blank image to generate an initial reconstructed image;

[0124] The second image processing unit is used to perform a preset opening operation on the initial reconstructed image to eliminate noise of the initial reconstructed image.

[0125] like Fig. 9 As shown, another embodiment of the present invention further provides a fusion display device for an endoscopic image area (hereinafter referred to as a fusion display device), which adds a fusion display module 600 on the basis of the real-time positioning device shown in the above embodiment, and the fusion display module 600 is used to obtain a plurality of endoscopic screen image sets of different types, and sequentially drive the calculation module, the clustering module, the marking module and the positioning module to generate an endoscopic area image corresponding to each endoscopic screen image set, and display the fusion result on the target device after splicing and fusing the plurality of endoscopic area images. The fusion display device of this embodiment can place the captured endoscopic area images of various types on a target device, such as a display, thereby eliminating interference information for subsequent further processing or analysis, such as using an artificial intelligence-assisted diagnosis module for disease diagnosis, to obtain a more accurate diagnosis result.

[0126] Fig.10 FIG. 1 is a schematic diagram of a real-time positioning device for an endoscopic image region provided by an embodiment of the present invention (hereinafter referred to as a real-time positioning device). Fig.10 As shown, the real-time positioning device 8 of this embodiment includes: a processor 80, a readable storage medium 81, and a computer program 82 stored in the readable storage medium 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned various method embodiments are implemented, for example Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of the modules in the above-mentioned device embodiments are realized, for example Figure 8 Functionality of the modules shown.

[0127] Fig.11 FIG. 1 is a schematic diagram of the structure of a fusion display device for an endoscopic image area (hereinafter referred to as a fusion display device) provided by an embodiment of the present invention. Fig.11As shown, the fusion display device 9 of this embodiment includes: a processor 90, a readable storage medium 91, and a computer program 92 stored in the readable storage medium 91 and executable on the processor 90. When the processor 90 executes the computer program 92, the steps in the above-mentioned various method embodiments are implemented, for example Figure 6 Alternatively, when the processor 90 executes the computer program 92, the functions of the modules in the above-mentioned device embodiments are realized, for example Fig. 9 Functionality of the modules shown.

[0128] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0131] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.

Claims

1. A real-time positioning method for an endoscopic image area, characterized in that: The following steps are involved: Step 1, obtaining an endoscopic screen image set, wherein the endoscopic screen image set includes a plurality of continuous frames of endoscopic screen images; Step 2, calculating the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images; Step 3, using a preset clustering method to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points; Step 4: classify all target pixels to generate image pixels and information pixels, and mark the image pixels to a blank image to generate an initial reconstructed image; Step 5, obtaining the target connected domain with the largest area in the initial reconstructed image, extracting the circumscribed rectangle of the target connected domain, and generating an endoscopic region image corresponding to the endoscopic screen image.

2. The real-time positioning method of the endoscopic image area according to claim 1, characterized in that: Calculating the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images specifically includes the following steps: Step 201, respectively collect the R channel pixel value of each pixel point in each frame of the endoscope screen image G channel pixel value and the B channel pixel value The endoscope screen image includes an image area and a text area; Step 202, calculate the standard deviation S of the R channel pixel value of the same pixel point in multiple frames of endoscopic screen images R (i, j), G channel pixel value standard deviation S G (i, j) and the standard deviation S of the B channel pixel value B (i,j) is: Step 203, calculate the comprehensive standard deviation S(i,j) of each pixel in the RGB three channels as follows: S(i,j)=max(S R (i,j),S G (i,j),S B (i,j)); Wherein, std{·} is a function for calculating the standard deviation, max() is a function for calculating the maximum value, K represents the number of image frames of the endoscopic screen image set, (i, j) represents the pixel points of the endoscopic screen image, and k represents the kth frame image.

3. The real-time positioning method of the endoscopic image area according to claim 2, characterized in that: Classify all target pixels and generate image pixels and information pixels, specifically: Step 401: grayscale K frames of endoscopic screen images respectively, and calculate the pixel grayscale value G of each target pixel point (m, n) in each frame of endoscopic screen image. k (m,n); Step 402, generating a corresponding grayscale value histogram H(m,n) according to the grayscale value of each target pixel point (m,n) on all grayscale images; Step 403, using a preset window function to process the gray value histogram, the preset window function is Win = ε (p-31) - ε (p-224), where ε is a step function and p is a gray level; Step 404, calculate the gray level probability distribution of the target pixel point (m, n) according to the processed gray value histogram, and calculate the information entropy E(m, n) of each target pixel point (m, n) as follows: in p l (m,n) represents the probability that the gray level of the target pixel (m,n) is l, H l (m,n) indicates the number of times the gray level of the target pixel (m,n) is l; Step 405, determining whether the information entropy is greater than a first preset value, if so, determining the corresponding target pixel point as an image pixel point, otherwise, determining it as an information pixel point.

4. The real-time positioning method of the endoscopic image area according to claim 2, characterized in that: Classify all target pixels and generate image pixels and information pixels, specifically: Step 406, obtaining the R channel pixel value of each target pixel point (m, n) in each frame of the endoscopic screen image G channel pixel value and the B channel pixel value k represents the kth frame image; Step 407, respectively calculate the difference of the pixel values ​​of each channel corresponding to each target pixel point (m, n) in each frame of the image, and obtain the maximum difference D k (m,n) is: Step 408, calculate the average value D(m,n) of the maximum difference corresponding to each target pixel point (m,n): D(m,n)=mean{D k (m,n),k=1,…,K}, where mean{·} is the function for calculating the mean value; Step 409, determining whether the average value is not greater than a second preset value, if so, determining the corresponding target pixel point as an information pixel point, otherwise, determining it as an image pixel point.

5. The real-time positioning method of the endoscopic image area according to any one of claims 1 to 4, characterized in that: The image pixels are marked on a blank image to generate an initial reconstructed image, specifically: Step 501, creating a blank image of the same size as the endoscope screen image, and marking all image pixels on the blank image to generate an initial reconstructed image; Step 502: Perform a preset opening operation on the initial reconstructed image to eliminate noise of the initial reconstructed image.

6. A method for fusion display of an endoscopic image area, based on the real-time positioning method according to any one of claims 1 to 5, characterized in that: It also includes the fusion display steps, specifically: Step 601, obtaining a plurality of endoscopic screen image sets of different types; Step 602, executing steps 1 to 5 for each endoscopic screen image set to generate an endoscopic region image corresponding to each endoscopic screen image set; Step 603: stitch and fuse the multiple endoscopic area images, and display the fusion result on the target device.

7. The method for fusion display of endoscopic image areas according to claim 6, characterized in that: The multiple endoscopic screen image sets of different types include endoscopic screen image sets corresponding to white light endoscopy, chromoendoscopy, magnifying endoscopy, confocal endoscopy, cytoendoscopy and / or ultrasonic endoscopy, respectively.

8. A real-time positioning device for an endoscopic image area, characterized in that: It includes acquisition module, calculation module, clustering module, marking module and positioning module. The acquisition module is used to acquire an endoscopic screen image set, wherein the endoscopic screen image set includes a plurality of continuous frames of endoscopic screen images; The calculation module is used to calculate the comprehensive standard deviation of the same pixel in the RGB three channels in multiple frames of endoscopic screen images; The clustering module is used to determine the pixel points whose comprehensive standard deviation is greater than a preset threshold as target pixel points using a preset clustering method; The marking module is used to classify all target pixels, generate image pixels and information pixels, and mark the image pixels to a blank image to generate an initial reconstructed image; The positioning module is used to obtain the target connected domain with the largest area in the initial reconstructed image, extract the circumscribed rectangle of the target connected domain, and generate an endoscopic area image corresponding to the endoscopic screen image.

9. A fusion display device for an endoscopic image area, based on the real-time positioning device according to claim 8, characterized in that: Also includes a fusion display module, The fusion display module is used to obtain multiple endoscopic screen image sets of different types, and sequentially drive the calculation module, the clustering module, the marking module and the positioning module to generate an endoscopic area image corresponding to each endoscopic screen image set, and display the fusion result on the target device after splicing and fusing the multiple endoscopic area images.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the real-time positioning method of the endoscopic image area described in any one of claims 1 to 5 above is implemented.

Images