Endoscope image processing method and endoscope system thereof

Abstract

The embodiment of the invention discloses an endoscope image processing method and an endoscope system thereof, and the method comprises the steps: obtaining a to-be-processed endoscope image and at least one frame of historical endoscope image, then recognizing a surgical instrument image in the to-be-processed endoscope image, enabling at least a part of target tissue to be shielded by the surgical instrument image, and then carrying out the recognition of the surgical instrument image in the to-be-processed endoscope image, the method comprises the following steps: determining a target tissue image which is not shielded by a surgical instrument in at least one frame of historical endoscope image, finally performing fusion processing on a to-be-processed endoscope image and the target tissue image to obtain a target endoscope image, and displaying the target endoscope image, different blocks in the surgical instrument image are subjected to transparency processing of different degrees, the transparency of the blocks closer to the edge of the surgical instrument is lower, the overall real contour of the surgical instrument can be clearly observed, meanwhile, the target tissue can be more remarkably observed in the central area of the surgical instrument, and the surgical instrument can be more accurately observed. And the overall display effect of the endoscope image is improved.

Description

Technical Field

[0001] The present application relates to the field of medical equipment technology, and in particular to an endoscopic image processing method and an endoscopic system thereof. Background Art

[0002] Medical endoscopes are indispensable tools in clinical diagnosis and surgery, helping doctors observe the internal tissues of the human body through real-time images. However, during endoscopic imaging, the presence of surgical instruments may block the doctor's view of key areas, affecting the accuracy and safety of the surgery. To solve this problem, a common method is to simply remove the surgical instrument image, but this may cause the doctor to miss the information about the surgical instrument. In order to see the information of the surgical instrument, there is currently a solution to make the surgical instrument image semi-transparent. However, this processing method will greatly reduce the display effect of the blocked image, and the endoscopic image display effect is poor.

[0003] Currently, there are also image processing algorithms to enhance the visualization of surgical instruments, such as superimposing virtual surgical instrument logos on images without surgical instruments. However, these logos cannot truly reflect the actual use status of surgical instruments, which is not convenient for doctors to perform surgical operations. Summary of the invention

[0004] The following is a summary of the subject matter described in detail in this application. This summary is not intended to limit the scope of the claims.

[0005] The embodiment of the present application provides an endoscopic image processing method and an endoscopic system thereof, which can not only avoid the obstruction of the surgical instrument that affects the doctor's observation of the internal tissue of the human body, but also preserve the actual usage status of the surgical instrument.

[0006] On the one hand, an embodiment of the present application provides a method for processing an endoscopic image, comprising:

[0007] Acquire an endoscopic image to be processed and at least one frame of historical endoscopic image;

[0008] Identifying a surgical instrument image in the endoscopic image to be processed, wherein at least a portion of the target tissue is obscured by the surgical instrument image;

[0009] determining a target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic images;

[0010] The endoscopic image to be processed and the target tissue image are fused to obtain the target endoscopic image. During the fusion process, different blocks in the surgical instrument image are processed with different degrees of transparency, wherein the closer the block is to the edge of the surgical instrument, the lower the transparency;

[0011] Display the target endoscope image.

[0012] Further, in some embodiments, determining the target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic image includes one of the following:

[0013] Inputting at least one frame of historical endoscopic image into an image processing model for prediction processing to obtain a first endoscopic image that is not blocked by the surgical instrument, wherein the first endoscopic image includes a target tissue image;

[0014] Alternatively, a first endoscopic image that is not blocked by the surgical instrument is selected from at least one frame of historical endoscopic images, wherein the first endoscopic image includes an image of the target tissue;

[0015] Alternatively, multiple frames of target historical endoscopic images that are not blocked by surgical instruments are selected from at least one frame of historical endoscopic images, and the multiple frames of target historical endoscopic images are combined to obtain a first endoscopic image, wherein the first endoscopic image includes a target tissue image.

[0016] Further, in some embodiments, combining and processing multiple frames of target historical endoscopic images to obtain a first endoscopic image includes one of the following:

[0017] Overlaying multiple frames of historical endoscopic images of the target to obtain a first endoscopic image;

[0018] Alternatively, multiple frames of target historical endoscopic images are spliced ​​together to obtain a first endoscopic image.

[0019] Further, in some embodiments, the endoscopic image to be processed and the target tissue image are fused to obtain the target endoscopic image, including: fusing the endoscopic image to be processed and a first endoscopic image including the target tissue image to obtain the target endoscopic image.

[0020] Furthermore, in some embodiments, the endoscopic image to be processed and the first endoscopic image including the target tissue image are fused to obtain the target endoscopic image, including:

[0021] Determine a target distance between each block in the surgical instrument image and the edge of the surgical instrument;

[0022] The target weight corresponding to each block is determined according to the target distance of each block, wherein the target weight is positively correlated with the transparency of the block, and the target distance is negatively correlated with the target weight;

[0023] The endoscopic image to be processed and the first endoscopic image are fused, wherein the block in the surgical instrument image is fused with the target tissue image according to the target weight.

[0024] Further, in some embodiments, the endoscopic image to be processed and the first endoscopic image are fused, including: determining a fusion area according to the surgical instrument image, and fusing the endoscopic image to be processed and the first endoscopic image based on the fusion area, wherein the target tissue image and the surgical instrument image are located within the fusion area.

[0025] Furthermore, in some embodiments, determining a fusion region according to the surgical instrument image, and fusing the endoscopic image to be processed and the first endoscopic image based on the fusion region includes:

[0026] Determining mask information according to the position of the surgical instrument image, wherein the mask information is used to determine a fusion area that needs to be fused during the fusion process;

[0027] The endoscopic image to be processed and the first endoscopic image are fused according to the mask information, wherein the image information of the endoscopic image to be processed is used for the image outside the fusion area, and for the image within the fusion area, each block is fused with the target tissue image according to the corresponding target weight of each block.

[0028] Further, in some embodiments, the endoscopic image to be processed and the first endoscopic image are fused, including: fusing the endoscopic image to be processed and the first endoscopic image according to the following formula to obtain a target endoscopic image:

[0029] K = W*I+(1-W)*T;

[0030] Wherein, K is the target endoscopic image, I is the endoscopic image to be processed, T is the first endoscopic image, W is the target weight, and the target weight W is expressed as:

[0031] W = 1 - f (d / D);

[0032] Wherein, d is the target distance, D is the gradient coefficient used to adjust the gradient range, and f(d / D) is the gradient function related to the target distance d, wherein d is positively correlated with f(d / D).

[0033] Further, in some embodiments, determining the target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic image includes one of the following:

[0034] Selecting a target historical endoscopic image that is not blocked by a surgical instrument from at least one frame of historical endoscopic images, and extracting a target tissue image from the target historical endoscopic image according to the position of the surgical instrument image;

[0035] Selecting multiple frames of target historical endoscopic images that are not blocked by the surgical instrument from at least one frame of historical endoscopic images, extracting multiple first target tissue images from each target historical endoscopic image according to the position of the surgical instrument image, and obtaining the target tissue image based on the multiple first target tissue images;

[0036] At least one frame of historical endoscopic image is input into the image processing model for prediction processing to obtain a first endoscopic image that is not blocked by the surgical instrument, and the target tissue image is extracted from the first endoscopic image according to the position of the surgical instrument image.

[0037] Further, in some embodiments, obtaining the target tissue image according to the plurality of first target tissue images includes one of the following:

[0038] Inputting a plurality of first target tissue images into an image processing model for prediction processing to obtain a target tissue image;

[0039] Alternatively, a plurality of first target tissue images are superimposed to obtain a target tissue image;

[0040] Alternatively, a target tissue image is selected from a plurality of first target tissue images according to a preset condition.

[0041] Furthermore, in some embodiments, the endoscopic image to be processed and the target tissue image are fused to obtain the target endoscopic image, including one of the following:

[0042] Determine a target distance between each block in the surgical instrument image and the edge of the surgical instrument;

[0043] The target weight corresponding to each block is determined according to the target distance of each block, wherein the target weight is positively correlated with the transparency of the block, and the target distance is negatively correlated with the target weight;

[0044] According to the target weights of each block in the surgical instrument image, the block is fused with the target tissue image to obtain a surgical instrument fused image;

[0045] The surgical instrument image in the endoscopic image to be processed is replaced with the surgical instrument fusion image to obtain a target endoscopic image.

[0046] Furthermore, in some embodiments, the above-mentioned method for processing endoscopic images further includes: obtaining a gradient coefficient for adjusting a gradient range, and setting a target weight according to the gradient coefficient and the target distance.

[0047] Further, in some embodiments, the gradient coefficient is determined by at least one of the following parameters:

[0048] The type of surgical instrument, wherein the type of surgical instrument is set by user input or obtained by identifying the surgical instrument image;

[0049] The location of the surgical instrument;

[0050] The area, diameter or length of the surgical instrument image;

[0051] The type or shooting magnification of the endoscope camera currently shooting the endoscope image to be processed.

[0052] Furthermore, in some embodiments, the block is a single pixel point in the surgical instrument image or is composed of multiple pixel points in the surgical instrument image.

[0053] Further, in some embodiments, determining a target distance between each block in the surgical instrument image and the edge of the surgical instrument includes: determining an edge contour of the surgical instrument, and calculating the shortest distance between each block in the surgical instrument image and the edge contour as the target distance.

[0054] Furthermore, in some embodiments, before the endoscopic image to be processed and the target tissue image are fused, the method further includes: performing brightness matching processing on the endoscopic image to be processed and the target tissue image to match the brightness of the endoscopic image to be processed and the target tissue image.

[0055] Furthermore, in some embodiments, the surgical instrument image includes an edge contour area and a center area, and different areas in the surgical instrument image are processed with different degrees of transparency, including:

[0056] The first transparency processing is applied to the blocks in the edge contour area;

[0057] The second transparency is applied to the blocks in the central area, and the first transparency is smaller than the second transparency.

[0058] Further, in some embodiments, the first transparency is 0% and the second transparency is 100%.

[0059] On the other hand, an embodiment of the present application provides an endoscope system, characterized in that it includes:

[0060] Endoscope;

[0061] Display device;

[0062] The endoscope host is connected to the endoscope and is used to obtain the endoscopic image data taken by the endoscope. The endoscope host is also used to execute the above-mentioned endoscopic image processing method to obtain the target endoscopic image and display it through the display device.

[0063] The embodiment of the present application includes at least the following beneficial effects: by acquiring an endoscopic image to be processed and at least one frame of historical endoscopic image, then identifying a surgical instrument image in the endoscopic image to be processed, wherein at least part of the target tissue is blocked by the surgical instrument image, then determining a target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic image, and then being able to finely determine the target tissue image in the frame image, thereby improving the authenticity of the target tissue image, and finally, performing fusion processing on the endoscopic image to be processed and the target tissue image to obtain a target endoscopic image, and displaying the target endoscopic image, and in the process of fusion processing, performing different degrees of transparency processing on different blocks in the surgical instrument image, wherein the closer the block to the edge of the surgical instrument is, the lower the transparency is, thereby being able to clearly observe the overall true outline of the surgical instrument, and at the same time, enabling the central area of ​​the surgical instrument to more clearly and significantly observe the target tissue than the edge area, which not only accurately reflects the actual use status of the surgical instrument but also takes into account the display of the true situation of the target tissue, thereby improving the overall display effect of the endoscopic image.

[0064] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0066] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present application;

[0067] Figure 2 An optional flowchart of the method for processing endoscopic images provided in an embodiment of the present application;

[0068] Figure 3 The target endoscopic image is obtained after the endoscopic image to be processed provided in the embodiment of the present application is fused;

[0069] Figure 4 Provided in the embodiments of this application Figure 1 An optional flow chart of step 203;

[0070] Figure 5 An optional flow chart for combining and processing multiple frames of historical endoscopic images of a target provided in an embodiment of the present application;

[0071] Figure 6 An optional flowchart of step A provided in an embodiment of the present application;

[0072] Figure 7 An optional flowchart of step B provided in an embodiment of the present application;

[0073] Figure 8 Provided for the embodiments of this application Figure 1 An optional flow chart of step 203;

[0074] Fig. 9 An optional flow chart for obtaining a target tissue image according to a plurality of first target tissue images provided in an embodiment of the present application;

[0075] Fig.10 Provided for the embodiments of this application Figure 2 An optional flowchart of step 204;

[0076] Fig.11 Provided for the embodiments of this application Figure 2 An optional flow chart before step 204;

[0077] Fig.12 An optional structural schematic diagram of an endoscope system provided in an embodiment of the present application;

[0078] Fig.13 A partial structural block diagram of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present application needs to obtain the target object attribute information, the target object's separate permission or separate consent will be obtained by means of a pop-up window or jumping to a confirmation page, etc., and after the separate permission or separate consent of the target object is clearly obtained, the necessary target object-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0081] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0082] An endoscope is a medical device used to directly observe the internal structure of the human body, and is commonly used in clinical diagnosis and surgical operations. An endoscope transmits images to an external display device through a thin, soft tube with a lens and light source at the front end, helping doctors observe the internal conditions of the body without having to perform large-scale surgical incisions.

[0083] Mask information refers to information used to select certain areas in an image. In image processing, it is usually a binary image (such as a black and white image), where the white part represents the selected area and the black part represents the unprocessed area. Through the mask information, the image processing program can apply specific operations (such as filtering, enhancement, sharpening, etc.) to the specified area.

[0084] Gradient refers to an effect of gradual transition between colors, brightness, values ​​or other attributes. In image processing, gradient is often used for smooth transition, edge enhancement or image masking effect. For example, when applying an image mask, gradient can achieve a gradual change from transparent to opaque, producing a soft transition.

[0085] Reference Figure 1 , Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present application, the implementation environment includes an endoscope host 101, an endoscope 102, and a display device 103, wherein the endoscope host 101 and the endoscope 102 are connected via communication, and the endoscope host 101 and the display device 103 are connected via communication.

[0086] Exemplarily, an interactive process of an endoscopic image processing method is as follows: the endoscope 102 captures multiple frames of endoscopic images and continuously sends each frame of the endoscopic image to the endoscope host 101 in the form of a video stream. After receiving the video stream, the endoscope host 101 parses the video stream and obtains the endoscopic image to be processed and at least one frame of historical endoscopic image. Then, the endoscope host 101 identifies the surgical instrument image in the endoscopic image to be processed, and determines the target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic image. Then, the endoscope host 101 performs fusion processing on the endoscopic image to be processed and the target tissue image to obtain the target endoscopic image in the fusion processing process. Finally, the endoscope host 101 sends the target endoscopic image to the display device 103. After receiving the target endoscopic image, the display device 103 displays the target endoscopic image.

[0087] The endoscope host 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, the server 102 can also be a node server in a blockchain network.

[0088] The endoscope host 101 and the endoscope 102 may be directly or indirectly connected via wired or wireless communication. The endoscope host 101 and the endoscope 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0089] It should be noted that the data transmission format of the communication connection in the embodiments of the present application includes but is not limited to JSON format (JavaScript Object Notation), XML format (Extens ib leMarkup Language), HTML format (HyperText Markup Language) and other data transmission formats.

[0090] The method provided in the embodiments of the present application can be applied to different scenarios, including but not limited to cloud technology, artificial intelligence and other scenarios.

[0091] On the one hand, the principles of the endoscopic image processing method provided in the embodiments of the present application are described in detail below.

[0092] Reference Figure 2 As shown, Figure 2 An optional flowchart of an endoscopic image processing method provided in an embodiment of the present application, the endoscopic image processing method includes but is not limited to the following steps 201 to 205.

[0093] Step 201: Acquire an endoscopic image to be processed and at least one frame of historical endoscopic image.

[0094] The historical endoscopic image is an endoscopic image frame that takes the current frame of the endoscopic image to be processed as a starting frame and is before the starting frame.

[0095] In one possible implementation, the endoscope continuously performs endoscopic imaging of the target tissue and generates a corresponding video stream, and then the endoscope continuously sends the video stream to the endoscope host. After receiving the video stream, the endoscope host parses the video stream and stores it in the local memory. Next, the endoscope host obtains the latest frame of the endoscope image containing the surgical instrument at the current moment as the endoscopic image to be processed, and then the endoscope host uses the current frame of the endoscopic image to be processed as the starting frame, and reads at least one frame of historical endoscopic image in the local memory, thereby being able to obtain the real-time endoscopic image to be processed and the corresponding historical endoscopic image in the current video stream, providing real-time conditions for subsequent image processing operations and improving the authenticity of subsequent image processing operations.

[0096] It should be noted that the video stream includes images of surgical instruments not appearing in the endoscope lens, images of surgical instruments moving in the endoscope lens, and images of surgical instruments stationary under the endoscope lens, which is not specifically limited in this application.

[0097] Step 202: Identify the surgical instrument image in the endoscopic image to be processed.

[0098] Wherein, in the endoscopic image to be processed, at least part of the target tissue is blocked by the surgical instrument image.

[0099] In a possible implementation, the endoscopic image to be processed is first gray-scaled, that is, the endoscopic image to be processed is converted from a color image to a gray-scale image, thereby reducing the computational dimension of the data; then, Gaussian blur, mean filtering and other methods are used to remove noise in the image, and then the surgical instrument image in the endoscopic image to be processed is segmented and identified through an image segmentation algorithm to obtain a segmented image region of the surgical instrument image; finally, the segmented image region of the surgical instrument image is edge processed through morphological operations such as dilation and erosion, and the segmented image region of the surgical instrument image is filled through a filling algorithm, thereby obtaining a complete and accurate surgical instrument image, which provides a prerequisite for subsequent image processing operations and improves the accuracy of subsequent image processing operations.

[0100] Step 203: Determine the target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic image.

[0101] In one possible implementation, in at least one frame of historical endoscopic image, image position projection is performed based on the surgical instrument image identified in the above-mentioned endoscopic image to be processed, so as to determine the target tissue image that is not obscured by the surgical instrument in at least one frame of historical endoscopic image.

[0102] Step 204: performing fusion processing on the endoscopic image to be processed and the target tissue image to obtain a target endoscopic image.

[0103] In one possible implementation, the endoscopic image to be processed and the target tissue image are aligned at a geometric angle to avoid image misalignment, wherein an image registration algorithm (such as based on feature point matching or transformation matrix) can also be used for image registration during image alignment; then, the aligned endoscopic image to be processed and the target tissue image are fused through an image fusion algorithm to obtain a target endoscopic image.

[0104] It should be noted that, during the fusion process, different blocks in the surgical instrument image are processed with different degrees of transparency, wherein the closer the block is to the edge of the surgical instrument, the lower the transparency.

[0105] Among them, the target endoscopic image includes a surgical instrument fusion image and an endoscopic image to be processed. The surgical instrument fusion image includes a surgical instrument image after transparency processing and a target tissue image. The surgical instrument image after transparency processing is located in the upper layer of the target tissue image, and the surgical instrument fusion image is located in the upper layer of the endoscopic image to be processed.

[0106] Reference Figure 3 As shown, Figure 3 The target endoscopic image is obtained by fusion processing of the endoscopic image to be processed provided in the embodiment of the present application, Figure 3 It can be seen that the target endoscopic image obtained after the fusion processing can clearly observe the overall true outline of the surgical instrument. At the same time, the central area of ​​the surgical instrument can observe the target tissue more clearly and significantly than the edge area, and the closer the block is to the edge of the surgical instrument, the lower the transparency. Therefore, it can not only accurately reflect the actual use status of the surgical instrument but also show the actual situation of the target tissue, thereby improving the overall display effect of the endoscopic image.

[0107] Step 205: Display the target endoscopic image.

[0108] Specifically, the endoscope host sends the target endoscopic image to the display device, and after receiving the target endoscopic image, the display device renders and displays the target endoscopic image.

[0109] In one possible implementation, the display device receives the target endoscopic image, and the display device also receives the AR model image. The display device then renders the target endoscopic image and the AR model image simultaneously, and finally superimposes the AR model image onto the target endoscopic image, wherein the AR model image is located in the upper layer of the endoscopic image to be processed, and the surgical instrument fusion image is located in the upper layer of the AR model image, so that the surgical instrument fusion image can always be displayed in the target endoscopic image, that is, the overall outline of the surgical instrument and the target tissue image can always be displayed, thereby avoiding the situation where the AR model image obscures the overall outline of the surgical instrument, and improving the overall display effect of the endoscopic image.

[0110] Among them, the AR model image can be a three-dimensional virtual model of various human organs or tissues, including but not limited to the heart, kidneys, liver, pancreas, etc. In one embodiment, the AR model image can also be a surgical planning model (such as a planned cutting path of a scalpel), a virtual model of a medical equipment model (such as a virtual scalpel, etc.), which is not specifically limited in this application.

[0111] Reference Figure 4 As shown, Figure 4 Provided for the embodiments of this application Figure 1 An optional flowchart of step 203 in the figure, the method for processing the endoscopic image includes but is not limited to one of the following steps 401 to 403.

[0112] Step 401: input at least one frame of historical endoscopic image into an image processing model for prediction processing to obtain a first endoscopic image that is not blocked by the surgical instrument.

[0113] The first endoscopic image includes a target tissue image.

[0114] In one possible implementation, at least one frame of historical endoscopic image is adjusted to the size required by the model input layer, and then the pixel value of at least one frame of historical endoscopic image is scaled to a specific range (such as 0 to 1 or -1 to 1) to speed up training and improve model performance. The scaled historical endoscopic image is then input into the image processing model for predictive processing to obtain a first endoscopic image that is not obscured by the surgical instrument.

[0115] It should be noted that the image processing model can be an SVM image processing model, a DeepLab image processing model, a U-Net image processing model, or a SegNet image processing model, and this application does not make any specific limitations.

[0116] Among them, when the image processing model is a deep learning type of image processing model, the network structure of the encoder and decoder corresponding to the model can adopt an Inception network structure, a ResNet network structure, a MobileNet network structure, a SwinTransformer network structure, a ConvNext network structure, or an FPN network structure, and this application does not make any specific limitations.

[0117] Step 402: Alternatively, a first endoscopic image that is not blocked by a surgical instrument is selected from at least one frame of historical endoscopic images.

[0118] The first endoscopic image includes a target tissue image.

[0119] In a possible implementation, a complete first endoscopic image that is not blocked by the surgical instrument is directly selected from at least one frame of historical endoscopic images by using an image segmentation algorithm.

[0120] It should be noted that the image segmentation algorithm may be a maximum inter-class variance method or a watershed algorithm, and this application does not make any specific limitation.

[0121] Step 403: Alternatively, multiple frames of target historical endoscopic images that are not blocked by surgical instruments are selected from at least one frame of historical endoscopic images, and the multiple frames of target historical endoscopic images are combined to obtain a first endoscopic image.

[0122] The first endoscopic image includes a target tissue image.

[0123] In a possible implementation, multiple frames of target historical endoscopic images that are not blocked by surgical instruments are selected from at least one frame of historical endoscopic image through an image segmentation algorithm, and the multiple frames of target historical endoscopic images are combined to obtain a first endoscopic image.

[0124] It should be noted that the image segmentation algorithm may be a maximum inter-class variance method or a watershed algorithm, and this application does not make any specific limitation.

[0125] Reference Figure 5 As shown, Figure 5 An optional flowchart for combining and processing multiple frames of historical endoscopic images of a target provided in an embodiment of the present application, wherein the method for combining and processing multiple frames of historical endoscopic images of a target includes but is not limited to one of the following steps 501 to 502.

[0126] Step 501: Overlay multiple frames of historical endoscopic images of a target to obtain a first endoscopic image;

[0127] In one possible implementation, multiple frames of target historical endoscopic images are aligned, and then the grayscale values ​​of corresponding blocks in all aligned target historical endoscopic images are added and saturated, that is, blocks exceeding the grayscale range are cropped to the maximum or minimum value to obtain a first endoscopic image.

[0128] In one possible implementation, multiple frames of target historical endoscopic images are aligned in image position, and then corresponding blocks in all aligned target historical endoscopic images are linearly superimposed according to fixed weights, which is equivalent to using the grayscale value of one image as the transparent value of another image (i.e., the grayscale value is 0), thereby obtaining a first endoscopic image, thereby reducing the distortion of the first endoscopic image and providing the authenticity of the first endoscopic image.

[0129] Step 502: Alternatively, multiple frames of target historical endoscopic images are spliced ​​together to obtain a first endoscopic image.

[0130] In one possible implementation, multiple frames of target historical endoscopic images are aligned, and then feature points (such as corner points, edge points, etc.) in each frame of the target historical endoscopic image are extracted, and then a correspondence between the feature points is established, and the transformation relationship of each frame of the target historical endoscopic image is determined based on the relationship between the corresponding feature points. Then, based on the transformation relationship of each frame of the target historical endoscopic image, a transformation model between each frame of the target historical endoscopic image is correspondingly established (the transformation model includes an affine transformation model, a perspective transformation model, etc.), and further, based on the established transformation model, each frame of the target historical endoscopic image is converted into a coordinate system of a reference image to achieve a unified coordinate transformation, and finally, the overlapping areas of the target historical endoscopic images after the coordinate transformation of each frame are fused to obtain a spliced ​​first endoscopic image, thereby reducing the distortion of the first endoscopic image and providing the authenticity of the first endoscopic image.

[0131] In one possible implementation, multiple frames of target historical endoscopic images are aligned, and then, using the overlapping area of ​​one image as a template, the most similar matching block is searched in another image, and the transformation relationship of each frame of the target historical endoscopic image is determined according to the relationship between each matching block. Then, according to the transformation relationship of each frame of the target historical endoscopic image, a transformation model between each frame of the target historical endoscopic image is established (the transformation model includes an affine transformation model, a perspective transformation model, etc.). Further, according to the established transformation model, each frame of the target historical endoscopic image is converted into the coordinate system of the reference image to achieve a unified coordinate transformation. Finally, the overlapping areas of the target historical endoscopic images after the coordinate transformation of each frame are fused to obtain a spliced ​​first endoscopic image, thereby reducing the distortion of the first endoscopic image and providing the authenticity of the first endoscopic image.

[0132] Furthermore, the above step 204 also includes step A: step A: fusing the endoscopic image to be processed and the first endoscopic image including the target tissue image to obtain the target endoscopic image.

[0133] In one possible implementation, the endoscopic image to be processed and the first endoscopic image containing the target tissue image are aligned at a geometric angle to avoid image misalignment, wherein an image registration algorithm (such as based on feature point matching or transformation matrix) can also be used for image registration during image alignment; then, the aligned endoscopic image to be processed and the first endoscopic image containing the target tissue image are fused through an image fusion algorithm to obtain the target endoscopic image, thereby reducing the distortion of the first endoscopic image and providing the authenticity of the first endoscopic image.

[0134] Reference Figure 6 As shown, Figure 6 An optional flowchart of step A provided in an embodiment of the present application, the processing method of step A is but not limited to the following steps 601 to 603.

[0135] Step 601: Determine the target distance between each block in the surgical instrument image and the edge of the surgical instrument.

[0136] In one possible implementation, when the block is a regular polygonal block, the centroid of each block is determined, and then the distance from the centroid of each block to each point in its corresponding nearest edge is calculated, and the distance value of the block is continuously updated until the distance value of the block is the minimum distance value. Finally, the above process is repeated until the shortest edge distance of all blocks is calculated, and the target distance of each block from the edge of the surgical instrument is obtained, so that the target distance can be accurately obtained, thereby improving the authenticity of subsequent image processing.

[0137] In one possible implementation, when the block is an irregular polygonal block, the edge vertices of each block are found, and then the distances from the edge vertices of each block to each point in its corresponding nearest edge are calculated respectively, and the distance value of the block is continuously updated until the distance value of the block is the minimum distance value. Finally, the above process is repeated until the shortest edge distances of all blocks are calculated, and the target distances of each block from the edge of the surgical instrument are obtained, so that the target distance can be accurately obtained, thereby improving the authenticity of subsequent image processing.

[0138] Step 602: Determine the target weight corresponding to each block according to the target distance of each block.

[0139] Among them, the target weight is positively correlated with the transparency of the block, and the target distance is negatively correlated with the target weight.

[0140] In a possible implementation, the target distance of each block is standardized, for example, the target distance of each block is standardized to a value range of [0, 1], and then the target weight corresponding to each block is determined according to a weight function.

[0141] Step 603: fusing the endoscopic image to be processed and the first endoscopic image.

[0142] Among them, the blocks in the surgical instrument image and the target tissue image are fused according to the target weight.

[0143] In one possible implementation, the endoscopic image to be processed and the first endoscopic image are aligned at a geometric angle to avoid image misalignment, wherein an image registration algorithm (such as based on feature point matching or transformation matrix) can also be used for image alignment; then, the aligned endoscopic image to be processed and the first endoscopic image are fused according to a target weight to obtain a target endoscopic image, and then a clear and accurate target endoscopic image is obtained, and the overall true outline of the surgical instrument can be clearly observed in the target endoscopic image, and the target tissue can also be clearly observed in the non-edge area of ​​the surgical instrument, thereby improving the overall display effect of the endoscopic image.

[0144] It should be noted that step 603 also includes step B, which is: determining a fusion area according to the surgical instrument image, and fusing the endoscopic image to be processed and the first endoscopic image based on the fusion area.

[0145] The target tissue image and the surgical instrument image are located in the fusion area.

[0146] In one possible implementation, a surgical instrument image, an endoscopic image to be processed, and a first endoscopic image are read, and the surgical instrument image is denoised to improve the accuracy of edge detection and region segmentation. An edge detection algorithm (such as Canny edge detection) is then used to extract edge information in the surgical instrument image. Then, a number of fusion sub-regions are determined by threshold segmentation, contour extraction, or other segmentation methods. Then, the fusion sub-regions extracted from the surgical instrument image are synthesized to obtain a fusion region. Finally, the endoscopic image to be processed and the first endoscopic image are aligned, and the endoscopic image to be processed and the first endoscopic image are fused based on the fusion region.

[0147] Reference Figure 7 As shown, Figure 7 An optional flowchart of step B provided in an embodiment of the present application, the processing method of step B is but not limited to the following steps 701 to 702.

[0148] Step 701: Determine mask information according to the position of the surgical instrument image.

[0149] The mask information is used to determine the fusion area that needs to be fused during the fusion process.

[0150] In one possible implementation, the edges in the surgical instrument image are first extracted, a binary mask matrix is ​​generated based on the extracted edges, the corresponding edge portions in the mask matrix are set to 1, and other regions in the mask matrix are set to 0 to obtain mask information.

[0151] It should be noted that the extraction of edges in the surgical instrument image is as follows: first, the surgical instrument image is Gaussian filtered to smooth the surgical instrument image and reduce the influence of noise, then the gradients of the surgical instrument image in the horizontal and vertical directions are calculated by the Sobel operator, and then the total gradient amplitude and gradient direction of the surgical instrument image are calculated by combining the horizontal gradient and the vertical gradient. Then, after the gradient is calculated, the edge is refined in the gradient direction by non-maximum suppression, and non-edge pixels are suppressed, and edge pixels are retained (that is, for each pixel, if it is not the maximum value in its gradient direction, it is set to 0), and then two thresholds (high threshold and low threshold) are used to distinguish strong edges, weak edges and non-edges. Among them, a strong edge refers to a point with a gradient value higher than a high threshold, a weak edge is a point between a high threshold and a low threshold, and a point below a low threshold is considered not to be an edge and is directly suppressed; finally, the final edge detection is completed by edge connection. It should be noted that if a weak edge pixel is connected to a strong edge pixel, the weak edge pixel is retained, otherwise it is suppressed, thereby ensuring that the detected edge is continuous and the pseudo edge is removed.

[0152] Step 702: performing fusion processing on the endoscopic image to be processed and the first endoscopic image according to the mask information.

[0153] Among them, for the images outside the fusion area, the image information of the endoscope image to be processed is used, and for the images within the fusion area, each block is fused with the target tissue image according to the corresponding target weight of each block.

[0154] In one possible implementation, a weighted average is performed on the endoscopic image to be processed and the first endoscopic image according to the value in the mask information, and then transparency fusion processing is performed on the images in the fusion area according to the target weights corresponding to each block in the endoscopic image to be processed and the first endoscopic image, wherein the value in the mask information determines the weight of each image in the fusion process.

[0155] It should be noted that for the edge areas in the fusion process, gradient mask information can be used to smooth the transition effect and avoid the edges being too abrupt.

[0156] It should be noted that, in the above-mentioned process of fusing the endoscopic image to be processed and the first endoscopic image, the endoscopic image to be processed and the first endoscopic image are fused by the following formula to obtain the target endoscopic image:

[0157] K = W*I+(1-W)*T;

[0158] Wherein, K is the target endoscopic image, I is the endoscopic image to be processed, T is the first endoscopic image, W is the target weight, and the target weight W is expressed as:

[0159] W = 1 - f (d / D);

[0160] Wherein, d is the target distance, D is the gradient coefficient used to adjust the gradient range, and f(d / D) is the gradient function related to the target distance d, wherein d is positively correlated with f(d / D).

[0161] It should be noted that the gradient function is a linear function or a nonlinear function, and this application does not make any specific limitation.

[0162] Reference Figure 8 As shown, Figure 8 Provided in the embodiments of this application Figure 1 An optional flowchart of step 203 in the figure, the method for processing the endoscopic image includes but is not limited to one of the following steps 801 to 803.

[0163] Step 801: Select a target historical endoscopic image that is not blocked by a surgical instrument from at least one frame of historical endoscopic images, and extract a target tissue image from the target historical endoscopic image according to the position of the surgical instrument image.

[0164] In one possible implementation, an occlusion analysis is performed on at least one frame of historical endoscopic image, and by detecting feature points in the historical endoscopic image, it is determined whether these feature points are reduced due to occlusion. If the number or distribution of feature points is significantly reduced, it can be determined that the historical endoscopic image is occluded by the surgical instrument; if the number or distribution of feature points remains unchanged, it can be determined that the historical endoscopic image is not occluded by the surgical instrument, and then, the historical endoscopic image that is not occluded by the surgical instrument is selected as the target historical endoscopic image, and then, the target tissue image is extracted from the target historical endoscopic image according to the position of the surgical instrument image, so that the target tissue image can be accurately extracted, thereby improving the accuracy of image extraction.

[0165] In one possible implementation, a mask matrix of at least one frame of historical endoscopic image is generated according to a foreground / background segmentation algorithm or an edge detection algorithm, and a corresponding occlusion score is calculated according to the mask matrix, wherein the occlusion score can be calculated based on the proportion of the area of ​​the historical endoscopic image occluded by the surgical instrument, and then, the historical endoscopic image with the lowest occlusion score is selected as the target historical endoscopic image, and then, the position coordinates of the surgical instrument image are mapped to the corresponding position of the target historical endoscopic image by perspective transformation or geometric mapping, and finally, according to the mapped coordinates, the corresponding target tissue image is cropped and extracted from the target historical endoscopic image, thereby being able to accurately extract the target tissue image, thereby improving the accuracy of image extraction.

[0166] Step 802: Select multiple frames of target historical endoscopic images that are not blocked by surgical instruments from at least one frame of historical endoscopic images, extract multiple first target tissue images from each target historical endoscopic image according to the position of the surgical instrument image, and obtain the target tissue image based on the multiple first target tissue images.

[0167] In a possible implementation, a mask matrix of at least one frame of historical endoscopic images is generated according to a foreground / background segmentation algorithm or an edge detection algorithm, and a corresponding occlusion score is calculated according to the mask matrix, wherein the occlusion score can be calculated based on the area ratio of the historical endoscopic images occluded by the surgical instrument, then, multiple frames of historical endoscopic images with low occlusion scores are selected, and the position coordinates of the surgical instrument images are respectively mapped to the corresponding positions of the multiple frames of historical endoscopic images with low occlusion scores by perspective transformation or geometric mapping, then, according to the corresponding mapped coordinates, the first target tissue image is cropped and extracted from each frame of the historical endoscopic image with low occlusion scores to obtain multiple first target tissue images, and finally, according to the corresponding mapped coordinates, the multiple first target tissue images are image stitched to obtain the target tissue image, thereby being able to accurately extract the target tissue image, thereby improving the accuracy of image extraction. Wherein, the coordinates between the multiple first target tissue images are mutually filled based on the position of the surgical instrument image, and each first target tissue image has the lowest occlusion score on the corresponding historical endoscopic image.

[0168] In a possible implementation, an occlusion analysis is performed on at least one frame of historical endoscopic images, and by detecting feature points in the historical endoscopic images, it is determined whether these feature points are reduced due to occlusion. If the number or distribution of feature points is significantly reduced, it can be determined that the historical endoscopic image is occluded by the surgical instrument; if the number or distribution of feature points remains unchanged, it can be determined that the historical endoscopic image is not occluded by the surgical instrument, and then multiple frames of historical endoscopic images with low occlusion scores are selected, and the position coordinates of the surgical instrument image are respectively mapped to the corresponding positions of the multiple frames of historical endoscopic images with low occlusion scores by perspective transformation or geometric mapping, and then according to the corresponding mapped coordinates, the first target tissue image is cropped and extracted from each frame of the historical endoscopic image with a low occlusion score to obtain multiple first target tissue images, thereby being able to accurately extract the target tissue image, thereby improving the accuracy of image extraction. Among them, the coordinates between the multiple first target tissue images are mutually supplemented based on the position of the surgical instrument image, and the number and distribution of feature points of each first target tissue image on the corresponding historical endoscopic image are the most stable.

[0169] Step 803: Input at least one frame of historical endoscopic image into the image processing model for predictive processing to obtain a first endoscopic image that is not blocked by the surgical instrument, and extract the target tissue image from the first endoscopic image based on the position of the surgical instrument image.

[0170] In one possible implementation, at least one frame of historical endoscopic image is input into an image processing model for predictive processing to obtain a first endoscopic image that is not obstructed by the surgical instrument. Then, the position coordinates of the surgical instrument image are mapped to the corresponding positions of the target historical endoscopic image through perspective transformation or geometric mapping. Finally, based on the mapped coordinates, the corresponding target tissue image is cropped and extracted from the target historical endoscopic image. The target tissue image can then be accurately extracted through the image processing model, thereby improving the accuracy of image extraction.

[0171] Among them, the image processing model can also be a YOLO image processing model or a Mask R-CNN image processing model, which is not specifically limited in this application.

[0172] Reference Fig. 9 As shown, Fig. 9 An optional flowchart for obtaining a target tissue image based on multiple first target tissue images is provided in an embodiment of the present application. The method for obtaining a target tissue image based on multiple first target tissue images includes but is not limited to one of the following steps 901 to 903.

[0173] Step 901: Input a plurality of first target tissue images into an image processing model for prediction processing to obtain a target tissue image.

[0174] In one possible implementation, multiple first target tissue images are subjected to image preprocessing (for example, image resizing, normalization, channel conversion, batch dimension addition, etc.), and then the multiple preprocessed first target tissue images are input into an image processing model for predictive processing to obtain a target tissue image.

[0175] Step 902: Alternatively, multiple first target tissue images are superimposed to obtain a target tissue image.

[0176] In one possible implementation, multiple first target tissue images are weighted and superimposed in a certain ratio to obtain a target tissue image. For example, there are currently two first target tissue images, the weight of the first first target tissue image is set to 60%, and the weight of the second first target tissue image is set to 40%. Then, the synthesized target tissue image is 60% of the first first target tissue image + 40% of the second first target tissue image.

[0177] In one possible implementation, multiple first target tissue images are overlaid on each other and synthesized by transparency superposition to obtain a target tissue image. For example, there are currently two first target tissue images. The two first target tissue images are overlaid on each other, and then the transparency of the upper half of the first first target tissue image is set to 0, the transparency of the upper half of the first first target tissue image is set to 100%, the transparency of the upper half of the second first target tissue image is set to 100%, the transparency of the lower half of the second first target tissue image is set to 0, and finally the two first target tissue images are overlaid and synthesized by transparency to obtain a target tissue image, wherein the content of the upper half of the target tissue image comes from 100% of the first first target tissue image, and the content of the lower half of the target tissue image comes from 100% of the second first target tissue image.

[0178] In one possible implementation, a corresponding mask matrix is ​​generated based on multiple first target tissue images, and the multiple first target tissue images are superimposed according to the mask matrix to obtain a target tissue image. For example, there are currently two first target tissue images, and the corresponding mask matrix generated based on the two first target tissue images is [0,1], wherein the value of "1" indicates that the block comes from the first first target tissue image, and the value of "0" indicates that the block comes from the second first target tissue image. The two first target tissue images are superimposed according to the mask matrix to obtain a target tissue image, wherein the left half of the target tissue image comes from the first first target tissue image, and the right half of the target tissue image comes from the second first target tissue image.

[0179] Step 903: Alternatively, a target tissue image is selected from a plurality of first target tissue images according to a preset condition.

[0180] In a possible implementation, when the preset condition is that the similarity between first target tissue images is greater than or equal to 95%, then according to the preset condition, a first target tissue image with a similarity greater than or equal to 95% is selected as the target tissue image from multiple first target tissue images.

[0181] In one possible implementation, when the preset condition is that the exposure of the first target tissue image is less than or equal to 50%, then according to the preset condition, the first target tissue image with an exposure less than or equal to 50% is selected as the target tissue image from multiple first target tissue images.

[0182] Reference Fig.10 As shown, Fig.10 Provided for the embodiments of this application Figure 2 An optional flowchart of step 204 in the figure, the method for processing endoscopic images includes but is not limited to the following steps 1001 to 1004.

[0183] Step 1001: Determine the target distance between each block in the surgical instrument image and the edge of the surgical instrument.

[0184] In one possible implementation, when the block is a regular polygonal block, the center of mass of each block is determined, and then the distance from the center of mass of each block to its corresponding nearest edge is calculated, and the distance value of the block is continuously updated until the distance value of the block is the minimum distance value. Finally, the above process is repeated until the shortest edge distance of all blocks is calculated, and the target distance of each block from the edge of the surgical instrument is obtained.

[0185] In one possible implementation, when the block is an irregular polygonal block, the edge vertices of each block are found, and then the distance from the edge vertex of each block to its corresponding nearest edge is calculated, and the distance value of the block is continuously updated until the distance value of the block is the minimum distance value. Finally, the above process is repeated until the shortest edge distances of all blocks are calculated, and the target distance of each block from the edge of the surgical instrument is obtained.

[0186] It should be noted that step 901 also includes step C, step C: determining the edge contour of the surgical instrument, and calculating the shortest distance between each block in the surgical instrument image and the edge contour as the target distance.

[0187] In a possible implementation, the edge contour of the surgical instrument is determined as follows: first, the surgical instrument image is Gaussian filtered to smooth the surgical instrument image and reduce the influence of noise, then the gradient of the surgical instrument image in the horizontal and vertical directions is calculated by the Sobel operator, and then the horizontal gradient and the vertical gradient are combined to calculate the total gradient amplitude and gradient direction of the surgical instrument image. Then, after the gradient is calculated, the edge is refined in the gradient direction by non-maximum suppression, and non-edge pixels are suppressed, and edge pixels are retained (that is, for each pixel, if it is not the maximum value in its gradient direction, it is set to 0), and then two thresholds (high threshold and low threshold) are used to distinguish strong edges, weak edges and non-edges. Among them, a strong edge refers to a point with a gradient value higher than a high threshold, a weak edge is a point between a high threshold and a low threshold, and a point below a low threshold is considered not to be an edge and is directly suppressed; finally, the final edge detection is completed by edge connection. It should be noted that if a weak edge pixel is connected to a strong edge pixel, the weak edge pixel is retained, otherwise the weak edge pixel is suppressed, thereby ensuring that the detected edge is continuous and removing pseudo edges, thereby improving the accuracy of the edge contour.

[0188] Step 1002: Determine the target weight corresponding to each block according to the target distance of each block.

[0189] Among them, the target weight is positively correlated with the transparency of the block, and the target distance is negatively correlated with the target weight.

[0190] In a possible implementation, the target distance of each block is standardized, for example, the target distance of each block is standardized to a value range of [0, 1], and then the target weight corresponding to each block is determined according to a weight function.

[0191] It should be noted that the weight function can be:

[0192] W=1-f(d / D)

[0193] Wherein, d is the target distance, D is the gradient coefficient used to adjust the gradient range, and f(d / D) is the gradient function related to the target distance d, wherein d is positively correlated with f(d / D).

[0194] Step 1003: According to the target weight of each block in the surgical instrument image, the block is fused with the target tissue image to obtain a surgical instrument fused image.

[0195] In one possible implementation, in a low-weight block, the features of the surgical instrument image corresponding to the block position will dominate, while the content of the target tissue image corresponding to the block position will gradually fade or combine with the features of the surgical instrument image; and in a high-weight block, the features of the surgical instrument image corresponding to the block position will dominate, while the content of the target tissue image corresponding to the block position will gradually fade or combine with the features of the surgical instrument image, and then the target weights of each block in the surgical instrument image are converted into transparency values, and each block is fused with the target tissue image to obtain a surgical instrument fusion image, wherein, in the surgical instrument fusion image, the high-weight blocks correspond to displaying more content of the target tissue image, and the low-weight blocks correspond to displaying more content of the surgical instrument image.

[0196] Step 1004: Replace the surgical instrument image in the endoscopic image to be processed with the surgical instrument fusion image to obtain a target endoscopic image.

[0197] In one possible implementation, the surgical instrument image in the endoscopic image to be processed is cropped, and then the surgical instrument fusion image is filled into the blank area in the endoscopic image to be processed. Finally, the filled area in the endoscopic image to be processed is edge transitioned to obtain the target endoscopic image.

[0198] Furthermore, the method for processing the endoscopic image also includes but is not limited to step D.

[0199] Step D: Obtain a gradient coefficient for adjusting the gradient range, and set the target weight according to the gradient coefficient and the target distance.

[0200] In one possible implementation, the gradient coefficient used to adjust the gradient range is 0.8, and the target distance from a certain block to the edge of the surgical instrument is 6 pixels. The target weight is the gradient coefficient 0.8*6=4.8, thereby providing the user with an adjustable gradient coefficient. The gradient coefficient can also be adjusted in a customized manner, thereby enabling the user to customize the overall display effect of the target endoscopic image, making it convenient for the user to adaptively adjust the overall display effect of the target endoscopic image according to different usage scenarios.

[0201] In a possible implementation, the gradient coefficient used to adjust the gradient range is 2, and the target distance from a certain block to the edge of the surgical instrument is 2 pixel values, then the target weight is the gradient coefficient 2 2 =4.

[0202] It should be noted that the gradient coefficient is determined by at least one of the following parameters: 1. The type of surgical instrument (e.g., scalpel, scissors, hemostatic forceps, irrigator, etc.); 2. The part of the surgical instrument (e.g., the blade of the scalpel, the edge of the scissors, the jaws of the hemostatic forceps, the suction port of the aspirator, etc.); 3. The area, diameter or length of the surgical instrument image; 4. The type or shooting magnification of the endoscope camera currently shooting the endoscope image to be processed. Among them, the type of surgical instrument is set by the user input or obtained by identification based on the surgical instrument image.

[0203] It should also be noted that a block is a single pixel point in the surgical instrument image or is composed of multiple pixel points in the surgical instrument image.

[0204] Reference Fig.11 As shown, Fig.11 Provided in the embodiments of this application Figure 2 An optional flowchart before step 204, the method for processing endoscopic images includes but is not limited to the following steps 1101.

[0205] Step 1101: Perform brightness matching processing on the endoscopic image to be processed and the target tissue image to make the brightness of the endoscopic image to be processed and the target tissue image match.

[0206] In a possible implementation, first, the brightness distribution of the endoscopic image to be processed and the brightness distribution of the target tissue image are calculated, wherein calculating the brightness distribution includes calculating the average brightness value of all pixels in the image and calculating the distribution of different brightness values ​​in the image; then, based on the brightness distribution of the endoscopic image to be processed and the brightness distribution of the target tissue image, the brightness value of the target tissue image is adjusted to match the brightness of the endoscopic image to be processed by linear transformation, thereby matching the brightness of the endoscopic image to be processed and the target tissue image, thereby reducing the difference between the endoscopic image to be processed and the target tissue image, avoiding that the image obtained after the fusion of the endoscopic image to be processed and the target tissue image is too abrupt, and improving the overall display effect of the endoscopic image.

[0207] In a possible implementation, the conversion formula of the above linear transformation may be: X=a*H+b;

[0208] Wherein, X is the brightness value of the target tissue image, H is the brightness value of the endoscope image to be processed, a is the brightness scaling factor, and b is the brightness translation factor.

[0209] Furthermore, for the surgical instrument image, the surgical instrument image may be divided into an edge contour area and a central area, wherein the block in the edge contour area is closer to the edge of the surgical instrument than the block in the central area.

[0210] It should be noted that the above process of performing different degrees of transparency processing on different blocks in the surgical instrument image also includes but is not limited to the following steps E to F.

[0211] Step E: Apply the first transparency processing to the block in the edge contour area.

[0212] Specifically, the block of the layer of the edge contour area is processed transparently, that is, the transparency of the block of the edge contour area is set to a first transparency, wherein the first transparency can be any value from 0 to 100%.

[0213] Step D: Apply the second transparency processing to the blocks in the central area.

[0214] Specifically, the blocks of the layer in the center area are processed transparently, that is, the transparency of the blocks in the center area is set to the second transparency, wherein the second transparency can be any value from 0 to 100%.

[0215] It should be noted that the first transparency is smaller than the second transparency.

[0216] In one possible implementation, by setting the transparency of the blocks in the central area to 0% (i.e., the first transparency is 0%) and the transparency of the blocks in the edge contour area to 100% (i.e., the second transparency is 100%), the user can focus more on the content of the central area and reduce the interference of other factors, so that the central area of ​​the surgical instrument can further clearly and significantly observe the target tissue than the edge area, thereby improving the overall display effect of the endoscopic image.

[0217] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0218] On the other hand, the present application also provides an endoscope system, referring to Fig.12 , Fig.12This is an optional structural diagram of an endoscope system provided in an embodiment of the present application, wherein the endoscope system includes an endoscope 1201, a display device 1202, and an endoscope host 1203, wherein the endoscope host 1203 is connected to the endoscope 1201, and the endoscope host 1203 is connected to the display device 1202;

[0219] The endoscope host 1203 is used to obtain the endoscopic image data taken by the endoscope 1201 . The endoscope host 1203 is also used to execute the above-mentioned endoscopic image processing method to obtain the target endoscopic image and display it through the display device 1202 .

[0220] The above-mentioned endoscopic system and the above-mentioned endoscopic image processing method are based on the same inventive concept. The data processing process of the data processing system 1200 describes the endoscopic image processing method of the embodiment of the present application, by acquiring the endoscopic image to be processed and at least one frame of historical endoscopic image, and then identifying the surgical instrument image in the endoscopic image to be processed, wherein at least part of the target tissue is blocked by the surgical instrument image, and then determining the target tissue image that is not blocked by the surgical instrument in at least one frame of historical endoscopic image, and then being able to finely determine the target tissue image in the frame image, thereby improving the authenticity of the target tissue image, and finally, The endoscopic image to be processed and the target tissue image are fused to obtain the target endoscopic image, and the target endoscopic image is displayed. During the fusion process, different blocks in the surgical instrument image are processed with different degrees of transparency, wherein the closer the block is to the edge of the surgical instrument, the lower the transparency, so that the overall true outline of the surgical instrument can be clearly observed. At the same time, the central area of ​​the surgical instrument can observe the target tissue more clearly and significantly than the edge area, which can not only accurately reflect the actual use status of the surgical instrument but also take into account the display of the true situation of the target tissue, thereby improving the overall display effect of the endoscopic image.

[0221] Further, refer to Fig.12 It should be noted that the above-mentioned endoscope system also includes a light source device 1204 and a light guide 1205, wherein the endoscope 1201 includes a mirror tube 1206, a camera assembly 1207 and an optical mount 1208, wherein the light source device 1204 is used to provide an illumination light source, the mirror tube 1206 is used to transmit illumination and realize endoscopic imaging, the camera assembly 1207 is used to collect reflected light signals and generate real-time video streams, the optical mount 1208 is used to adjust and control the light source intensity, and the light source device 1204 is connected to the light source interface of the endoscope 1201 through the light guide 1205.

[0222] It should be noted that the endoscope host 1203 is connected to the endoscope 1201 via a communication cable 1209 , and the endoscope host 1203 is connected to the display device 1202 via a communication cable 1209 .

[0223] The endoscope host for executing the above-mentioned endoscopic image processing method provided in the embodiment of the present application may be a server, referring to Fig.13 , Fig.13 Partial structural block diagram of a server provided in an embodiment of the present application, the server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1310 and a second memory 1330, and one or more storage media 1340 (e.g., one or more mass storage devices) storing application programs 1343 or data 1342. Among them, the second memory 1330 and the storage medium 1340 can be short-term storage or permanent storage. The program stored in the storage medium 1340 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1310 can be configured to communicate with the storage medium 1340 and execute a series of instruction operations in the storage medium 1340 on the server.

[0224] The server may also include one or more second power supplies 1320, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1360, and / or one or more operating systems 1341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0225] The processor in the server 1300 can be used to execute the method for processing endoscopic images.

[0226] An embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the endoscopic image processing method of each of the aforementioned embodiments.

[0227] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned endoscopic image processing method.

[0228] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present application, for example, it can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0229] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0230] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0231] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0235] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0236] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions under the shared conditions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for processing an endoscopic image, characterized in that: The method comprises: Acquire an endoscopic image to be processed and at least one frame of historical endoscopic image; Identifying a surgical instrument image in the endoscopic image to be processed, wherein at least a portion of the target tissue is obscured by the surgical instrument image; Determine a target tissue image that is not blocked by a surgical instrument in at least one frame of the historical endoscopic image; Performing a fusion process on the endoscopic image to be processed and the target tissue image to obtain a target endoscopic image, and during the fusion process, performing different degrees of transparency processing on different blocks in the surgical instrument image, wherein the closer the block is to the edge of the surgical instrument, the lower the transparency; The target endoscopic image is displayed.

2. The method for processing an endoscopic image according to claim 1, characterized in that: Determining the target tissue image that is not blocked by the surgical instrument in at least one frame of the historical endoscopic image comprises one of the following: Inputting at least one frame of the historical endoscopic image into an image processing model for prediction processing to obtain a first endoscopic image that is not blocked by the surgical instrument, wherein the first endoscopic image includes the target tissue image; Alternatively, a first endoscopic image that is not blocked by the surgical instrument is selected from at least one frame of the historical endoscopic image, wherein the first endoscopic image includes the target tissue image; Alternatively, multiple frames of target historical endoscopic images that are not blocked by the surgical instrument are selected from at least one frame of the historical endoscopic image, and the multiple frames of the target historical endoscopic images are combined to obtain a first endoscopic image, wherein the first endoscopic image includes the target tissue image.

3. The method for processing an endoscopic image according to claim 2, characterized in that: The combining and processing of the multiple frames of the target historical endoscopic images to obtain the first endoscopic image includes one of the following: Overlaying multiple frames of the target historical endoscopic images to obtain the first endoscopic image; Alternatively, multiple frames of the target historical endoscopic images are spliced ​​together to obtain the first endoscopic image.

4. The method for processing an endoscopic image according to claim 2 or 3, characterized in that: The step of fusing the endoscopic image to be processed and the target tissue image to obtain a target endoscopic image includes: The endoscopic image to be processed and the first endoscopic image including the target tissue image are fused to obtain a target endoscopic image.

5. The method for processing an endoscopic image according to claim 4, characterized in that: The step of fusing the endoscopic image to be processed and the first endoscopic image including the target tissue image to obtain a target endoscopic image includes: Determine a target distance between each of the blocks in the surgical instrument image and an edge of the surgical instrument; Determining a target weight corresponding to each of the blocks according to the target distance of each of the blocks, wherein the target weight is positively correlated with the transparency of the block, and the target distance is negatively correlated with the target weight; The endoscopic image to be processed and the first endoscopic image are fused, wherein the block in the surgical instrument image is fused with the target tissue image according to the target weight.

6. The method for processing an endoscopic image according to claim 5, characterized in that: The method further comprises: fusing the endoscopic image to be processed with the first endoscopic image, comprising: A fusion region is determined according to the surgical instrument image, and the endoscopic image to be processed and the first endoscopic image are fused based on the fusion region, wherein the target tissue image and the surgical instrument image are located within the fusion region.

7. The method for processing an endoscopic image according to claim 6, characterized in that: The step of determining a fusion region according to the surgical instrument image, and fusing the endoscopic image to be processed and the first endoscopic image based on the fusion region, comprises: Determining mask information according to the position of the surgical instrument image, wherein the mask information is used to determine a fusion area that needs to be fused during the fusion processing; The endoscopic image to be processed and the first endoscopic image are fused according to the mask information, wherein the image information of the endoscopic image to be processed is used for the image outside the fusion area, and for the image within the fusion area, each of the blocks is fused with the target tissue image according to the target weight corresponding to each of the blocks.

8. The method for processing an endoscopic image according to claim 5, characterized in that: The fusing the endoscopic image to be processed and the first endoscopic image comprises: fusing the endoscopic image to be processed and the first endoscopic image by the following formula to obtain a target endoscopic image: K = W*I+(1-W)*T; Wherein, K is the target endoscopic image, I is the endoscopic image to be processed, T is the first endoscopic image, W is the target weight, and the target weight W is expressed as: W = 1 - f (d / D); Wherein, d is the target distance, D is the gradient coefficient used to adjust the gradient range, and f(d / D) is the gradient function related to the target distance d, wherein d is positively correlated with f(d / D).

9. The method for processing an endoscopic image according to claim 1, characterized in that: Determining the target tissue image that is not blocked by the surgical instrument in at least one frame of the historical endoscopic image comprises one of the following: Selecting a target historical endoscopic image that is not blocked by the surgical instrument from at least one frame of the historical endoscopic image, and extracting the target tissue image from the target historical endoscopic image according to the position of the surgical instrument image; Selecting multiple frames of target historical endoscopic images that are not blocked by the surgical instrument from at least one frame of the historical endoscopic image, extracting multiple first target tissue images from each of the target historical endoscopic images according to the position of the surgical instrument image, and obtaining a target tissue image based on the multiple first target tissue images; At least one frame of the historical endoscopic image is input into an image processing model for prediction processing to obtain a first endoscopic image that is not blocked by the surgical instrument, and the target tissue image is extracted from the first endoscopic image according to the position of the surgical instrument image.

10. The method for processing an endoscopic image according to claim 9, characterized in that: The step of obtaining a target tissue image according to the plurality of first target tissue images comprises one of the following: Inputting the plurality of the first target tissue images into an image processing model for prediction processing to obtain the target tissue image; Alternatively, a plurality of the first target tissue images are superimposed to obtain the target tissue image; Alternatively, the target tissue image is selected from the plurality of first target tissue images according to a preset condition.

11. The method for processing an endoscopic image according to claim 9, characterized in that: The step of fusing the endoscopic image to be processed and the target tissue image to obtain a target endoscopic image includes one of the following: Determine a target distance between each of the blocks in the surgical instrument image and an edge of the surgical instrument; Determining a target weight corresponding to each of the blocks according to the target distance of each of the blocks, wherein the target weight is positively correlated with the transparency of the block, and the target distance is negatively correlated with the target weight; According to the target weight of each of the blocks in the surgical instrument image, the blocks are fused with the target tissue image to obtain a surgical instrument fused image; The surgical instrument image in the endoscopic image to be processed is replaced with the surgical instrument fusion image to obtain the target endoscopic image.

12. The method for processing an endoscopic image according to any one of claims 5 to 8 and 11, characterized in that: The method further comprises: A gradient coefficient for adjusting a gradient range is obtained, and the target weight is set according to the gradient coefficient and the target distance.

13. The method for processing an endoscopic image according to claim 8 or 12, characterized in that: The gradient coefficient is determined by at least one of the following parameters: The type of the surgical instrument, wherein the type of the surgical instrument is set by user input or obtained by identification based on the surgical instrument image; the location of the surgical instrument; The area, diameter or length of the surgical instrument image; The type or shooting magnification of the endoscope camera currently shooting the endoscope image to be processed.

14. The method for processing an endoscopic image according to any one of claims 1 to 3 and 5 to 11, characterized in that: The block is a single pixel point in the surgical instrument image or is composed of multiple pixel points in the surgical instrument image.

15. The method for processing an endoscopic image according to any one of claims 5 to 9 and 11, characterized in that: Determining the target distance between each of the blocks in the surgical instrument image and the edge of the surgical instrument includes: The edge contour of the surgical instrument is determined, and the shortest distance between each of the blocks in the surgical instrument image and the edge contour is calculated as the target distance.

16. The method for processing an endoscopic image according to any one of claims 1 to 3 and 5 to 11, characterized in that: Before fusing the endoscopic image to be processed and the target tissue image, the method further includes: A brightness matching process is performed on the endoscopic image to be processed and the target tissue image, so that the brightness of the endoscopic image to be processed and the target tissue image are matched.

17. The method for processing an endoscopic image according to claim 1, characterized in that: The surgical instrument image includes an edge contour area and a central area, and performing different degrees of transparency processing on different blocks in the surgical instrument image includes: Applying a first transparency process to the block in the edge contour area; The block in the central area is processed with a second transparency, and the first transparency is smaller than the second transparency.

18. The method for processing an endoscopic image according to claim 17, characterized in that: The first transparency is 0%, and the second transparency is 100%.

19. An endoscope system, characterized in that: include: Endoscope; Display device; An endoscope host is connected to the endoscope, and is used to obtain endoscopic image data taken by the endoscope. The endoscope host is also used to execute the endoscopic image processing method described in any one of claims 1 to 18 to obtain a target endoscopic image and display it through the display device.

Images