Method and related equipment for switching 2D and 3D images based on endoscopic surgery scene

Abstract

The present application discloses a method and related equipment for switching 2D and 3D images based on endoscopic surgery scenes, which is used to solve the problem of constantly manually switching between 3D and 2D image displays during surgery, which is cumbersome to operate, reduces surgical efficiency, and prolongs the time. The method includes: acquiring multiple frames of images captured by the endoscope at intervals of a specified duration; determining whether the endoscope is in a moving state based on the degree of change in the image content of the multiple frames of images; if in a moving state, determining to output a 2D image; if in a stationary state, determining to output a 3D image. Using this method, 2D images and 3D images can be adaptively switched and displayed according to the scene, without the doctor manually switching between 3D and 2D image displays, thereby avoiding the trouble of repeatedly manually operating the endoscope display during surgery, shortening the surgery time, improving surgery efficiency, and enhancing the user experience.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a method and related equipment for switching between 2D and 3D images based on an endoscopic surgery scene. Background Art

[0002] Traditional 2D endoscopes can only display two-dimensional images, but there is an error between the two-dimensional image seen by the eyes and the actual three-dimensional relationship, which inevitably leads to hand-eye coordination problems during surgery; 3D endoscopes restore the three-dimensional surgical field of view in real vision and have a magnifying effect, overcoming the visual differences and inconveniences caused by traditional 2D endoscopes, and can more realistically display the tissue parts of the human body, improve the speed and accuracy of surgery, and bring convenience to doctors. However, in actual use, sometimes the long-term display of 3D images can cause doctors to have nausea, vomiting, dizziness, visual fatigue and other problems, affecting the effect of the surgery and the experience is poor.

[0003] In the prior art, doctors need to manually operate the endoscope display to switch between 2D and 3D image display. Although this method is simple to operate, it requires doctors to constantly operate the display during the operation, which is very troublesome and affects the operation time and efficiency. The actual experience is not good. Summary of the invention

[0004] The purpose of this application is to provide a method and related equipment for switching 2D and 3D images based on endoscopic surgery scenes, so as to solve the problem of constantly manually switching 3D and 2D image displays during surgery, which is cumbersome to operate and reduces surgical efficiency and prolongs surgery time.

[0005] In a first aspect, the present application provides a method for switching 2D and 3D images based on an endoscopic surgery scene, the method comprising:

[0006] Acquire multiple frames of images collected by the endoscope at intervals of a specified time length;

[0007] Determining whether the endoscope is in a moving state based on a degree of change in image content of the multiple frames of images;

[0008] If it is in a moving state, determine to output a 2D image;

[0009] If it is in a stationary state, it is determined to output a 3D image.

[0010] In a possible implementation, determining whether the endoscope is in a moving state based on the degree of change of the image contents of the multiple frames of images specifically includes:

[0011] Acquire at least one set of image pairs from the multiple frames of images, wherein each set of image pairs is composed of two adjacent frames of images from the multiple frames of images;

[0012] Determining the degree of change of the image content of each image pair in sequence according to the acquisition time sequence, and if the degree of change of the image content of any image pair is less than a degree threshold, determining that the endoscope is in a stationary state;

[0013] If the degree of change of the image content of any image pair is greater than or equal to the degree threshold, the degree of change of the image content of the next image pair is determined; and if the degree of change of the image content of each image pair is greater than or equal to the degree threshold, it is determined that the endoscope is in a moving state.

[0014] In a possible implementation, determining whether the endoscope is in a moving state based on the degree of change of the image contents of the multiple frames of images specifically includes:

[0015] Acquire at least one set of image pairs from the multiple frames of images, wherein each set of image pairs is composed of two adjacent frames of images from the multiple frames of images;

[0016] Determining in parallel the degree of change in image content for each image pair;

[0017] If the degree of change of the image content of any image pair is less than a degree threshold, it is determined that the endoscope is in a stationary state;

[0018] If the degree of change of the image content of each image pair is greater than or equal to the degree threshold, it is determined that the endoscope is in a moving state.

[0019] In a possible implementation manner, for any image pair, determining the degree of change of the image content of the image pair specifically includes:

[0020] Selecting at least one sampling region with the same position in the image pair;

[0021] For each sampling area that has at least one identically located sampling area, do the following:

[0022] Determine the sum of each of the three components to be processed, namely, the R value, the G value, and the B value in the sampling area of ​​each frame image in the image pair, and the average value of each component to be processed;

[0023] Based on the sum of each to-be-processed component and the average value of each to-be-processed component, calculate the deviation of the sum of each to-be-processed component and the deviation of the average value of each to-be-processed component in the three to-be-processed components of R value, G value and B value in the sampling area in the image pair;

[0024] If the deviation of the sum of each of the three to-be-processed components of the R value, G value, and B value of any sampling area in the image pair and the deviation of the average value of each to-be-processed component are both less than a preset deviation threshold, it is determined that the degree of change of the image content of the image pair is less than the degree threshold;

[0025] If the deviation of at least one of the deviation of the sum of each of the three components to be processed, namely the R value, G value, and B value of each sampling area in the image pair and the deviation of the average value of each component to be processed is greater than or equal to a preset deviation threshold, it is determined that the degree of change of the image content of the image pair is greater than or equal to the degree threshold.

[0026] In a possible implementation manner, calculating the deviation of the sum of each of the three to-be-processed components of R value, G value, and B value in the sampling area in the image pair and the deviation of the average value of each to-be-processed component specifically includes:

[0027] For any one of the three components to be processed, namely, the R value, the G value, and the B value in the sampling area in the image pair, the following operations are performed:

[0028] The sum of the components to be processed in the sampling area in the previous frame image of the image pair is subtracted from the sum of the components to be processed in the sampling area in the next frame image to obtain a difference in the sum of the components to be processed;

[0029] Dividing the difference of the sum of the components to be processed by the sum of the components to be processed in the sampling area in the next frame of image, to obtain the deviation of the sum of the components to be processed in the sampling area in the image pair;

[0030] The difference between the average values ​​of the components to be processed in the sampling area in the previous frame image and the average value of the components to be processed in the sampling area in the next frame image in the image pair is obtained;

[0031] The difference of the average values ​​of the components to be processed is divided by the average value of the components to be processed in the sampling area in the next frame of image to obtain the deviation of the average value of the components to be processed in the sampling area in the image pair.

[0032] In a possible implementation manner, the multiple frames of images specifically include:

[0033] Multiple consecutive frames of images; or,

[0034] A multi-frame image with a sampling interval of n frames, where n is a positive integer.

[0035] In a possible implementation manner, for any image pair, determining the degree of change of the image content of the image pair specifically includes:

[0036] Selecting at least one region of biological tissue from a previous frame of image in the image pair as at least one template region, and recording a first position coordinate of each template region in the previous frame of image;

[0037] Detecting each of the template regions in a subsequent frame of the image pair, and recording a second position coordinate of each of the template regions in the subsequent frame of the image;

[0038] Calculating the displacement of each of the template regions based on the first position coordinates and the second position coordinates;

[0039] If the displacement of any of the template regions is less than a preset displacement threshold, determining that the degree of change of the image content of the image pair is less than a degree threshold;

[0040] If the displacement of each of the template regions is greater than or equal to a preset displacement threshold, it is determined that the degree of change of the image content of the image pair is greater than or equal to a degree threshold.

[0041] In a possible implementation, the method further includes:

[0042] If the template region cannot be detected in the next frame image in the image pair, it is determined that the displacement of the template region is greater than or equal to a preset displacement threshold.

[0043] In a possible implementation, after determining that the endoscope is in a moving state based on the degree of change in the image content of the multiple frames of images and before outputting the 2D image, the method further includes:

[0044] If the output is switched from the 3D image to the 2D image, the last frame of image before the switching is acquired, and the last frame of image before the switching is continuously displayed as the current image; and,

[0045] Switching from the 3D image to the 2D image within a time threshold;

[0046] After switching to the 2D image, ending display of the current image and outputting the 2D image;

[0047] After determining that the endoscope is in a stationary state based on the degree of change of the image contents of the multiple frames of images and before outputting the 3D image, the method further includes:

[0048] If the output is switched from a 2D image to the 3D image, the last frame of image before the switching is acquired, and the last frame of image before the switching is continuously displayed as the current image; and,

[0049] Switching from the 2D image to the 3D image within a time threshold;

[0050] After switching to the 3D image, the display of the current image ends and the 3D image is output.

[0051] In a second aspect, the present application provides an endoscope device, including an endoscope, a processor, a display, and a memory:

[0052] The endoscope is used to collect images;

[0053] The memory is used to store a computer program executable by the processor;

[0054] The display is used to display the image;

[0055] The processor is connected to the memory and is configured to execute the instructions to implement the method for switching 2D and 3D images based on an endoscopic surgery scene as described in any one of the first aspects above.

[0056] In a third aspect, the present application provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by an endoscopic device, the endoscopic device can perform the method of switching 2D and 3D images based on endoscopic surgery scenes as described in any one of the first aspects above.

[0057] In a fourth aspect, the present application provides a computer program product, including a computer program:

[0058] When the computer program is executed by a processor, the method for switching 2D and 3D images based on an endoscopic surgery scene as described in any one of the first aspects above is implemented.

[0059] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0060] The embodiment of the present application obtains multiple frames of images collected by the endoscope at intervals of a specified time; based on the degree of change in the image content of the multiple frames of images, determines whether the endoscope is in a moving state; if it is in a moving state, determines to output a 2D image; if it is in a stationary state, determines to output a 3D image. In this way, 2D images and 3D images can be adaptively switched and displayed according to the scene, without the doctor manually switching between 3D and 2D image displays, thereby avoiding the trouble of repeatedly manually operating the endoscope display during the operation, shortening the operation time, improving the operation efficiency, and enhancing the user experience.

[0061] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 An application scenario diagram of the method for switching 2D and 3D images based on an endoscopic surgery scene provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of the structure of an endoscope device provided in an embodiment of the present application;

[0065] Figure 3A A flowchart of a method for switching 2D and 3D images based on an endoscopic surgery scene provided in an embodiment of the present application;

[0066] Figure 3B A schematic diagram of multiple frames of images collected by an endoscope provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of a process for determining the degree of change of image content of any image pair provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of sampling areas with the same position in the image alignment provided by an embodiment of the present application;

[0069] Figure 6 A schematic diagram of another process for determining the degree of change of the image content of any image pair provided in an embodiment of the present application;

[0070] Figure 7 A schematic diagram of a process of switching a 2D image to a 3D image provided in an embodiment of the present application;

[0071] Figure 8 A schematic diagram of the process of switching a 3D image to a 2D image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0073] Furthermore, in the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0074] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0075] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0076] Endoscope: A detection instrument that integrates traditional optics, ergonomics, precision machinery, modern electronics, mathematics, and software. It has image sensors, optical lenses, light sources, mechanical devices, etc. The endoscope can enter the human body through the natural channels of the human body, or enter the human body through a small incision made during surgery. When using it, the endoscope is introduced into the pre-examined organ, and the changes in the relevant parts can be directly observed. At the same time, the endoscope can also be used to see lesions that cannot be displayed by X-rays, so it is very useful for doctors. For example, with the help of an endoscope, a doctor can observe ulcers or tumors in the stomach and formulate the best treatment plan based on this. The quality of the image displayed by the endoscope directly affects the use effect of the endoscope and also marks the development level of endoscopic technology. At present, 2D endoscopes are widely used in the market, and 3D endoscopes are relatively less.

[0077] Traditional 2D endoscopes can only display two-dimensional images, but there is an error between the two-dimensional image seen by the eyes and the actual three-dimensional relationship, and it is inevitable that hand-eye coordination will occur during surgery. Compared with traditional 2D endoscopes, the advantages of 3D endoscopes are fast positioning, fast image capture, and a greater sense of depth. 3D endoscopes restore the three-dimensional surgical field of view in real vision and have a magnifying effect. They overcome the visual differences and inconveniences caused by traditional 2D endoscopes, can more realistically display the tissue parts of the human body, improve the speed and accuracy of surgery, and bring convenience to doctors. However, in actual use, sometimes the long-term display of 3D images can cause doctors to have problems such as nausea, vomiting, dizziness, and visual fatigue, affecting the effect of the surgery and the experience is poor.

[0078] In the prior art, doctors need to manually operate the endoscope display to switch between 2D and 3D image display. Although this method is simple to operate, it requires doctors to constantly operate the display during the operation, which is very troublesome and affects the operation time and efficiency. The actual experience is not good.

[0079] In view of this, the present application provides a method and related equipment for switching 2D and 3D images based on endoscopic surgery scenes, so as to solve the problem of constantly manually switching 3D and 2D image displays during surgery, which is cumbersome to operate and reduces surgical efficiency and prolongs surgery time.

[0080] The inventive concept of the present application can be summarized as follows: acquiring multiple frames of images collected by the endoscope at intervals of a specified duration; determining whether the endoscope is in a moving state based on the degree of change in the image content of the multiple frames of images; if in a moving state, determining to output a 2D image; if in a stationary state, determining to output a 3D image. In this way, it is possible to judge the state of the endoscope according to the degree of change in the image content of the multiple frames of images collected by the endoscope. If the endoscope moves, it is possible to switch from 3D to 2D image display; if the endoscope changes from moving to stationary, it is possible to switch from 2D to 3D, thereby adaptively switching between 2D and 3D images, avoiding the trouble of doctors repeatedly manually operating the endoscope to switch images during surgery, shortening the surgery time, and improving surgery efficiency.

[0081] After introducing the main inventive ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0082] refer to Figure 1, which is an application scenario diagram of the method for switching 2D and 3D images based on endoscopic surgery scenes provided in an embodiment of the present application. In the application scenario diagram, a doctor 101, an endoscopic device 102, and a patient 103 are included. Among them:

[0083] The endoscope device 102 is used to enter and exit the body of the patient 103, scan and collect images of various tissue parts in the human body, and determine the state of the endoscope device 102 through the degree of image change of the collected multiple frames of images, and process the collected images according to the determined state, and finally display the processed images and provide them to the doctor 101 for viewing.

[0084] Doctor 101 is used to control the endoscope device 102 to enter and exit the patient 103 and control the movement of the endoscope device 102 in the patient 103, and to find the lesion site between the various organs in the patient 103 by viewing the image displayed by the endoscope device 102, and to perform surgical cutting and reconstruction after determining the lesion site.

[0085] Patient 103 consents to doctor 101 controlling endoscope device 102 to enter and exit his body, and consents to doctor 101 performing surgery on him.

[0086] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 1 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0087] Reference Figure 2 , is a schematic diagram of the structure of the endoscope device provided in the embodiment of the present application. Among them, this part can be composed of Figure 1 The partial modules or functional components of the endoscope device 102 shown are implemented, and only the main components will be described below, while other components, such as memory, controller, control circuit, etc., will not be described here in detail.

[0088] like Figure 2 As shown, the endoscope device 102 includes an endoscope 210 and an endoscope display 220. The endoscope 210 has an image sensor, an optical lens, a light source, a mechanical device, etc. (not shown in the figure), can enter the body through the natural channels of the human body, and is mainly used to collect multiple frames of images of human tissue parts at specified intervals. The endoscope 210 can be used to see lesions that cannot be displayed by X-rays, which is very useful for doctors.

[0089] The endoscope display 220 is composed of an image processing module 221 and a module screen 222. The endoscope display 220 is a dedicated display mainly used to receive images collected by the endoscope 210 and display them.

[0090] The image processing module 221 includes an image motion determination module 2211 and a display control module 2212, and is mainly used to process images collected by the endoscope 210. The image processing module 221 can be a movement or a FPGA (Field Programmable Gate Array) chip, and this application does not limit this.

[0091] The image motion determination module 2211 is mainly used to determine the degree of change of the image collected by the endoscope 210, determine whether the endoscope is in a relatively static state or a fast moving state, and send a control instruction for displaying the image to the display control module 2212. The display control module 2212 is mainly used to control whether the image is displayed in 3D or 2D.

[0092] The module screen 222 may include a plurality of backlight partitions and a display portion of a TCON (Timer Control Register) (not shown in the figure), and each backlight partition may emit light to illuminate the endoscope display 220 .

[0093] In order to facilitate understanding of the method for switching 2D and 3D images based on an endoscopic surgery scene provided in an embodiment of the present application, this is further explained below in conjunction with the accompanying drawings.

[0094] Figure 3A The flowchart of the method for switching 2D and 3D images based on the endoscopic surgery scene provided in the embodiment of the present application is as follows. Figure 3A As shown, the method comprises the following steps:

[0095] In step 301, multiple frames of images collected by an endoscope are acquired at intervals of a specified time length.

[0096] In a possible implementation manner, the multiple frames of images specifically include: multiple frames of continuous images; or multiple frames of images with a sampling interval of n frames; wherein n is a positive integer.

[0097] In a possible implementation, when observing changes in the image content of multiple frames of images, capturing multiple frames of continuous images can more sensitively and meticulously observe changes in the image content than capturing multiple frames of images with a sampling interval of n frames. Therefore, according to actual usage, it can be set whether to capture multiple frames of continuous images or to set the sampling interval to n frames and capture one frame of image every n frames.

[0098] Among them, n can also be set according to actual conditions, and this application does not impose any restrictions on this.

[0099] In one possible implementation, to determine whether the output image is a 2D image or a 3D image, it is first necessary to determine the state of the endoscope. If the endoscope is in a moving state, the image contents of the multiple frames of images collected by the endoscope are different and have large changes. If the endoscope is in a stationary state, the image contents of the multiple frames of images collected by the endoscope change very little or almost no change. Therefore, it is necessary to determine the degree of change of the image content of the multiple frames of images based on the multiple frames of images collected by the endoscope, thereby determining the state of the endoscope.

[0100] In step 302, based on the degree of change of the image content of the multiple frames of images, it is determined whether the endoscope is in a moving state. If it is in a moving state, in step 303, it is determined to output a 2D image. If it is in a stationary state, in step 304, it is determined to output a 3D image.

[0101] In a possible implementation, in order to avoid a situation where a large change occurs between two adjacent frames of images, and the images after these two frames of images are relatively stable and have not changed significantly, a temporary misjudgment of whether the endoscope has moved may occur. That is, when only two frames of images are used to determine whether the endoscope has moved, the accuracy of the judgment structure needs to be improved. Therefore, in order to make up for this deficiency, the embodiment of the present application will select at least one set of image pairs for comparison of the degree of change, and finally determine the state of the endoscope based on the comparison results of the degree of change of at least one set of image pairs. It can be implemented as follows:

[0102] Acquire at least one set of image pairs from the multiple frames of images, wherein each set of image pairs is composed of two adjacent frames of images from the multiple frames of images;

[0103] Determining the degree of change of the image content of each image pair in sequence according to the acquisition time sequence, and if the degree of change of the image content of any image pair is less than a degree threshold, determining that the endoscope is in a stationary state;

[0104] If the degree of change of the image content of any image pair is greater than or equal to the degree threshold, the degree of change of the image content of the next image pair is determined; and if the degree of change of the image content of each image pair is greater than or equal to the degree threshold, it is determined that the endoscope is in a moving state.

[0105] like Figure 3B FIG. 1 is a schematic diagram of multiple frames of images collected by an endoscope according to an embodiment of the present application. Figure 3BIn the example, three frames of images collected by the endoscope are acquired in the order of the time axis, that is, one frame of image is collected at time T-1, one frame of image is collected at time T, and one frame of image is collected at time T+1. According to the order of the acquisition time, the degree of change of the image content of the image pair consisting of the one frame of image collected at time T-1 and the one frame of image collected at time T is first determined. If the degree of change of the image content of this image pair is less than the degree threshold, it is determined that the endoscope is in a stationary state; if the degree of change of the image content of this image pair is greater than or equal to the degree threshold, then the degree of change of the image content of the image pair consisting of the one frame of image collected at time T and the one frame of image collected at time T+1 is determined. If the degree of change of the image content of this image pair is greater than or equal to the degree threshold, it is finally determined that the endoscope is in a moving state; if the degree of change of the image content of this image pair is less than the degree threshold, it is determined that the endoscope is in a stationary state.

[0106] Therefore, the state of the endoscope can be determined by using a method of determining the degree of change in the image content of each image pair in order of acquisition time and comparing the results of each image pair in turn.

[0107] This method can determine that the endoscope is in a stationary state when the degree of change of the image content of any image pair is less than the degree threshold, and there is no need to determine the degree of change of the image content of the subsequent image pairs. However, if the degree of change of the image content of any image pair is greater than or equal to the degree threshold, it is necessary to determine the degree of change of the next image pair, and it is necessary to compare again and again, which makes the efficiency low.

[0108] Therefore, in a possible implementation, if the processing capability of the endoscope processor allows, the degree of change of the image content of each image pair can also be determined simultaneously, so as to determine the state of the endoscope. Collecting the degree of change of the image content based on the multiple frames of images to determine whether the endoscope is in a moving state specifically includes:

[0109] Acquire at least one set of image pairs from the multiple frames of images, wherein each set of image pairs is composed of two adjacent frames of images from the multiple frames of images;

[0110] Determining in parallel the degree of change in image content for each image pair;

[0111] If the degree of change of the image content of any image pair is less than a degree threshold, it is determined that the endoscope is in a stationary state;

[0112] If the degree of change of the image content of each image pair is greater than or equal to the degree threshold, it is determined that the endoscope is in a moving state.

[0113] For example, while determining Figure 3BThe change degree of the image content of the image pair consisting of a frame of image collected at time T-1 and a frame of image collected at time T, and the change degree of the image content of the image pair consisting of a frame of image collected at time T and a frame of image collected at time T+1 are shown in the figure, and then the result of the change degree of the image content of the two groups of image pairs is compared with the degree threshold. If the change degree of the image content of one group of image pairs in the two groups of image pairs is less than the degree threshold, it is determined that the endoscope is in a stationary state; if the change degree of the image content of the two groups of image pairs is greater than or equal to the degree threshold, it is determined that the endoscope is in a moving state.

[0114] Therefore, the degree of change in the image content of each image pair can be determined in parallel, and the degree of change in the image content of all image pairs can be obtained at the same time, so that the state of the endoscope can be determined by comparing all the results once, and multiple comparisons are not allowed, which saves time and improves efficiency.

[0115] In a possible implementation, whether the method of determining the degree of change of the image content of each image pair in sequence according to the acquisition time sequence or determining the degree of change of the image content of each image pair in parallel, it is necessary to determine the degree of change of the image content of the image pair. Figure 4 , is a flow chart of determining the degree of change of the image content of any image pair provided by an embodiment of the present application. For any image pair, determining the degree of change of the image content of the image pair specifically includes the following steps:

[0116] In step 401, at least one sampling region with the same position is selected in the image pair.

[0117] It can be implemented as follows: Figure 5 In the two frames of the image pair shown, considering the size of the field of view of the endoscope acquisition device during acquisition, the edge position of the image is selected as the sampling area. If the edge position does not change significantly, it means that the field of view of the acquisition has not changed significantly, that is, the acquired image has not changed significantly. In addition, because the center position of the image is generally the focus of the operation during the operation, three small blocks of the same size are selected as sampling areas at these three positions of the two frames of images, respectively, and are recorded as area 1, area 2, area 3, area 1', area 2', and area 3'. The position of the upper left corner coordinates of the first sampling area of ​​the two frames of images is (a, b), the position of the upper left corner coordinates of the second sampling area is (c, d), and the position of the upper left corner coordinates of the third sampling area is (e, f). Therefore, the positions of the three sampling areas of the two frames of images are the same, that is, area 1 and area 1' are sampling areas with the same position, area 2 and area 2' are sampling areas with the same position, and area 3 and area 3' are sampling areas with the same position.

[0118] In step 402, for each sampling area in at least one sampling area with the same position, the following operations are performed: the sum of each of the three components to be processed, namely, the R value, G value, and B value in the sampling area of ​​each frame image in the image pair and the average value of each component to be processed are determined, that is, each component to be processed of each pixel point in the sampling area is added to obtain the sum of each component to be processed, which is recorded as R1sum, G1sum, and B1sum; the average value of each component to be processed of each pixel point in the sampling area is obtained, which is recorded as R1avg, G1avg, and B1avg.

[0119] It can be implemented as follows: Figure 5 The sum of each of the three components to be processed, namely, the R value, the G value, and the B value in region 1, region 2, region 3, region 1', region 2', and region 3', and the average value of each component to be processed are recorded as:

[0120] Previous frame image:

[0121] Area 1: TR1sum, TG1sum, TB1sum, TR1avg, TG1avg, TB1avg

[0122] Area 2: TR2sum, TG2sum, TB2sum, TR2avg, TG2avg, TB2avg

[0123] Area 3: TR3sum, TG3sum, TB3sum, TR3avg, TG3avg, TB3avg

[0124] Next frame image:

[0125] Region 1': R1sum, G1sum, B1sum, R1avg, G1avg, B1avg

[0126] Region 2': R2sum, G2sum, B2sum, R2avg, G2avg, B2avg

[0127] Region 3': R3sum, G3sum, B3sum, R3avg, G3avg, B3avg

[0128] In step 403, based on the sum of each component to be processed and the average value of each component to be processed, the deviation of the sum of each component to be processed and the deviation of the average value of each component to be processed in the three components to be processed, namely, R value, G value, and B value in the sampling area in the image pair are calculated.

[0129] In a possible implementation, determining the deviation of the sum of each component and the deviation of the average value of each component to be processed may be implemented as follows:

[0130] For any one of the three components to be processed, namely, the R value, the G value, and the B value in the sampling area in the image pair, the following operations are performed:

[0131] The sum of the components to be processed in the sampling area in the previous frame image of the image pair is subtracted from the sum of the components to be processed in the sampling area in the next frame image to obtain a difference in the sum of the components to be processed;

[0132] Dividing the difference of the sum of the components to be processed by the sum of the components to be processed in the sampling area in the next frame of image, to obtain the deviation of the sum of the components to be processed in the sampling area in the image pair;

[0133] The difference between the average values ​​of the components to be processed in the sampling area in the previous frame image and the average value of the components to be processed in the sampling area in the next frame image in the image pair is obtained;

[0134] The difference of the average values ​​of the components to be processed is divided by the average value of the components to be processed in the sampling area in the next frame of image to obtain the deviation of the average value of the components to be processed in the sampling area in the image pair.

[0135] In step 404, the deviation of the sum of each of the three components to be processed, namely, the R value, the G value, and the B value of any sampling area in the image pair and the deviation of the average value of each component to be processed are compared to see whether at least one of the deviations is greater than or equal to a preset deviation threshold; if the deviation of the sum of each of the three components to be processed, namely, the R value, the G value, and the B value of any sampling area in the image pair and the deviation of the average value of each component to be processed are both less than the preset deviation threshold, then in step 405, it is determined that the degree of change of the image content of the image pair is less than the degree threshold;

[0136] If at least one of the deviation of the sum of the three components to be processed, namely the R value, G value, and B value of each sampling area in the image pair and the deviation of the average value of each component to be processed is greater than or equal to a preset deviation threshold, then in step 406, it is determined that the degree of change in the image content of the image pair is greater than or equal to the degree threshold.

[0137] It can be implemented as: Figure 5 The deviation of the sum of the three components to be processed, namely, the R value, the G value, and the B value of the sampling area with the same position in the two frames of the image pair, and the deviation of the average value of each component to be processed can be recorded as:

[0138]

[0139]

[0140]

[0141] The preset deviation threshold is set to K. If all the above results are less than K, that is, the results of the above six sets of data are within the threshold range, then it is considered Figure 5 The degree of change in the image content of the image pair in is less than the degree threshold. If one of the results obtained from the above six sets of data is greater than or equal to K, it is considered Figure 5 The degree of change of the image content of the image pair in is greater than or equal to the degree threshold. Wherein, because the calculated deviation is a ratio, the preset deviation threshold can be set to the same value.

[0142] Therefore, the degree of change in the image content can be determined by comparing the degree of change in the pixel data of the image in the sampling area at the same position in the image captured by the endoscope, and the state of the endoscope can be judged, thereby determining whether the output image is a 2D image or a 3D image.

[0143] In a possible implementation, the state of the endoscope can also be determined by observing the position change of the same biological tissue in multiple frames of images. Figure 6 , is another flow chart of determining the degree of change of the image content of any image pair provided by an embodiment of the present application. For any image pair, determining the degree of change of the image content of the image pair specifically includes the following steps:

[0144] In step 601, at least one region of biological tissue is selected from a previous frame image in the image pair as at least one template region, and the first position coordinates of each template region in the previous frame image are recorded.

[0145] In step 602, each of the template regions is detected in the next frame of the image pair, and the second position coordinates of each of the template regions in the next frame of the image are recorded.

[0146] In a possible implementation, if the template region cannot be detected in a subsequent frame image in the image pair, it is determined that the displacement of the template region is greater than or equal to a preset displacement threshold.

[0147] In step 603, the displacement of each of the template regions is calculated based on the first position coordinates and the second position coordinates.

[0148] For example, it can be implemented as follows: the first position coordinates of the template area in the previous frame image are (a, b), the second position coordinates of the template area in the next frame image are (c, d), and the distance between the two coordinate points, that is, the displacement D of the template area is:

[0149]

[0150] Among them, other methods can also be used to calculate the displacement of the template area, and this application does not limit this.

[0151] In step 604, it is determined whether the displacement of any of the template regions is less than a preset displacement threshold; if the displacement of any of the template regions is less than the preset displacement threshold, then in step 605, it is determined that the degree of change of the image content of the image pair is less than the degree threshold; if the displacement of each of the template regions is greater than or equal to the preset displacement threshold, then in step 606, it is determined that the degree of change of the image content of the image pair is greater than or equal to the degree threshold.

[0152] Therefore, the degree of change in the image content of the image pair can be determined by the position change of the same biological tissue in multiple frames of images, thereby determining whether the endoscope is in a moving state and outputting a 2D image or a 3D image.

[0153] In a possible implementation, in order to prevent a black screen or freeze and the like from occurring when switching images during the process of performing 2D display and 3D display in the endoscope according to the above method, in order to ensure smooth image switching and make it difficult for a user to perceive the actual switching process, in an embodiment of the present application, after determining that the endoscope is in a moving state and before outputting a 2D image based on the degree of change in the image content of the multiple frames of images, if the output is switched from the 3D image to the 2D image, the last frame of image before the switching is obtained, and the last frame of image before the switching is continuously displayed as the current image; and, switching from the 3D image to the 2D image within a time threshold; after switching to the 2D image, ending the display of the current image, and outputting the 2D image.

[0154] Similarly, based on the degree of change in the image content of the multiple frames of images, after determining that the endoscope is in a stationary state and before outputting a 3D image, in an embodiment of the present application, when switching from a 2D image to the 3D image output, the last frame of image before the switch is obtained, and the last frame of image before the switch is continuously displayed as the current image; and, switching from the 2D image to the 3D image within a time threshold; after switching to the 3D image, ending the display of the current image, and outputting the 3D image.

[0155] like Figure 7 As shown, it is a schematic diagram of the process of switching from a 2D image to a 3D image provided by an embodiment of the present application. Figure 8 FIG. 1 is a schematic diagram of a process of switching a 3D image to a 2D image provided by an embodiment of the present application. Figure 7 and Figure 8In the process, after determining the state of the endoscope based on the degree of change of the image content of the multiple frames of images, the last frame of image before switching is read, and this image is kept in the current display. At the same time, the endoscope switches from 3D image to 2D image or from 2D image to 3D image. The processor of the endoscope can control this switching process within time t. After the switching is completed, the display of the current image ends and the switched image is displayed.

[0156] The duration threshold may vary according to the processing capability of the processor of the endoscope. The smaller the duration threshold is, the faster the switching process is and the smoother the image switching is.

[0157] In this way, smooth switching of images can be achieved, and black screen or freeze phenomena can be prevented during the image switching process, making it difficult for users to perceive the actual switching process.

[0158] Based on the foregoing description, the embodiment of the present application obtains multiple frames of images collected by the endoscope at intervals of a specified duration; determines the degree of change of the image content of the multiple frames of images by various means, and determines whether the endoscope is in a moving state based on the degree of change of the image content of the multiple frames of images; if it is in a moving state, determines to output a 2D image; if it is in a stationary state, determines to output a 3D image. In this way, the state of the endoscope can be judged according to the degree of change of the image content of the multiple frames of images collected by the endoscope, so that the display of 2D images and 3D images can be adaptively switched according to the scene, avoiding the trouble of the doctor repeatedly manually operating the endoscope to switch images during the operation, shortening the operation time, improving the operation efficiency, and improving the user experience.

[0159] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0160] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for switching between 2D and 3D images based on an endoscopic surgery scene, characterized in that: The method comprises: Acquire multiple frames of images collected by the endoscope at intervals of a specified time length; Determine whether the endoscope is in a moving state based on the degree of change of the image content of the multiple frames of images; the determination of whether the endoscope is in a moving state based on the degree of change of the image content of the multiple frames of images specifically includes: acquiring at least one group of image pairs from the multiple frames of images, wherein each group of image pairs is composed of two adjacent frames of images in the multiple frames of images; determining the degree of change of the image content of each image pair in parallel; if the degree of change of the image content of any image pair is less than a degree threshold, determine that the endoscope is in a stationary state; if the degree of change of the image content of each image pair is greater than or equal to the degree threshold, determine that the endoscope is in a moving state; For any image pair, determining the degree of change of the image content of the image pair specifically includes: selecting at least one sampling area with the same position in the image pair; for each sampling area in the at least one sampling area with the same position, performing the following operations: determining the sum of each of the three to-be-processed components of R value, G value, and B value in the sampling area of ​​each frame image in the image pair and the average value of each to-be-processed component; based on the sum of each to-be-processed component and the average value of each to-be-processed component, calculating the deviation of the sum of each of the three to-be-processed components of R value, G value, and B value in the sampling area in the image pair and the deviation of the average value of each component to be processed; if the deviation of the sum of each of the three components to be processed, namely, the R value, the G value, and the B value of any sampling area in the image pair and the deviation of the average value of each component to be processed are both less than the preset deviation threshold, it is determined that the degree of change of the image content of the image pair is less than the degree threshold; if the deviation of at least one of the deviation of the sum of each of the three components to be processed, namely, the R value, the G value, and the B value of each sampling area in the image pair and the deviation of the average value of each component to be processed is greater than or equal to the preset deviation threshold, it is determined that the degree of change of the image content of the image pair is greater than or equal to the degree threshold; The calculation of the deviation of the sum of each of the three to-be-processed components of R value, G value, and B value in the sampling area of ​​the image pair and the deviation of the average value of each to-be-processed component specifically includes: for any to-be-processed component of the three to-be-processed components of R value, G value, and B value in the sampling area of ​​the image pair, the following operations are performed: the sum of the to-be-processed components in the sampling area in the previous frame image of the image pair is subtracted from the sum of the to-be-processed components in the sampling area in the next frame image to obtain the difference of the sums of the to-be-processed components; the difference of the sums of the to-be-processed components is divided by The sum of the components to be processed in the sampling area in one frame of image is obtained to obtain the deviation of the sum of the components to be processed in the sampling area in the image pair; the average value of the components to be processed in the sampling area in the previous frame of image in the image pair is subtracted from the average value of the components to be processed in the sampling area in the next frame of image to obtain the difference of the average values ​​of the components to be processed; the difference of the average values ​​of the components to be processed is divided by the average value of the components to be processed in the sampling area in the next frame of image to obtain the deviation of the average value of the components to be processed in the sampling area in the image pair; If it is in a moving state, determine to output a 2D image; If it is in a stationary state, it is determined to output a 3D image.

2. The method according to claim 1, characterized in that The determining whether the endoscope is in a moving state based on the degree of change of the image contents of the multiple frames of images specifically includes: Acquire at least one set of image pairs from the multiple frames of images, wherein each set of image pairs is composed of two adjacent frames of images from the multiple frames of images; Determining the degree of change of the image content of each image pair in sequence according to the acquisition time sequence, and if the degree of change of the image content of any image pair is less than a degree threshold, determining that the endoscope is in a stationary state; If the degree of change of the image content of any image pair is greater than or equal to the degree threshold, the degree of change of the image content of the next image pair is determined; and if the degree of change of the image content of each image pair is greater than or equal to the degree threshold, it is determined that the endoscope is in a moving state.

3. The method according to claim 1, characterized in that The multiple frames of images specifically include: Multiple consecutive frames of images; or, A multi-frame image with a sampling interval of n frames, where n is a positive integer.

4. The method according to claim 2, characterized in that For any image pair, determining the degree of change of the image content of the image pair specifically includes: Selecting at least one region of biological tissue from a previous frame of image in the image pair as at least one template region, and recording a first position coordinate of each template region in the previous frame of image; Detecting each of the template regions in a subsequent frame of the image pair, and recording a second position coordinate of each of the template regions in the subsequent frame of the image; Calculating the displacement of each of the template regions based on the first position coordinates and the second position coordinates; If the displacement of any of the template regions is less than a preset displacement threshold, determining that the degree of change of the image content of the image pair is less than a degree threshold; If the displacement of each of the template regions is greater than or equal to a preset displacement threshold, it is determined that the degree of change of the image content of the image pair is greater than or equal to a degree threshold.

5. The method according to claim 4, characterized in that The method further comprises: If the template region cannot be detected in the next frame image in the image pair, it is determined that the displacement of the template region is greater than or equal to a preset displacement threshold.

6. The method according to claim 1, characterized in that After determining that the endoscope is in a moving state based on the degree of change of the image contents of the multiple frames of images and before outputting the 2D image, the method further includes: If the output is switched from the 3D image to the 2D image, the last frame of image before the switching is acquired, and the last frame of image before the switching is continuously displayed as the current image; and, Switching from the 3D image to the 2D image within a time threshold; After switching to the 2D image, ending display of the current image and outputting the 2D image; After determining that the endoscope is in a stationary state based on the degree of change of the image contents of the multiple frames of images and before outputting the 3D image, the method further includes: If the output is switched from a 2D image to the 3D image, the last frame of image before the switching is acquired, and the last frame of image before the switching is continuously displayed as the current image; and, Switching from the 2D image to the 3D image within a time threshold; After switching to the 3D image, the display of the current image ends and the 3D image is output.

7. An endoscope device, characterized in that: Includes endoscope, processor, display and memory: The endoscope is used to collect images; The memory is used to store a computer program executable by the processor; The display is used to display the image; The processor is connected to the memory and is configured to execute instructions to implement the method for switching 2D and 3D images based on an endoscopic surgery scene according to any one of claims 1 to 6.

Images