Endoscopic image processing method, device, readable medium and electronic device

Abstract

The present disclosure relates to a method, device, readable medium and electronic device for processing endoscopic images, and relates to the field of image processing technology. The method comprises: obtaining a set of tissue images collected by an endoscope in a tissue to be tested. According to the set of tissue images, the depth image and posture parameters corresponding to each tissue image are determined. According to the posture parameters corresponding to each tissue image, the motion trajectory of the endoscope is determined. According to the depth image corresponding to each tissue image, the contour of the tissue to be tested is determined. According to the motion trajectory and the contour of the tissue to be tested, the blind area ratio during the endoscopic inspection is determined. The present disclosure determines the motion trajectory of the endoscope and the contour of the tissue to be tested based on the depth image and posture parameters corresponding to the tissue image, and thereby determines the blind area ratio during the inspection process, which can realize the monitoring of the inspection range, effectively avoid missed inspections, and thus ensure the effectiveness of the endoscopic inspection.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to a method, device, readable medium and electronic device for processing endoscopic images. Background Art

[0002] Endoscopic examination is a common and effective means of inspection. It has been widely used in the medical field because it can visually observe the internal tissues of the human body. When the endoscope enters the internal tissues of the human body for inspection, there may be blind spots in the field of view. If the blind spots are too large, it may lead to missed inspections, which further renders the inspection invalid. An optional solution is to perform three-dimensional modeling based on the images collected by the endoscope to determine the proportion of the blind spots. The accuracy of the three-dimensional modeling directly affects the accuracy of the blind spot ratio. Since human tissues (such as the intestines, stomach, etc.) are soft tissues, the endoscope will inevitably touch the tissue wall of the soft tissues when entering the soft tissues, causing a large displacement of the soft tissues, resulting in errors in the results of the three-dimensional modeling. At the same time, if polyps are encountered during the inspection, the inspectors will flush or remove the polyps, which will also reduce the accuracy of the three-dimensional modeling. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for processing an endoscopic image, the method comprising:

[0005] Acquire a tissue image set collected by an endoscope in the tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to the collection time;

[0006] Determine, according to the tissue image set, a depth image and a posture parameter corresponding to each of the tissue images;

[0007] Determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images;

[0008] Determining the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images;

[0009] The blind area ratio during the endoscopic examination is determined according to the motion trajectory and the contour of the tissue to be examined.

[0010] In a second aspect, the present disclosure provides an endoscope image processing device, the device comprising:

[0011] An acquisition module, used to acquire a tissue image set acquired by an endoscope in the tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to acquisition time;

[0012] A positioning module, used for determining a depth image and a posture parameter corresponding to each of the tissue images according to the tissue image set;

[0013] A trajectory determination module, used to determine the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images;

[0014] A contour determination module, used to determine the contour of the tissue to be tested according to the depth image corresponding to each of the tissue images;

[0015] A processing module is used to determine the blind area ratio during the endoscopic examination process according to the motion trajectory and the contour of the tissue to be examined.

[0016] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of the present disclosure.

[0017] In a fourth aspect, the present disclosure provides an electronic device, including:

[0018] a storage device having a computer program stored thereon;

[0019] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0020] Through the above technical scheme, the present disclosure first obtains tissue images collected by the endoscope in the tissue to be tested at multiple collection times. Then, based on the tissue image set, the depth image and posture parameters corresponding to each tissue image are determined. Thereafter, the motion trajectory of the endoscope is determined according to the posture parameters corresponding to each tissue image, and the contour of the tissue to be tested is determined according to the depth image corresponding to each tissue image. Finally, based on the motion trajectory and the contour of the tissue to be tested, the blind area ratio during the endoscopic inspection is determined. The present disclosure determines the motion trajectory of the endoscope and the contour of the tissue to be tested according to the depth image and posture parameters corresponding to the tissue image, and thereby determines the blind area ratio during the inspection process, which can realize the monitoring of the inspection range and effectively avoid missed inspections, thereby ensuring the effectiveness of the endoscopic inspection.

[0021] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0023] Figure 1 is a flow chart of a method for processing an endoscopic image according to an exemplary embodiment;

[0024] Figure 2 is a flow chart of another method for processing an endoscopic image according to an exemplary embodiment;

[0025] Figure 3 is a flow chart of another method for processing an endoscopic image according to an exemplary embodiment;

[0026] Figure 4 is a schematic diagram showing the outline of a tissue to be tested according to an exemplary embodiment;

[0027] Figure 5 is a schematic diagram of a positioning model according to an exemplary embodiment;

[0028] Figure 6 is a flow chart of another method for processing an endoscopic image according to an exemplary embodiment;

[0029] Figure 7 is a schematic diagram showing a depth sub-model and a posture sub-model according to an exemplary embodiment;

[0030] Figure 8 is a flow chart of a training positioning model according to an exemplary embodiment;

[0031] Fig. 9 is a schematic diagram of another posture sub-model according to an exemplary embodiment;

[0032] Fig.10 is a flowchart of another training positioning model according to an exemplary embodiment;

[0033] Fig.11 is a flow chart of another method for processing an endoscopic image according to an exemplary embodiment;

[0034] Fig.12 is a block diagram of a device for processing an endoscopic image according to an exemplary embodiment;

[0035] Fig.13 is a block diagram of another apparatus for processing endoscopic images according to an exemplary embodiment;

[0036] Fig.14 is a block diagram of another apparatus for processing endoscopic images according to an exemplary embodiment;

[0037] Fig.15 is a block diagram of another apparatus for processing endoscopic images according to an exemplary embodiment;

[0038] Fig.16 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0040] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0041] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0042] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0045] Figure 1is a flowchart of a method for processing an endoscopic image according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0046] Step 101 : obtaining a tissue image set collected by an endoscope in a tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to the collection time.

[0047] For example, when performing an endoscopic examination, the endoscope will continuously collect tissue images in the tissue to be tested according to a preset collection cycle to obtain a tissue image set. The tissue image set may include multiple tissue images arranged according to the collection time, and the interval between the collection times corresponding to any two adjacent tissue images is the collection cycle. Specifically, multiple tissue images collected within a preset time (for example: 30s) can be used as a tissue image set, or a preset number of tissue images (for example: 100) collected continuously can be used as a tissue image set, and the present disclosure does not specifically limit this. It should be noted that the endoscope described in the embodiments of the present disclosure can be, for example, a colonoscope, a gastroscope, etc. If the endoscope is a colonoscope, then the tissue to be tested is the intestine, and the tissue image is the intestinal image. If the endoscope is a gastroscope, then the tissue to be tested can be the esophagus, stomach, or duodenum, and the above-mentioned tissue image can be an esophageal image, a stomach image, or a duodenal image. The endoscope can also be used to collect images of other tissues, and the present disclosure does not specifically limit this.

[0048] During the endoscopic examination, many invalid images may be collected due to unstable lens insertion techniques or inappropriate endoscope positions, such as images blocked by obstacles, excessive exposure, and low clarity. These invalid images will interfere with the inspection results of the endoscope. Therefore, after obtaining the tissue image set, it can be first determined whether the multiple tissue images included therein are valid to filter out invalid tissue images. If a certain tissue image is an invalid image, the tissue image can be directly discarded. If the tissue image is a valid image, the tissue image can be retained to obtain a filtered tissue image set, which can reduce unnecessary data processing and improve processing speed. For example, each tissue image in the tissue image set can be identified using a pre-trained recognition model to determine whether the tissue image is valid. The recognition model can be, for example, CNN (English: Convolutional Neural Networks, Chinese: Convolutional Neural Network) or LSTM (English: Long Short-Term Memory, Chinese: Long Short-Term Memory Network), or it can be an Encoder in a Transformer (such as Vision Transformer), etc., and the present disclosure does not specifically limit this. Furthermore, each tissue image in the tissue image set may be preprocessed, which can be understood as enhancing the data included in each tissue image. In order to ensure the quality of the tissue image, the preprocessing will not modify the blur or color of the tissue image, so the preprocessing may include: multi-crop processing, flip processing (including: left-right flip, up-down flip, rotation, etc.), random affine transformation, size transformation (English: Resize), etc. The preprocessed tissue image finally obtained may be an image of a specified size (for example, 384*384).

[0049] Step 102: Determine the depth image and posture parameters corresponding to each tissue image according to the tissue image set.

[0050] For example, for each tissue image in the tissue image set, the depth image and posture parameters corresponding to each tissue image can be determined in turn. The depth image corresponding to each tissue image includes the depth of each pixel in the tissue image (which can also be understood as the distance), so the corresponding depth image can reflect the geometric shape of the visible surface in the tissue image without being affected by the texture, color, etc. in the tissue image. In other words, the structural information of the tissue to be measured corresponding to the tissue image can be characterized by the corresponding depth image. The posture parameters corresponding to each tissue image can be understood as the posture parameters of the endoscope when acquiring the tissue image. The posture parameters corresponding to multiple consecutive tissue images can characterize the movement process of the endoscope in the tissue to be measured. The posture parameters may include, for example, a rotation matrix and a translation vector.

[0051] Step 103: determining the motion trajectory of the endoscope according to the posture parameters corresponding to each tissue image.

[0052] Step 104: determining the contour of the tissue to be tested according to the depth image corresponding to each tissue image.

[0053] For example, the posture parameters corresponding to each tissue image in the tissue image set can characterize the movement process of the endoscope in the tissue to be measured. Therefore, the movement trajectory of the endoscope in the tissue to be measured can be obtained according to the posture parameters corresponding to each tissue image. The movement trajectory may include the position of the endoscope when collecting each tissue image, and may also include the angle of the endoscope when collecting each tissue image.

[0054] At the same time, the corresponding depth image can characterize the structural information of the tissue corresponding to the tissue image. Therefore, the outline of the tissue to be tested can be obtained according to the depth image corresponding to each tissue image. The outline of the tissue to be tested can reflect the overall shape of the tissue to be tested, and can also be understood as a template of the tissue to be tested. Taking the endoscope as a colonoscope as an example, the tissue to be tested is the intestine, and the outline of the tissue to be tested can be a distorted cylinder. Specifically, the center line of the tissue to be tested can be determined according to the multiple depth images corresponding to the tissue image set, and then the outline of the tissue to be tested can be obtained according to a preset radius. Modeling can also be performed based on the multiple depth images corresponding to the tissue image set to obtain the outline of the tissue to be tested, and the present disclosure does not make specific limitations on this. It should be noted that Figure 1 The execution order of step 103 and step 104 shown is an exemplary implementation method. Step 104 may be executed first and then step 103, or step 103 and step 104 may be executed simultaneously. The present disclosure does not make any specific limitation on this.

[0055] Step 105 , determining the blind area ratio during the endoscopic examination process according to the motion trajectory and the contour of the tissue to be examined.

[0056] For example, after obtaining the motion trajectory and the contour of the tissue to be measured, the field of view area of ​​the endoscope when collecting each tissue image can be determined according to the position and angle of the endoscope when collecting each tissue image. The field of view area can be understood as the area of ​​the tissue to be measured that the endoscope can observe when collecting the tissue image. Then the field of view area corresponding to each tissue image can be spliced ​​to obtain the area that can be observed during the endoscopic examination, thereby obtaining the blind area ratio during the endoscopic examination. Specifically, the observation area ratio can be determined based on the ratio of the area that can be observed during the endoscopic examination to the contour of the tissue to be measured, and then the blind area ratio can be determined, that is, the blind area ratio = 1-observation area ratio. The blind area ratio can be understood as the ratio of the blind area (i.e., the part that cannot be observed in the field of view of the endoscope) to the contour of the tissue to be measured during the endoscopic examination. Determining the blind area ratio based on the depth image and the motion trajectory can timely reflect the inspection range during the inspection process, thereby avoiding missed inspections and ensuring the effectiveness of the endoscopic examination.

[0057] In summary, the present disclosure first obtains tissue images collected by the endoscope at multiple collection times in the tissue to be tested. Then, based on the tissue image set, the depth image and posture parameters corresponding to each tissue image are determined. Thereafter, the motion trajectory of the endoscope is determined according to the posture parameters corresponding to each tissue image, and the contour of the tissue to be tested is determined according to the depth image corresponding to each tissue image. Finally, based on the motion trajectory and the contour of the tissue to be tested, the blind area ratio during the endoscopic inspection is determined. The present disclosure determines the motion trajectory of the endoscope and the contour of the tissue to be tested based on the depth image and posture parameters corresponding to the tissue image, and thereby determines the blind area ratio during the inspection process, which can realize the monitoring of the inspection range and effectively avoid missed inspections, thereby ensuring the effectiveness of the endoscopic inspection.

[0058] In one implementation, step 102 may be implemented as follows:

[0059] According to each tissue image and the historical tissue image corresponding to the tissue image, the depth image and posture parameters corresponding to the tissue image are determined by a pre-trained positioning model, and the acquisition time of the historical tissue image is before the acquisition time of the tissue image.

[0060] For example, each tissue image and the historical tissue image corresponding to the tissue image can be sequentially input into a pre-trained positioning model, so that the positioning model determines the depth image and posture parameters corresponding to the tissue image based on the tissue image and the corresponding historical tissue image. The acquisition time of the corresponding historical tissue image is before the acquisition time of the tissue image, that is, in the tissue image set, the corresponding historical tissue image is located before the tissue image, and can be the previous tissue image before the tissue image in the tissue image set. For example, the tissue image acquired by the endoscope at time t can be represented as It , then the historical organizational image corresponding to the organizational image can be expressed as I t-1 , that is, the image collected by the endoscope at time t-1.

[0061] The positioning model can be understood as a SLAM (Simultaneous Localization and Mapping) model, which can synchronously determine the corresponding depth image and posture parameters based on each tissue image and the historical tissue image corresponding to the tissue image. The positioning model can determine the depth image corresponding to each tissue image, without adding a depth sensor during endoscopic inspection, which is easy to operate and saves costs. At the same time, the positioning model can determine the posture parameters to accurately obtain the motion trajectory of the endoscope.

[0062] In another implementation, the posture parameters may include a rotation matrix and a translation vector, and the motion trajectory may include the position and angle of the endoscope when acquiring each tissue image. Accordingly, the implementation of step 103 is:

[0063] The position and angle of the endoscope when acquiring the tissue image are determined according to the rotation matrix and translation vector corresponding to each tissue image, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image.

[0064] For example, the position and angle of the endoscope when collecting the tissue image can be determined according to the posture parameters corresponding to each tissue image, and then the position and angle of the endoscope when collecting all tissue images can be arranged in the order indicated by the collection time, so as to obtain the motion trajectory of the endoscope. Specifically, the position of the endoscope when collecting the tissue image can be determined according to the position of the endoscope when collecting the historical tissue image corresponding to the tissue image and the translation vector corresponding to the tissue image, and the angle of the endoscope when collecting the tissue image can be determined according to the angle of the endoscope when collecting the historical tissue image corresponding to the tissue image and the rotation matrix corresponding to the tissue image.

[0065] For example, the position and angle of the first tissue image in the tissue image set can be set to a preset initial position and initial angle, and then the position and angle of the second tissue image can be determined based on the position and angle of the first tissue image, and the rotation matrix and translation vector corresponding to the second tissue image. Then, based on the position and angle of the second tissue image, and the rotation matrix and translation vector corresponding to the third tissue image, the position and angle of the third tissue image are determined, and so on, to obtain the motion trajectory of the endoscope in the tissue to be tested.

[0066] Figure 2is a flowchart of another method for processing an endoscopic image according to an exemplary embodiment. Figure 2 As shown, step 104 may include:

[0067] Step 1041 , determining the center line of the tissue to be tested according to the depth image corresponding to each tissue image.

[0068] Step 1042, determining the contour of the tissue to be tested according to the center line of the tissue to be tested.

[0069] For example, the midpoint of the tissue to be tested in the tissue image can be determined according to the depth image corresponding to each tissue image, and then the midpoints of the tissue to be tested in each tissue image can be connected to obtain the center line of the tissue to be tested. Then, the contour of the tissue to be tested is determined according to the preset radius and center line. Taking the tissue image as an intestinal image and the tissue to be tested as the intestine as an example, the contour of the intestine is a cylinder established according to the preset radius and center line. Specifically, the method for determining the midpoint of the tissue to be tested in each tissue image can first determine the distance to the boundary in the tissue image (for example, it can include: left boundary, right boundary, upper boundary, lower boundary, etc.), and then determine the point with equal distance from each boundary in the depth image as the midpoint of the tissue to be tested in the tissue image.

[0070] Figure 3 is a flowchart of another method for processing an endoscopic image according to an exemplary embodiment. Figure 3 As shown, the implementation of step 105 may include:

[0071] Step 1051, determining the field of view area corresponding to each tissue image according to the position and angle of the endoscope when acquiring each tissue image and the viewing angle of the endoscope.

[0072] Step 1052: determine the total field of view area according to the field of view area corresponding to each tissue image.

[0073] Step 1053, determining the blind area ratio according to the total visual field area and the contour of the tissue to be measured.

[0074] For example, after obtaining the motion trajectory and the contour of the tissue to be measured, the field of view of the endoscope when collecting each tissue image can be determined based on the position and angle of the endoscope when collecting each tissue image, as well as the viewing angle of the endoscope itself. The viewing angle of the endoscope is determined by the optical lens of the endoscope, and the viewing angle can be, for example, 100 degrees or 120 degrees. The field of view can be understood as the area of ​​the tissue to be measured covered by the tissue image. Figure 4Take the outline of the tissue to be measured as an example, where the thick solid line represents the outline of the tissue to be measured (for the convenience of display, a two-dimensional cross-section is used here to represent the outline of the tissue to be measured. In actual situations, the outline of the tissue to be measured is three-dimensional, for example, it can be a cylinder), where k(0) represents the position of the endoscope at time t0, and correspondingly, the angle of the endoscope at time t0 can be expressed as α(0) (it should be noted that α(0) is not shown in Figure 4 The viewing angle of the endoscope is Then, the field of view corresponding to the tissue image collected at time t0 is from point A to point B on the contour. Specifically, the Monte Carlo method can be used to evenly distribute X test points (X ≥ 100) on the contour of the tissue to be tested, and then determine the area of ​​the field of view according to the number of test points included in the field of view.

[0075] Afterwards, the visual field area corresponding to each tissue image can be spliced ​​to obtain the total visual field area. Specifically, the visual field area corresponding to each tissue image can be summed, and the summed result can be used as the total visual field area, or the test points covered in the visual field area corresponding to each tissue image can be unioned to obtain the total number of test points included in the total visual field area as the total visual field area. Finally, the ratio of the total visual field area to the total area of ​​the contour of the tissue to be tested can be used as the observation area ratio, and then (1-observation area ratio) can be used as the blind area ratio. For example, the total number of test points included in the total visual field area is Y, that is, Y test points are covered in the area that can be observed during the endoscopic examination, and there are a total of X test points distributed on the contour of the tissue to be tested. Then, the observation area ratio can be determined as Y / X first, and then the blind area ratio can be further determined as 1-Y / X.

[0076] In one implementation, step 1051 may be implemented by the following steps:

[0077] Step 1) According to the posture parameters corresponding to each tissue image, the position of the endoscope when acquiring the tissue image is converted into a center position corresponding to the center line of the tissue to be measured.

[0078] Step 2) Determine the central viewing angle corresponding to the central position according to the posture parameters corresponding to the tissue image, the viewing angle of the endoscope, and the angle of the endoscope when acquiring the tissue image.

[0079] Step 3) Determine the maximum visual field area corresponding to the center position.

[0080] Step 4) Determine the field of view area corresponding to the tissue image according to the central viewing angle and the maximum field of view area.

[0081] For example, in order to quickly determine the field of view area corresponding to each tissue image and improve the efficiency of image processing, the position and viewing angle of the endoscope can be first converted to the center position and center viewing angle on the center line of the tissue to be measured. It can be understood that the field of view that can be observed by the endoscope at the position when the tissue image is collected and the angle when the tissue image is collected is the same as the field of view that can be observed by the endoscope at the center position and the angle when the tissue image is collected. Figure 4 As shown, d(0) indicates that the position of the endoscope at time t0 is converted to the center position on the center line, and the viewing angle of the endoscope is The corresponding central viewing angle may be δ, so that the field of view observed by the endoscope at the central viewing angle at the angle when the tissue image is acquired on d(0) is also from point A to point B. Specifically, the central position and the corresponding central viewing angle may be determined in the following manner:

[0082] We can first draw a perpendicular line from the position of the endoscope when collecting the tissue image to the contour of the tissue to be measured, and the intersection of the perpendicular line and the center line is the center position, that is, d(0). Then we can The angle of the endoscope when acquiring the tissue image is geometrically transformed to obtain the central viewing angle δ.

[0083] Afterwards, the maximum field of view area corresponding to the center position can be determined based on the center position. The maximum field of view area corresponding to the center position can be understood as the maximum range that the endoscope can observe at the center position, that is, the maximum range that can be observed when the optical lens of the endoscope is rotated 360 degrees. Then, the field of view area corresponding to the tissue image can be determined based on the center viewing angle and the maximum field of view area. Specifically, the field of view area corresponding to the tissue image can be determined based on the ratio of the center viewing angle to 360 degrees and the product of the maximum field of view area. For example, the center viewing angle is 120 degrees, and the number of test points included in the maximum field of view area is 210, then the number of test points included in the field of view area corresponding to the tissue image is 210*(120 / 360)=70.

[0084] In one implementation, the structure of the positioning model can be as follows Figure 5 As shown, it includes: a depth sub-model and a posture sub-model. The input of the depth sub-model and the input of the posture sub-model are used as the input of the positioning model, and the output of the depth sub-model and the output of the posture sub-model are used as the output of the positioning model.

[0085] Figure 6 is a flowchart of another method for processing an endoscopic image according to an exemplary embodiment. Figure 6 As shown, the positioning model includes: a depth sub-model and a posture sub-model. Step 102 may include:

[0086] Step 1021: input the tissue image into the depth sub-model to obtain a depth image corresponding to the tissue image output by the depth sub-model.

[0087] For example, the tissue image can be used as the input of the depth sub-model, and the depth sub-model can output the depth image corresponding to the tissue image. The structure of the depth sub-model can be as follows: Figure 7 As shown in (a) in the figure, it can be a UNet structure, which includes multiple stride convolution layers (English: stride conv) to downsample the tissue image, for example, it can be downsampled to 1 / 8 of the resolution of the tissue image, and then multiple transpose convolution layers (English: transpose conv) are used to upsample to the resolution of the tissue image to obtain a depth image corresponding to the tissue image.

[0088] Step 1022: input the tissue image and the corresponding historical tissue image into the posture sub-model to obtain the posture parameters corresponding to the tissue image output by the posture sub-model.

[0089] For example, the tissue image and the corresponding historical tissue image can be used as inputs of the posture sub-model, and the posture sub-model can output the rotation matrix and translation vector corresponding to the tissue image. Specifically, the tissue image and the corresponding historical tissue image can be concatenated (English: Concat), and the concatenated result can be input into the posture sub-model. The structure of the posture sub-model can be as follows: Figure 7 As shown in (b), it can be a ResNet structure (for example, it can be ResNet34). The concatenation result of the tissue image and the corresponding historical tissue image is input into the initial convolutional pooling layer, passes through multiple residual blocks in the middle, and finally the fully connected layer outputs the rotation matrix and translation vector corresponding to the tissue image.

[0090] Figure 8 is a flowchart of a training positioning model according to an exemplary embodiment. Figure 8 As shown, the positioning model is trained through the following steps:

[0091] Step A: input the sample tissue image into the depth sub-model to obtain a sample depth image corresponding to the sample tissue image, and input the historical sample tissue image into the depth sub-model to obtain a historical sample depth image corresponding to the historical sample tissue image, where the historical sample tissue image is an image collected before the sample tissue image.

[0092] For example, the sample tissue image (denoted as I a ) as the input of the deep sub-model, the deep sub-model can output the sample depth image corresponding to the sample tissue image (denoted as Da ). Similarly, the historical sample organization image (denoted as I b ) as the input of the deep sub-model, the deep sub-model can output the historical sample depth image corresponding to the historical sample tissue image (denoted as D b ). The sample tissue image may be extracted from an endoscopic video, which may be a video recorded during a previous endoscopic examination, and may be obtained by selecting different endoscopic examinations for different users. Furthermore, when extracting frames from the endoscopic video, invalid images (such as images blocked by obstacles, images with excessive exposure, images with low clarity, etc.) may be filtered out. Correspondingly, the historical sample tissue image is the tissue image of the previous frame of the sample tissue image.

[0093] Step B, inputting the sample tissue image and the historical sample tissue image into the posture sub-model to obtain the sample posture parameters corresponding to the sample tissue image and the endoscope internal parameters for collecting the sample tissue image output by the posture sub-model, wherein the endoscope internal parameters include focal length and translation size.

[0094] For example, the sample tissue image and the historical sample tissue image can be used as inputs of the posture sub-model, and the posture sub-model can output the sample posture parameters corresponding to the sample tissue image and the endoscope internal parameters (expressed as K) for collecting the sample tissue image. Among them, the endoscope internal parameters may include focal length and translation size, and the sample posture parameters include a sample rotation matrix (expressed as R) and a sample translation vector (expressed as t). Specifically, the sample tissue image and the historical sample tissue image can be spliced ​​to input the spliced ​​result into the posture sub-model.

[0095] During the training phase, the posture sub-model can also add a linear layer (denoted as intrisic layer) on the basis of the convolutional pooling layer, multiple residual blocks and the fully connected layer, such as Fig. 9 As shown in Figure 2, the fully connected layer (represented as pose layer) outputs the sample pose parameters, and the linear layer can output the endoscope internal parameters. The endoscope internal parameters K can be in the form of:

[0096]

[0097] Among them, f x and f y Respectively represent the focal length of the endoscope in the X and Y directions (in pixels), c x and c y They represent the translation size of the origin in the X and Y directions (in pixels). The posture sub-model can obtain the endoscope internal parameters while obtaining the sample posture parameters. There is no need to calibrate the endoscope in advance, which is easy to operate. At the same time, it can be adapted to various endoscopes, which improves the applicability of the depth sub-model.

[0098] Step C, determining the target loss based on the endoscope internal parameters, the sample depth image, the historical sample depth image and the sample posture parameters.

[0099] In step D, the positioning model is trained using the back propagation algorithm with the goal of reducing the target loss.

[0100] For example, the positioning model can be trained using the back propagation algorithm based on the endoscope internal parameters, sample depth images, historical sample depth images and sample posture parameters, with the goal of reducing target loss. When training the positioning model, the sample tissue images and historical sample tissue images used to train the positioning model can be quickly obtained without pre-labeling, that is, the positioning model adopts an unsupervised learning training method.

[0101] Furthermore, the initial learning rate of the training positioning model can be set to: 1e-2, the batch size can be set to: 16*4, the optimizer can be selected to: SGD, the epoch can be set to: 500, and the size of the sample tissue image can be: 384×384.

[0102] Fig.10 is a flowchart of another training positioning model according to an exemplary embodiment. Fig.10 As shown, the implementation of step C may include:

[0103] Step C1, interpolating the historical sample tissue image according to the sample depth image, the sample posture parameter and the endoscope internal parameter to obtain an interpolated tissue image.

[0104] Step C2, determining the photometric loss according to the sample tissue image and the interpolated tissue image.

[0105] For example, the sample depth image, sample posture parameters and endoscope internal parameters can be used to perform differentiable bilinear interpolation processing on the historical sample tissue image to obtain an interpolated tissue image. The photometric loss is then determined based on the sample tissue image and the interpolated tissue image. The interpolated tissue image can be understood as an image obtained by observing the content in the sample tissue image from the perspective of collecting the historical sample tissue image. According to the principle of bundle adjustment, the pixel grayscale of the same spatial point should be fixed in each image. Therefore, when images collected from different perspectives are converted to another perspective, the pixels at the same position in the two images at the same perspective should be the same. Therefore, the photometric loss can be understood as the difference between the sample tissue image and the interpolated tissue image. For example, the photometric loss can be determined by formula 1:

[0106]

[0107] Among them, Lp represents the photometric loss, p represents the pixel, N represents the effective pixel in the sample tissue image, and |N| represents the number of effective pixels. a (p) represents the pixel value of p in the sample tissue image, I' a (p) represents the pixel value of p in the interpolated tissue image. || || 1 Indicates L 1 Norm, L 1 The norm is more robust to discrete points.

[0108] Step C3, determining the smoothing loss according to the gradient of the sample depth image and the gradient of the sample tissue image.

[0109] For example, in the low texture area of ​​the sample tissue image (or interpolated tissue image), due to the less image feature information, the performance of the photometric loss is weak, so a smoothing loss can be added as a regularization term to constrain the generated sample depth image. The smoothing loss can be determined based on the gradient of the sample depth image and the gradient of the sample tissue image. The smoothing loss can ensure that the sample depth image is generated under the guidance of the sample tissue image, so that the generated sample depth map can retain more gradient information at the edge, that is, the edge is more obvious and the detail information is richer. For example, the smoothing loss can be determined by formula 2:

[0110]

[0111] Among them, L s represents the smoothing loss, represents the gradient of p in the sample tissue image, represents the gradient of p in the sample depth image.

[0112] Step C4: transforming the sample depth image into a first depth image according to the sample posture parameters and the endoscope internal parameters.

[0113] Step C5: transforming the historical sample depth image into a second depth image according to the sample posture parameters and the endoscope internal parameters.

[0114] Step C6: determining a consistency loss according to the first depth image and the second depth image.

[0115] For example, since the sample tissue image and the historical sample tissue image face the same three-dimensional space, the sample depth image and the historical sample depth image have spatial consistency. The sample posture parameter and the endoscope internal parameter can be used to transform the sample depth image into a first depth image (expressed as ), and using the sample pose parameters and endoscope internal parameters, the historical sample depth image is transformed into a second depth image (denoted as D b'). The first depth image can be understood as a depth image obtained by converting the sample depth image through posture transformation and observing the content in the sample tissue image from the perspective of collecting the historical sample tissue image. The second depth image can be understood as a depth image obtained by interpolating the historical sample depth image to observe the content in the sample tissue image from the perspective of collecting the historical sample tissue image.

[0116] Then the consistency loss is determined based on the first depth image and the second depth image. In other words, the consistency loss can reflect the difference between the first depth image and the second depth image. Through training, the consistency can be propagated to multiple sample depth images, which also ensures the scale consistency of multiple sample depth images, which is equivalent to smoothing multiple sample depth images to ensure spatial consistency. For example, the consistency loss can be determined by formula 3:

[0117]

[0118] Among them, L G represents the consistency loss, represents the depth of p in the first depth image, D b '(p) represents the depth of p in the second depth image.

[0119] Step C7, determining the target loss according to the photometric loss, the smoothness loss and the consistency loss.

[0120] For example, the target loss can be determined based on the photometric loss, smoothness loss, and consistency loss. For example, the target loss can be obtained by weighted summing the photometric loss, smoothness loss, and consistency loss using Formula 4:

[0121] L=αL p +βL s +γL G Formula 4

[0122] Among them, α, β and γ are the weights corresponding to the photometric loss, smoothness loss and consistency loss respectively, among which α can be 0.7, β can be 0.7, and γ can be 0.3.

[0123] In yet another implementation, step C2 may include:

[0124] The photometric loss is determined according to the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.

[0125] For example, when the endoscope collects sample tissue images and historical sample tissue images, the illumination conditions may change. Therefore, SSIM (Structural Similarity) can be introduced to determine the photometric loss to avoid the interference of the change in illumination conditions on the photometric loss. SSIM can reflect the similarity of local structures. The improved photometric loss can be determined by formula 5:

[0126]

[0127] Among them, λ 1 and λ 2 They represent the preset weights respectively, and SSIM(p) represents the pixel-by-pixel SSIM between the sample tissue image and the interpolated tissue image. 1 can be 0.7, λ 2 It can be 0.3.

[0128] Furthermore, the pixel-by-pixel SSIM between the sample tissue image and the interpolated tissue image can be determined by Formula 6:

[0129]

[0130] Where x represents the image block centered on p in the sample tissue image (the size can be 3*3), y represents the image block of the same size centered on p in the interpolated tissue image, τ x represents the average value of the pixel values ​​in x, τ y represents the average value of the pixel value in y, σ x represents the standard deviation of the pixel values ​​in x, σ y Represents the standard deviation of the pixel values ​​in y. 1 and ε 2 Represents the preset constant, ε 1 For example, it can be 0.0001, ε 2 For example, it can be 0.0009.

[0131] Fig.11 is a flowchart of another method for processing an endoscopic image according to an exemplary embodiment. Fig.11 As shown, after step 105, the method may further include:

[0132] Step 106, outputting the blind area ratio, and issuing a prompt message when the blind area ratio is greater than or equal to a preset ratio threshold, the prompt message is used to indicate that there is a risk of missed detection.

[0133] For example, after determining the blind area ratio, the blind area ratio can be output, for example, the blind area ratio can be displayed in real time in the display interface for displaying tissue images, so as to display the inspection range during the endoscopic examination in real time. Further, when the blind area ratio is greater than or equal to the preset ratio threshold (for example, 20%), a prompt message can be issued to remind the doctor that there is a large blind area in the current field of view of the endoscope and there is a risk of missed detection. The presentation form of the prompt information may include: at least one of text, image, and sound. For example, the prompt information may be a text prompt or image prompt such as "the current risk of missed detection is high", "please recheck", "please withdraw the mirror", etc., and the prompt information may also be a voice prompt such as a beep of a specified frequency or an alarm sound. In this way, the doctor can adjust the direction of the endoscope, withdraw the mirror, or re-examine according to the prompt information. As a result, the blind area ratio can be monitored in real time during the doctor's endoscopic examination, and a prompt can be issued when the blind area ratio is large, so as to effectively avoid missed detection and ensure the effectiveness of the endoscopic examination.

[0134] In summary, the present disclosure first obtains tissue images collected by the endoscope at multiple collection times in the tissue to be tested. Then, based on the tissue image set, the depth image and posture parameters corresponding to each tissue image are determined. Thereafter, the motion trajectory of the endoscope is determined according to the posture parameters corresponding to each tissue image, and the contour of the tissue to be tested is determined according to the depth image corresponding to each tissue image. Finally, based on the motion trajectory and the contour of the tissue to be tested, the blind area ratio during the endoscopic inspection is determined. The present disclosure determines the motion trajectory of the endoscope and the contour of the tissue to be tested based on the depth image and posture parameters corresponding to the tissue image, and thereby determines the blind area ratio during the inspection process, which can realize the monitoring of the inspection range and effectively avoid missed inspections, thereby ensuring the effectiveness of the endoscopic inspection.

[0135] Fig.12 is a block diagram of a device for processing endoscopic images according to an exemplary embodiment. Fig.12 As shown, the apparatus 200 may include:

[0136] The acquisition module 201 is used to acquire a tissue image set acquired by an endoscope in the tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to acquisition time.

[0137] The positioning module 202 is used to determine the depth image and posture parameters corresponding to each tissue image according to the tissue image set.

[0138] The trajectory determination module 203 is used to determine the motion trajectory of the endoscope according to the posture parameters corresponding to each tissue image. The motion trajectory includes the position and angle of the endoscope when acquiring each tissue image.

[0139] The contour determination module 204 is used to determine the contour of the tissue to be measured according to the depth image corresponding to each tissue image.

[0140] The processing module 205 is used to determine the blind area ratio during the endoscopic examination process according to the motion trajectory and the contour of the tissue to be examined.

[0141] In one application scenario, the positioning module 202 may be used to:

[0142] According to each tissue image and the historical tissue image corresponding to the tissue image, the depth image and posture parameters corresponding to the tissue image are determined by a pre-trained positioning model, and the acquisition time of the historical tissue image is before the acquisition time of the tissue image.

[0143] In another application scenario, the posture parameters may include a rotation matrix and a translation vector, and the motion trajectory may include the position and angle of the endoscope when acquiring each tissue image. Accordingly, the trajectory determination module 203 may be used to:

[0144] The position and angle of the endoscope when acquiring the tissue image are determined according to the rotation matrix and translation vector corresponding to each tissue image, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image.

[0145] Fig.13 is a block diagram of another apparatus for processing endoscopic images according to an exemplary embodiment. Fig.13 As shown, the contour determination module 204 may include:

[0146] The centerline determination submodule 2041 is used to determine the centerline of the tissue to be measured according to the depth image corresponding to each tissue image.

[0147] The contour determination submodule 2042 is used to determine the contour of the tissue to be measured according to the center line of the tissue to be measured.

[0148] Fig.14 is a block diagram of another apparatus for processing endoscopic images according to an exemplary embodiment. Fig.14 As shown, the processing module 205 may include:

[0149] The field of view determination submodule 2051 is used to determine the field of view area corresponding to each tissue image according to the position and angle of the endoscope when acquiring each tissue image, and the viewing angle of the endoscope.

[0150] The total visual field determination submodule 2052 is used to determine the total visual field area according to the visual field area corresponding to each tissue image.

[0151] The blind area determination submodule 2053 is used to determine the blind area ratio according to the total visual field area and the contour of the tissue to be measured.

[0152] In one implementation, the field of view determination submodule 2051 may be used to implement the following steps:

[0153] Step 1) According to the posture parameters corresponding to each tissue image, the position of the endoscope when acquiring the tissue image is converted into a center position corresponding to the center line of the tissue to be measured.

[0154] Step 2) Determine the central viewing angle corresponding to the central position according to the posture parameters corresponding to the tissue image, the viewing angle of the endoscope, and the angle of the endoscope when acquiring the tissue image.

[0155] Step 3) Determine the maximum visual field area corresponding to the center position.

[0156] Step 4) Determine the field of view area corresponding to the tissue image according to the central viewing angle and the maximum field of view area.

[0157] In one implementation, the positioning model includes: a depth sub-model and a posture sub-model. The positioning module 202 can be used to:

[0158] The tissue image is input into the depth sub-model to obtain a depth image corresponding to the tissue image output by the depth sub-model. The tissue image and the corresponding historical tissue image are input into the posture sub-model to obtain posture parameters corresponding to the tissue image output by the posture sub-model.

[0159] In another implementation, the positioning model is trained by the following steps:

[0160] Step A: input the sample tissue image into the depth sub-model to obtain a sample depth image corresponding to the sample tissue image, and input the historical sample tissue image into the depth sub-model to obtain a historical sample depth image corresponding to the historical sample tissue image, where the historical sample tissue image is an image collected before the sample tissue image.

[0161] Step B, inputting the sample tissue image and the historical sample tissue image into the posture sub-model to obtain the sample posture parameters corresponding to the sample tissue image and the endoscope internal parameters for collecting the sample tissue image output by the posture sub-model, wherein the endoscope internal parameters include focal length and translation size.

[0162] Step C, determining the target loss based on the endoscope internal parameters, the sample depth image, the historical sample depth image and the sample posture parameters.

[0163] In step D, the positioning model is trained using the back propagation algorithm with the goal of reducing the target loss.

[0164] In yet another implementation, the implementation of step C may include:

[0165] Step C1, interpolating the historical sample tissue image according to the sample depth image, the sample posture parameter and the endoscope internal parameter to obtain an interpolated tissue image.

[0166] Step C2, determining the photometric loss according to the sample tissue image and the interpolated tissue image.

[0167] Step C3, determining the smoothing loss according to the gradient of the sample depth image and the gradient of the sample tissue image.

[0168] Step C4: transforming the sample depth image into a first depth image according to the sample posture parameters and the endoscope internal parameters.

[0169] Step C5: transforming the historical sample depth image into a second depth image according to the sample posture parameters and the endoscope internal parameters.

[0170] Step C6: determining a consistency loss according to the first depth image and the second depth image.

[0171] Step C7, determining the target loss according to the photometric loss, the smoothness loss and the consistency loss.

[0172] In yet another implementation, step C2 may include:

[0173] The photometric loss is determined according to the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.

[0174] Fig.15 is a block diagram of another apparatus for processing endoscopic images according to an exemplary embodiment. Fig.15 As shown, the device 200 may also include:

[0175] The prompt module 206 is used to output the blind area ratio after determining the blind area ratio during the endoscopic inspection process based on the motion trajectory and the contour of the tissue to be inspected, and to issue a prompt message when the blind area ratio is greater than or equal to a preset ratio threshold. The prompt message is used to indicate the risk of missed detection.

[0176] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0177] In summary, the present disclosure first obtains tissue images collected by the endoscope at multiple collection times in the tissue to be tested. Then, based on the tissue image set, the depth image and posture parameters corresponding to each tissue image are determined. Thereafter, the motion trajectory of the endoscope is determined according to the posture parameters corresponding to each tissue image, and the contour of the tissue to be tested is determined according to the depth image corresponding to each tissue image. Finally, based on the motion trajectory and the contour of the tissue to be tested, the blind area ratio during the endoscopic inspection is determined. The present disclosure determines the motion trajectory of the endoscope and the contour of the tissue to be tested based on the depth image and posture parameters corresponding to the tissue image, and thereby determines the blind area ratio during the inspection process, which can realize the monitoring of the inspection range and effectively avoid missed inspections, thereby ensuring the effectiveness of the endoscopic inspection.

[0178] Reference below Fig.16 , which shows a schematic diagram of the structure of an electronic device (e.g., the execution subject of the embodiment of the present disclosure, which may be a terminal device or a server) 300 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.16 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0179] like Fig.16 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0180] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.16The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0181] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0182] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0183] In some embodiments, the terminal devices and servers may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0184] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0185] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a tissue image set collected by the endoscope in the tissue to be tested, and the tissue image set includes multiple tissue images arranged according to the collection time; determines the depth image and posture parameters corresponding to each of the tissue images according to the tissue image set; determines the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images; determines the contour of the tissue to be tested according to the depth image corresponding to each of the tissue images; determines the blind spot ratio during the endoscopic inspection process according to the motion trajectory and the contour of the tissue to be tested.

[0186] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, an acquisition module may also be described as a "module for acquiring a tissue image set".

[0189] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0190] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] According to one or more embodiments of the present disclosure, Example 1 provides a method for processing endoscopic images, including: obtaining a tissue image set collected by an endoscope in a tissue to be tested, the tissue image set including a plurality of tissue images arranged according to the collection time; determining, based on the tissue image set, a depth image and posture parameters corresponding to each of the tissue images; determining, based on the posture parameters corresponding to each of the tissue images, a motion trajectory of the endoscope; determining, based on the depth image corresponding to each of the tissue images, a contour of the tissue to be tested; and determining, based on the motion trajectory and the contour of the tissue to be tested, a blind spot ratio during the endoscopic inspection process.

[0192] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, which determines the depth image and posture parameters corresponding to each of the tissue images based on the tissue image set, including: determining the depth image and posture parameters corresponding to the tissue image through a pre-trained positioning model based on each tissue image and the historical tissue image corresponding to the tissue image, wherein the acquisition time of the historical tissue image is before the acquisition time of the tissue image.

[0193] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the posture parameters include a rotation matrix and a translation vector, and the motion trajectory includes the position and angle of the endoscope when acquiring each of the tissue images; determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images includes: determining the position and angle of the endoscope when acquiring the tissue image according to the rotation matrix and translation vector corresponding to each of the tissue images, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image.

[0194] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, which determines the contour of the tissue to be measured based on the depth image corresponding to each of the tissue images, including: determining the center line of the tissue to be measured based on the depth image corresponding to each of the tissue images; determining the contour of the tissue to be measured based on the center line of the tissue to be measured.

[0195] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, which determines the blind area ratio in the endoscopic inspection process based on the motion trajectory and the contour of the tissue to be measured, including: determining the field of view area corresponding to the tissue image based on the position and angle of the endoscope when acquiring each tissue image, and the viewing angle of the endoscope; determining the total field of view area based on the field of view area corresponding to each tissue image; determining the blind area ratio based on the total field of view area and the contour of the tissue to be measured.

[0196] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, which determines the field of view area corresponding to the tissue image based on the position and angle of the endoscope when acquiring each of the tissue images, and the viewing angle of the endoscope, including: converting the position of the endoscope when acquiring the tissue image to a center position corresponding to the center line of the tissue to be measured based on the posture parameters corresponding to each of the tissue images; determining the center viewing angle corresponding to the center position based on the posture parameters corresponding to the tissue image, the viewing angle of the endoscope, and the angle of the endoscope when acquiring the tissue image; determining the maximum field of view area corresponding to the center position; and determining the field of view area corresponding to the tissue image based on the center viewing angle and the maximum field of view area.

[0197] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 2, wherein the positioning model includes: a depth sub-model and a posture sub-model; the depth image and posture parameters corresponding to the tissue image are determined in turn according to each tissue image and the historical tissue image corresponding to the tissue image through a pre-trained positioning model, including: inputting the tissue image into the depth sub-model to obtain the depth image corresponding to the tissue image output by the depth sub-model; inputting the tissue image and the corresponding historical tissue image into the posture sub-model to obtain the posture parameters corresponding to the tissue image output by the posture sub-model.

[0198] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7, wherein the positioning model is trained by the following steps: inputting a sample tissue image into the depth sub-model to obtain a sample depth image corresponding to the sample tissue image, and inputting a historical sample tissue image into the depth sub-model to obtain a historical sample depth image corresponding to the historical sample tissue image, wherein the historical sample tissue image is an image acquired before the sample tissue image; inputting the sample tissue image and the historical sample tissue image into the posture sub-model to obtain sample posture parameters corresponding to the sample tissue image and endoscope internal parameters for acquiring the sample tissue image, output by the posture sub-model, wherein the endoscope internal parameters include focal length and translation size; determining the target loss based on the endoscope internal parameters, the sample depth image, the historical sample depth image and the sample posture parameters; and training the positioning model using a back propagation algorithm with the goal of reducing the target loss.

[0199] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 8, wherein the target loss is determined according to the endoscope internal parameters, the sample depth image, the historical sample depth image and the sample posture parameters, including: interpolating the historical sample tissue image according to the sample depth image, the sample posture parameters and the endoscope internal parameters to obtain an interpolated tissue image; determining the photometric loss according to the sample tissue image and the interpolated tissue image; determining the smoothing loss according to the gradient of the sample depth image and the gradient of the sample tissue image; transforming the sample depth image into a first depth image according to the sample posture parameters and the endoscope internal parameters; transforming the historical sample depth image into a second depth image according to the sample posture parameters and the endoscope internal parameters; determining the consistency loss according to the first depth image and the second depth image; and determining the target loss according to the photometric loss, the smoothing loss and the consistency loss.

[0200] According to one or more embodiments of the present disclosure, Example 10 provides the method of Example 9, wherein determining the photometric loss based on the sample tissue image and the interpolated tissue image includes: determining the photometric loss based on the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.

[0201] According to one or more embodiments of the present disclosure, Example 11 provides the method of Examples 1 to 10, after determining the blind area ratio in the endoscopic inspection process based on the motion trajectory and the contour of the tissue to be tested, the method further includes: outputting the blind area ratio, and issuing a prompt message when the blind area ratio is greater than or equal to a preset ratio threshold, wherein the prompt message is used to indicate the existence of a risk of missed detection.

[0202] According to one or more embodiments of the present disclosure, Example 12 provides an endoscopic image processing device, comprising: an acquisition module, used to acquire a tissue image set collected by an endoscope in a tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to the collection time; a positioning module, used to determine, based on the tissue image set, a depth image and posture parameters corresponding to each of the tissue images; a trajectory determination module, used to determine a motion trajectory of the endoscope based on the posture parameters corresponding to each of the tissue images; a contour determination module, used to determine the contour of the tissue to be tested based on the depth image corresponding to each of the tissue images; and a processing module, used to determine the blind spot ratio during the endoscopic inspection process based on the motion trajectory and the contour of the tissue to be tested.

[0203] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the methods described in Examples 1 to 11 when executed by a processing device.

[0204] According to one or more embodiments of the present disclosure, Example 14 provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of the method described in Examples 1 to 11.

[0205] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0206] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0207] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for processing an endoscopic image, characterized in that: The method comprises: Acquire a tissue image set collected by an endoscope in the tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to the collection time; Determine, according to the tissue image set, a depth image and a posture parameter corresponding to each of the tissue images; Determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images; Determining the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images; Determining the blind area ratio during the endoscopic examination according to the motion trajectory and the contour of the tissue to be examined; The posture parameters include a rotation matrix and a translation vector, and the motion trajectory includes the position and angle of the endoscope when acquiring each of the tissue images; determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images includes: Determine the position and angle of the endoscope when acquiring the tissue image according to the rotation matrix and translation vector corresponding to each tissue image, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image; Determining the contour of the tissue to be measured according to the depth image corresponding to each tissue image includes: Determining the center line of the tissue to be tested according to the depth image corresponding to each of the tissue images; Determining the contour of the tissue to be measured according to the center line of the tissue to be measured; Determining the blind area ratio during the endoscopic inspection process according to the motion trajectory and the contour of the tissue to be inspected includes: Determine the field of view area corresponding to each tissue image according to the position and angle of the endoscope when acquiring each tissue image and the viewing angle of the endoscope; Determine a total visual field area according to the visual field area corresponding to each of the tissue images; The blind area ratio is determined according to the total visual field area and the contour of the tissue to be measured.

2. The method according to claim 1, characterized in that Determining the depth image and posture parameters corresponding to each tissue image according to the tissue image set includes: According to each tissue image and the historical tissue image corresponding to the tissue image, the depth image and posture parameters corresponding to the tissue image are determined by a pre-trained positioning model, and the acquisition time of the historical tissue image is before the acquisition time of the tissue image.

3. The method according to claim 1, characterized in that Determining the field of view area corresponding to each tissue image according to the position and angle of the endoscope when acquiring each tissue image and the viewing angle of the endoscope includes: According to the posture parameters corresponding to each tissue image, the position of the endoscope when acquiring the tissue image is converted into a center position corresponding to the center line of the tissue to be measured; Determining a central viewing angle corresponding to the central position according to a posture parameter corresponding to the tissue image, a viewing angle of the endoscope, and an angle of the endoscope when acquiring the tissue image; Determine the maximum visual field area corresponding to the center position; The field of view area corresponding to the tissue image is determined according to the central viewing angle and the maximum field of view area.

4. The method according to claim 2, characterized in that: The positioning model includes: a depth sub-model and a posture sub-model; The method of determining the depth image and posture parameters corresponding to the tissue image by a pre-trained positioning model according to each tissue image and the historical tissue image corresponding to the tissue image in turn includes: Inputting the tissue image into the depth sub-model to obtain a depth image corresponding to the tissue image output by the depth sub-model; The tissue image and the corresponding historical tissue image are input into the posture sub-model to obtain the posture parameters corresponding to the tissue image output by the posture sub-model.

5. The method according to claim 4, characterized in that The positioning model is trained by the following steps: Inputting a sample tissue image into the depth sub-model to obtain a sample depth image corresponding to the sample tissue image, and inputting a historical sample tissue image into the depth sub-model to obtain a historical sample depth image corresponding to the historical sample tissue image, wherein the historical sample tissue image is an image acquired before the sample tissue image; Inputting the sample tissue image and the historical sample tissue image into the posture sub-model to obtain the sample posture parameters corresponding to the sample tissue image and the endoscope internal parameters corresponding to the sample tissue image output by the posture sub-model, wherein the endoscope internal parameters include focal length and translation size; Determining target loss according to the endoscope internal parameters, the sample depth image, the historical sample depth image and the sample posture parameters; With the goal of reducing the target loss, the positioning model is trained using a back propagation algorithm.

6. The method according to claim 5, characterized in that The determining of the target loss according to the endoscope internal parameters, the sample depth image, the historical sample depth image and the sample posture parameters comprises: interpolating the historical sample tissue image according to the sample depth image, the sample posture parameter and the endoscope internal parameter to obtain an interpolated tissue image; determining a photometric loss based on the sample tissue image and the interpolated tissue image; Determining a smoothing loss according to a gradient of the sample depth image and a gradient of the sample tissue image; transforming the sample depth image into a first depth image according to the sample posture parameter and the endoscope internal parameter; transforming the historical sample depth image into a second depth image according to the sample posture parameter and the endoscope internal parameter; determining a consistency loss based on the first depth image and the second depth image; The target loss is determined according to the photometric loss, the smoothness loss and the consistency loss.

7. The method according to claim 6, characterized in that The determining of the photometric loss according to the sample tissue image and the interpolated tissue image comprises: The photometric loss is determined according to the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.

8. The method according to any one of claims 1 to 7, characterized in that After determining the blind area ratio during the endoscopic examination according to the motion trajectory and the contour of the tissue to be examined, the method further includes: The blind area ratio is output, and when the blind area ratio is greater than or equal to a preset ratio threshold, a prompt message is issued, wherein the prompt message is used to indicate that there is a risk of missed detection.

9. An endoscope image processing device, characterized in that: The device comprises: An acquisition module, used to acquire a tissue image set acquired by the endoscope in the tissue to be tested, wherein the tissue image set includes a plurality of tissue images arranged according to acquisition time; A positioning module, used for determining a depth image and a posture parameter corresponding to each of the tissue images according to the tissue image set; A trajectory determination module, used to determine the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images; A contour determination module, used to determine the contour of the tissue to be tested according to the depth image corresponding to each of the tissue images; A processing module, used for determining the blind area ratio during the endoscopic examination process according to the motion trajectory and the contour of the tissue to be examined; The posture parameters include a rotation matrix and a translation vector, the motion trajectory includes the position and angle of the endoscope when acquiring each of the tissue images, and the trajectory determination module is further used to determine the position and angle of the endoscope when acquiring the tissue image according to the rotation matrix and translation vector corresponding to each of the tissue images, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image; The contour determination module is further used to determine the center line of the tissue to be tested according to the depth image corresponding to each tissue image; and determine the contour of the tissue to be tested according to the center line of the tissue to be tested; The processing module is also used to determine the field of view area corresponding to the tissue image based on the position and angle of the endoscope when acquiring each tissue image, and the viewing angle of the endoscope; determine the total field of view area based on the field of view area corresponding to each tissue image; and determine the blind area ratio based on the total field of view area and the contour of the tissue to be measured.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

Images