Digestive tract modeling method, system and device and modeling equipment

Abstract

The invention relates to the technical field of image processing, and provides an alimentary canal modeling method, system and device and modeling equipment. The method comprises the following steps: detecting an alimentary canal of a to-be-detected object by using an endoscope; acquiring depth data of the insertion part of the endoscope inserted into the alimentary canal shot by the first external camera, a focus event identified based on an image shot by the second external camera, and position data of the head end part of the insertion part in the alimentary canal; according to the depth data, the position data and the focus event, determining the focus position in the alimentary canal; and according to the depth data, the position data and the focus position, constructing a digestive tract three-dimensional model for marking the focus position. By adopting the method, the accuracy of modeling the digestive tract and the accuracy of focus positioning can be improved, so that the reference value provided by the digestive tract three-dimensional model for doctors is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a digestive tract modeling method, system, device and modeling equipment. Background Art

[0002] In order to better assist doctors in examining the digestive tract, it is possible to combine the image data of the digestive tract captured by the camera at the end of the endoscope lens to construct a three-dimensional model of the digestive tract for the doctor's reference. However, in actual applications, it has been found that the three-dimensional models constructed based on image data generally lack accuracy, resulting in extremely limited reference value of the three-dimensional models for doctors. On this basis, if the doctor wants to use the three-dimensional model to record the location of the lesion during the examination, the three-dimensional model must be manually marked after the lesion is found. On the one hand, this will affect the doctor's normal intraoperative operation and is not conducive to the doctor's normal examination. On the other hand, due to the lack of accuracy of the three-dimensional model itself, and the problem of marking accuracy in the doctor's manual marking process itself, the credibility of the final lesion marking result is low, and its reference value to doctors is extremely limited. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a digestive tract modeling method, system, device and modeling equipment, which can improve the accuracy of digestive tract modeling and the accuracy of lesion positioning, thereby enhancing the reference value of the digestive tract three-dimensional model provided to doctors.

[0004] A first aspect of an embodiment of the present application provides a digestive tract modeling method, comprising:

[0005] During the process of examining the digestive tract of a subject using an endoscope, obtaining data on the depth of insertion of the insertion portion of the endoscope into the digestive tract captured by a first external camera, lesion events identified based on images captured by a second external camera, and position data on the tip of the insertion portion within the digestive tract;

[0006] determining the location of the lesion in the digestive tract based on the depth data, the location data, and the lesion event;

[0007] Based on the depth data, position data and lesion location, a three-dimensional model of the digestive tract is constructed with the lesion location marked.

[0008] In the embodiment of the present application, two external cameras are set, which are respectively recorded as the first external camera and the second external camera, wherein the first external camera is used to capture the depth data of the insertion portion of the endoscope inserted into the digestive tract, and the second external camera is used to capture the lesion event; in the process of using the endoscope to detect the digestive tract of the object to be tested, the depth data captured by the first external camera, the position data of the front end of the insertion portion in the digestive tract, and the lesion event identified based on the image captured by the second external camera are obtained, and then the lesion position of the digestive tract is determined based on the depth data, position data and lesion event, and finally a three-dimensional model of the digestive tract with the lesion position marked is constructed. In the above process, since the depth data can represent the true depth of the insertion portion of the endoscope inserted into the digestive tract, and the position data can reflect the true position reached by the insertion portion, the combination of the two can accurately reflect the true spatial movement of the head end, so the depth data and position data can be used to accurately quantify the distance and length of each position point of the digestive tract three-dimensional model, thereby realizing accurate three-dimensional modeling of the digestive tract. On this basis, the true position point of the lesion is found in combination with the identified lesion event, thereby improving the accuracy and credibility of lesion positioning. Finally, based on this authentic and reliable data, a 3D model of the digestive tract is constructed and the lesion locations are marked, enabling precise positioning and labeling of the lesions. This entire process requires no manual intervention from the doctor, as model building and lesion labeling are based on authentic and accurate data. Therefore, it offers excellent reference value and can more effectively assist doctors in their operations.

[0009] Furthermore, the depth data and lesion events in the embodiments of this application are sourced from external cameras outside the endoscope system, meaning that only the position data from the endoscope tip is required. Therefore, the software and hardware requirements for the endoscope system are extremely low, and no additional hardware is required. Therefore, the system is compatible with a wide range of endoscope systems from different manufacturers, types, and models, demonstrating strong compatibility and significant practical and promotional value.

[0010] In one implementation of the embodiment of the present application, a scale line for marking the insertion depth is provided on the insertion portion; the depth data is obtained by:

[0011] acquiring a first image of the insertion portion inserted into the digestive tract, captured by a first external camera;

[0012] The depth data is determined based on the scale line on the insertion portion in the first image that is closest to the insertion port of the digestive tract.

[0013] In one implementation of the embodiment of the present application, the focus event is determined by:

[0014] acquiring a second image of a display screen of the endoscope system captured by a second external camera;

[0015] If the display screen in the second image shows a frozen screen or specified status information, it is determined that a focus event has occurred.

[0016] Furthermore, if the display screen in the second image freezes or displays a specified status information, it is determined that a focus event has occurred, including:

[0017] If the display screen in the second image shows a frozen screen or a specified status information, a third image of the operation action of the operator of the endoscope captured by the second external camera is acquired;

[0018] If the operation action in the third image is a designated action, it is determined that a lesion event occurs.

[0019] In one implementation of the embodiment of the present application, determining the location of a lesion in the digestive tract based on the depth data, the location data, and the lesion event includes:

[0020] Based on the lesion event, determine the target moment for lesion discovery;

[0021] Searching for a first depth corresponding to a target time from the depth data, and searching for a target position corresponding to the target time from the position data;

[0022] The lesion position is determined according to the first depth and the target position.

[0023] In one implementation of the embodiment of the present application, after constructing the three-dimensional digestive tract model with the lesion location marked, the method further includes:

[0024] During the review of the subject to be tested, determining a second depth corresponding to the location of the lesion marked on the three-dimensional model of the digestive tract;

[0025] The control insertion portion is inserted into the digestive tract to a position corresponding to the second depth.

[0026] In one implementation of the embodiment of the present application, before constructing the three-dimensional digestive tract model with the lesion location marked, the method further includes:

[0027] Acquiring image data of the digestive tract captured by an endoscope camera;

[0028] Identify information about various parts of the digestive tract based on image data;

[0029] Construct a 3D model of the digestive tract with the lesion location marked, including:

[0030] Based on the site information, depth data and position data, a three-dimensional model of the digestive tract is constructed, which records the digestive tract site and marks the location of the lesion.

[0031] A second aspect of an embodiment of the present application provides a digestive tract modeling system, comprising: a modeling device, a first external camera and a second external camera;

[0032] During the process of examining the digestive tract of a subject using an endoscope:

[0033] The first external camera is used to capture a first image of the insertion portion of the endoscope inserted into the digestive tract;

[0034] The second external camera is used to capture a second image of the display screen of the endoscope system;

[0035] The modeling device is used to determine the depth data of the insertion portion into the digestive tract based on the first image, and to determine the lesion event based on the second image;

[0036] The modeling device is also used to determine the location of the lesion in the digestive tract based on the depth data, lesion events and the position data of the head end of the insertion part in the digestive tract; and to construct a three-dimensional model of the digestive tract that indicates the lesion location based on the depth data, position data and lesion location.

[0037] In one implementation of the embodiment of the present application, a scale line for marking the insertion depth is provided on the insertion portion; the modeling device is used to determine the depth data based on the scale line on the insertion portion in the first image that is closest to the insertion port of the digestive tract.

[0038] In one implementation of the embodiment of the present application, the modeling device is configured to determine that a lesion event has occurred if a display screen in the second image freezes or displays specified status information.

[0039] In one implementation of an embodiment of the present application, the second external camera is also used to capture a third image of the operating action of the endoscope operator; the modeling device is also used to determine the occurrence of a lesion event if the operating action in the third image is a specified action.

[0040] In one implementation of an embodiment of the present application, the modeling device is also used to determine the target moment of lesion discovery based on the lesion event; search for a first depth corresponding to the target moment from the depth data, and search for a target position corresponding to the target moment from the position data; and determine the lesion position based on the first depth and the target position.

[0041] In one implementation of an embodiment of the present application, the modeling device is also used to determine a second depth corresponding to the lesion location marked on the three-dimensional model of the digestive tract during a review of the object to be tested; and control the insertion part to be inserted into the position corresponding to the second depth in the digestive tract.

[0042] In one implementation of the embodiment of the present application, the modeling device is also used to obtain image data of the digestive tract captured by the camera of an endoscope; identify information of various parts of the digestive tract based on the image data; and construct a three-dimensional model of the digestive tract that records the parts of the digestive tract and marks the location of the lesion based on the part information, depth data, and position data.

[0043] A third aspect of the embodiments of the present application provides a digestive tract modeling device, comprising:

[0044] a data acquisition module for acquiring, during the process of examining the digestive tract of a subject using an endoscope, data on the depth of insertion of the insertion portion of the endoscope into the digestive tract captured by a first external camera, lesion events identified based on images captured by a second external camera, and position data on the tip of the insertion portion within the digestive tract;

[0045] a lesion location determination module, configured to determine the location of a lesion in the digestive tract based on the depth data, the location data, and the lesion event;

[0046] The model building module is used to build a three-dimensional model of the digestive tract with the lesion location marked based on the depth data, position data and lesion location.

[0047] The fourth aspect of an embodiment of the present application provides a modeling device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the digestive tract modeling method provided in the first aspect of an embodiment of the present application is implemented.

[0048] A fifth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the digestive tract modeling method provided in the first aspect of the embodiments of the present application.

[0049] A sixth aspect of the embodiments of the present application provides a computer program product, which, when executed on a modeling device, enables the modeling device to execute the digestive tract modeling method provided in the first aspect of the embodiments of the present application.

[0050] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of a digestive tract modeling system provided in an embodiment of the present application;

[0052] Figure 2 This is a flow chart of a digestive tract modeling method provided in an embodiment of the present application;

[0053] Figure 3 Schematic diagram of setting scale lines on the insertion portion of the endoscope provided in an embodiment of the present application;

[0054] Figure 4 Schematic diagram of a stereoscopic display inspection system for endoscopes provided in an embodiment of the present application;

[0055] Figure 5 This is a schematic diagram of implementing a split-screen display function using an endoscope display inspection system provided in an embodiment of the present application;

[0056] Figure 6 Schematic diagram of the structure of a digestive tract modeling device provided in an embodiment of the present application;

[0057] Figure 7 This is a schematic diagram of a modeling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details. In addition, in the description of the present application specification and the appended claims, the terms "first," "second," "third," etc. are only used to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0059] A 3D model of the digestive tract allows doctors to visualize the structure of various parts of the tract and manually mark the locations of possible lesions within the 3D model. However, current 3D models of the digestive tract are typically derived from image data captured by a camera at the end of an endoscope, combined with image processing algorithms. These 3D models are not correlated to the actual length of the digestive tract, and their accuracy and reliability are low. Doctors can typically only use these 3D models for general reference and cannot guide specific, detailed procedures, making them extremely limited in value.

[0060] In view of this, the embodiments of the present application provide a digestive tract modeling method, system, apparatus, and modeling device that can improve the accuracy of digestive tract modeling and lesion localization, thereby enhancing the reference value of the digestive tract three-dimensional model for doctors. For more specific technical implementation details of the embodiments of the present application, please refer to the various embodiments described below.

[0061] See also Figure 1, shows a digestive tract modeling system provided by an embodiment of the present application, which includes a modeling device, a first external camera and a second external camera, and the two external cameras can be connected to the modeling device in a wired or wireless manner. Among them, the first external camera and the second external camera can be cameras of any type and model. By adjusting the shooting angles of the two external cameras, the first external camera is used to capture the first image of the insertion part of the endoscope inserted into the digestive tract, and the second external camera is used to capture the second image of the display screen of the endoscope system or the doctor's operating action image. The modeling device can be any type of electronic device, for example, it can be an image processing device or an AI box of the endoscope system. At this time, the image data captured by the two external cameras can be directly transmitted to the endoscope system for processing, or the image data can be processed by other independent devices and then fed back to the endoscope system. The modeling device can also be an independent device outside the endoscope system, which can obtain the position data and image data of the head end of the endoscope insertion part, and can also obtain the image data captured by the two external cameras. The modeling device is used to determine the depth data of the insertion part inserted into the digestive tract based on the first image, determine the lesion event based on the second image and / or the doctor's operation action image, and then determine the lesion position in the digestive tract based on the depth data, lesion event and the position data of the head end of the insertion part in the digestive tract. Finally, based on the depth data, position data and lesion position, a three-dimensional model of the digestive tract is constructed with the lesion position marked.

[0062] In one implementation of the embodiment of the present application, a scale line for marking the insertion depth is provided on the insertion portion; the modeling device is used to determine the depth data based on the scale line on the insertion portion in the first image that is closest to the insertion port of the digestive tract.

[0063] In one implementation of the embodiment of the present application, the modeling device is configured to determine that a lesion event has occurred if a display screen in the second image freezes or displays specified status information.

[0064] In one implementation of an embodiment of the present application, the second external camera is also used to capture a third image of the operating action of the endoscope operator; the modeling device is also used to determine the occurrence of a lesion event if the operating action in the third image is a specified action.

[0065] In one implementation of an embodiment of the present application, the modeling device is also used to determine the target moment of lesion discovery based on the lesion event; search for a first depth corresponding to the target moment from the depth data, and search for a target position corresponding to the target moment from the position data; and determine the lesion position based on the first depth and the target position.

[0066] In one implementation of an embodiment of the present application, the modeling device is also used to determine a second depth corresponding to the lesion location marked on the three-dimensional model of the digestive tract during a review of the object to be tested; and control the insertion part to be inserted into the position corresponding to the second depth in the digestive tract.

[0067] In one implementation of the embodiment of the present application, the modeling device is also used to obtain image data of the digestive tract captured by the camera of an endoscope; identify information of various parts of the digestive tract based on the image data; and construct a three-dimensional model of the digestive tract that records the parts of the digestive tract and marks the location of the lesion based on the part information, depth data, and position data.

[0068] about Figure 1 For more specific working principles and descriptions of the system shown, please refer to the method embodiments described below.

[0069] It should be understood that the execution subjects of the various method embodiments of the present application are various types of modeling devices, such as image processing devices of endoscope systems, AI boxes, or any type of independent devices outside the endoscope system, such as mobile phones, tablet computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of the present application do not impose any restrictions on the specific type of the modeling device.

[0070] See also Figure 2 , showing a digestive tract modeling method provided in an embodiment of the present application, comprising:

[0071] 201. During an endoscope examination of a subject's digestive tract, obtaining depth data of an insertion portion of the endoscope inserted into the digestive tract captured by a first external camera, lesion events identified based on images captured by a second external camera, and position data of a tip of the insertion portion within the digestive tract;

[0072] In an embodiment of the present application, the object to be tested may be a human or other animal having a digestive tract in the body, for example, a patient with a disease in the digestive tract. In the process of using an endoscope to detect the digestive tract of the object to be tested, the insertion portion of the endoscope will extend into the digestive tract through the anus and other parts of the object to be tested. The positioning sensor provided at the head end of the insertion portion can collect the position data of the head end of the insertion portion in the digestive tract, such as the three-dimensional spatial coordinates of the head end in the digestive tract at each moment. The first external camera can be used to capture the depth data of the insertion portion of the endoscope inserted into the digestive tract, that is, the actual depth of the insertion portion inserted into the digestive tract. Specifically, the shooting angle of the first external camera can be adjusted to capture image data of the insertion portion of the endoscope inserted into the digestive tract. By analyzing the image data, the length of the insertion portion inserted into the digestive tract can be identified, thereby obtaining the depth data.

[0073] In one implementation of the embodiment of the present application, a scale line for marking the insertion depth is provided on the insertion portion; the depth data is obtained by:

[0074] (1) obtaining a first image of the insertion portion inserted into the digestive tract, captured by a first external camera;

[0075] (2) Determine depth data based on the scale line on the insertion portion in the first image that is closest to the insertion port of the digestive tract.

[0076] like Figure 3 As shown, a scale line for marking the insertion depth can be set on the insertion part of the endoscope. First, the first external camera is used to capture image data of the insertion part inserted into the digestive tract, which is recorded as the first image; then, the first image is analyzed to find the scale line on the insertion part in the first image that is closest to the insertion port of the digestive tract. Based on the scale line, the actual depth of the insertion part into the digestive tract can be determined, thereby obtaining the corresponding depth data. It can be understood that in the process of using an endoscope to detect the digestive tract of the object to be tested, the actual depth of the insertion part into the digestive tract is constantly changing. During this process, the first external camera will continuously capture multiple frames of images with timestamps. By identifying the scale lines in each frame of the image, the depth data of the insertion part into the digestive tract at each moment can be obtained.

[0077] The second external camera is mainly used to capture lesion events. Considering that the doctor will continuously observe the display screen of the endoscope system when using the endoscope to examine the digestive tract of the subject, if a suspicious lesion phenomenon is found in the screen, the doctor will usually perform a specified operation, such as manually pressing the freeze button of the endoscope to freeze the display screen, so as to observe the screen where the lesion may be more carefully. Therefore, the shooting angle of the second external camera can be adjusted to make it aim at the display screen of the endoscope system or the doctor's hand. Then, based on the image of the display screen or the image of the doctor's hand operation, it can be analyzed to determine whether a lesion event has occurred.

[0078] In one implementation of the embodiment of the present application, the focus event is determined by:

[0079] (1) acquiring a second image of a display screen of the endoscope system captured by a second external camera;

[0080] (2) If the display screen in the second image shows a frozen screen or a specified status information, it is determined that a lesion event has occurred.

[0081] In one implementation, the second external camera's shooting angle is aligned with the endoscope system's display screen. While the endoscope is being used to examine the digestive tract of the subject, the second external camera continuously captures image data from the display screen, which is recorded as a second image. The second image is analyzed. If it is found that the display screen has not changed or has changed very little over multiple consecutive frames, it can be determined that the display screen has frozen. In this case, it can be assumed that the doctor has discovered a lesion and frozen the screen for careful observation, thus determining that a lesion event has occurred. Alternatively, if a specified status message is found on the display screen of the second image, such as displaying "screen frozen state," or displaying the status message "suspicious lesion found" when a suspicious lesion is identified using an image target detection method, it can also be considered that a lesion event has occurred. When a lesion event is determined to have occurred based on the second image, the target moment of the lesion discovery event can be recorded. This target moment can be used to determine the lesion location in subsequent processing steps.

[0082] In another implementation of the embodiment of the present application, the focus event is determined by:

[0083] (1) acquiring a third image of an operation action of an operator of the endoscope captured by a second external camera;

[0084] (2) If the operation action in the third image is a designated action, it is determined that a lesion event occurs.

[0085] In another implementation, a second external camera is aimed at the hand of the endoscope operator (e.g., a doctor). While the endoscope is being used to examine the digestive tract of a subject, the second external camera continuously captures the operator's movements, which are recorded as third images. The third images are analyzed to identify the operator's movements. If the movement is a designated action, such as pressing the endoscope's freeze button or rotation button, the doctor is deemed to have discovered a lesion, and a lesion event is determined.

[0086] In another implementation of the embodiment of the present application, the focus event is determined by:

[0087] (1) acquiring a second image of a display screen of the endoscope system captured by a second external camera;

[0088] (2) if the display screen in the second image shows a frozen screen or a specified status information, obtaining a third image of the operation action of the operator of the endoscope captured by the second external camera;

[0089] (3) If the operation action in the third image is a designated action, it is determined that a lesion event occurs.

[0090] Since determining whether a lesion event has occurred solely based on the doctor's hand movements is not highly accurate, the accuracy of lesion event recognition can be improved by combining the display image and the doctor's hand movements. Specifically, the shooting range of the second external camera can be appropriately increased to cover the endoscope system's display image and the doctor's hand. This allows the second external camera to simultaneously capture both the display image and the doctor's hand movements, in which case the second and third images are the same image data. Alternatively, the shooting angle of the second external camera can be continuously switched back and forth between the display image and the doctor's hand, capturing the display image and the doctor's hand movements at intervals. In this case, the second and third images are different image data. Alternatively, a third external camera can be added, with the second and third external cameras capturing the display image and the doctor's hand movements, respectively. After obtaining the second and third images, a lesion event is determined to have occurred only if the display image in the second image shows a frozen screen or a specified status message, and the hand movements in the third image are the specified actions.

[0091] 202. Determine the location of the lesion in the digestive tract based on the depth data, the location data, and the lesion event;

[0092] After acquiring the aforementioned depth data, position data, and lesion events, the location of the lesion in the digestive tract can be determined based on these data. Since the depth data indicates the true depth of the endoscope's insertion portion into the digestive tract, and the position data reflects the true position reached by the insertion portion, combining the depth data and position data corresponding to the lesion event can determine the true depth of the insertion portion's insertion into the digestive tract and the true position reached by the insertion portion, thereby determining the lesion's location.

[0093] In one implementation of the embodiment of the present application, determining the location of a lesion in the digestive tract based on the depth data, the location data, and the lesion event includes:

[0094] (1) Determine the target time for lesion discovery based on the lesion event;

[0095] (2) searching the first depth corresponding to the target time from the depth data, and searching the target position corresponding to the target time from the position data;

[0096] (3) Determine the lesion location based on the first depth and the target location.

[0097] In the aforementioned step 201, when a lesion event is determined to have occurred, the timestamp of the corresponding second image or third image can be recorded as the target time when the doctor discovered the lesion. Then, the first depth corresponding to the target time is searched from the depth data, and the target position corresponding to the target time is searched from the position data, wherein the first depth represents the actual depth of the insertion portion corresponding to the lesion inserted into the digestive tract, and the target position represents the actual position reached by the insertion portion corresponding to the lesion. Therefore, the lesion position can be determined based on the first depth and the target position. For example, assuming that the target time for discovering the lesion determined based on the lesion event is T1, the depth data corresponding to time T1 is searched from the depth data, and the position data corresponding to time T1 is searched from the position data. Combining the depth data and position data corresponding to time T1, the true position of the lesion can be determined. Similarly, if there are multiple lesion events, assuming there are n, the target times of each of these lesion events are obtained respectively, and the depth data and position data corresponding to each target time are searched respectively, so that n lesion positions can be determined.

[0098] 203. Based on the depth data, position data and lesion location, a three-dimensional model of the digestive tract is constructed with the lesion location marked.

[0099] After determining the location of the lesion, a three-dimensional model of the digestive tract is constructed, indicating the location of the lesion, based on the depth data, position data, and lesion location. Since the depth data can indicate the true depth of the insertion portion of the endoscope into the digestive tract, and the position data can reflect the true position reached by the insertion portion, the combination of the two can accurately reflect the true spatial movement of the head end. Therefore, the depth data and position data can be used to accurately quantify the distance and length of each position point of the three-dimensional model of the digestive tract, thereby achieving accurate three-dimensional modeling of the digestive tract. On this basis, the true location point of the lesion is found in combination with the identified lesion event, thereby improving the accuracy and reliability of lesion positioning. Finally, based on these real and reliable data, a three-dimensional model of the digestive tract is constructed and the lesion location is marked, which can achieve accurate positioning and marking of the lesion location.

[0100] In addition, if the model is built based only on position data, only the position of the insertion part in the digestive tract can be known, but the actual depth of the insertion part in the digestive tract cannot be known. Considering that the shape of the digestive tract is tortuous, the actual spatial movement of the head of the insertion part cannot be determined by position data alone. In this way, the distance and length of each position point cannot be accurately quantified, and the accuracy of the obtained three-dimensional model is not high.

[0101] In one implementation of the embodiment of the present application, before constructing the three-dimensional digestive tract model with the lesion location marked, the method further includes:

[0102] (1) Obtaining image data of the digestive tract captured by an endoscope camera;

[0103] (2) Identify information about various parts of the digestive tract based on image data.

[0104] Accordingly, a three-dimensional model of the digestive tract is constructed to indicate the location of the lesion, including:

[0105] Based on the site information, depth data and position data, a three-dimensional model of the digestive tract is constructed, which records the digestive tract site and marks the location of the lesion.

[0106] To further enhance the reference value of the constructed three-dimensional model of the digestive tract, image data of the digestive tract captured by an endoscope camera can be obtained. Based on this image data, information about the various parts of the digestive tract can be identified, resulting in different digestive tract locations, such as the duodenum, jejunum, ileum, cecum, colon, rectum, and anal canal. Specifically, by extracting image features from the digestive tract image data and analyzing and processing these features to identify the digestive tract location to which they correspond, information about each part of the digestive tract can be determined. In this way, when constructing a three-dimensional model of the digestive tract, a three-dimensional model can be obtained that records the digestive tract locations and indicates the location of lesions. When constructing the three-dimensional model, existing model construction algorithms can be used, such as LBP feature extraction, SVM classifiers, and the more efficient CBP feature extraction, utilizing global data for feature extraction to ultimately obtain a three-dimensional model of the digestive tract. By examining this three-dimensional model, doctors can intuitively and accurately locate the location of lesions in the digestive tract and the part of the digestive tract where the lesions are located.

[0107] As an example, in an intraoperative scenario, when a biopsy is to be performed at a certain location in the digestive tract, the three-dimensional digestive tract models constructed using existing technologies can typically only help doctors roughly locate the biopsy site, but cannot determine the precise location of the biopsy. However, if the three-dimensional digestive tract model constructed using the method provided in the embodiments of the present application can accurately quantify the distance and length of each location point, it can determine the precise location of the biopsy.

[0108] The present application also provides a stereoscopic display inspection system for endoscopes, such as Figure 4 As shown, the system may be composed of a stereoscopic display structure system and an endoscope system. Among them, the stereoscopic display structure system may include a camera system and a monitoring system. The camera system may include the first external camera and the second external camera described above, which are used to collect image data of the insertion part of the endoscope inserted into the digestive tract and image data of the endoscope display screen. The monitoring system can be used to display stereoscopic content and diagnostic images, and use a capturer to locate and capture the doctor's gestures or cursor to complete the interaction of the touch light field. For example, the constructed three-dimensional model of the digestive tract can be displayed by the monitoring system, and the doctor's operating gestures can be captured by the capturer, thereby realizing various operating actions such as adjusting the viewing angle of the three-dimensional model of the digestive tract. The endoscope system mainly includes an image processing device of the endoscope, a light source host and an endoscope body. The body mainly includes structures such as an insertion part, an operating end and a connecting end. The head end of the insertion part is provided with a positioning sensor. In addition, as Figure 5 As shown, the monitoring system can also realize the split-screen display function, which can simultaneously display the three-dimensional model of the digestive tract and the image captured by the endoscope camera on one screen so that doctors can perform comparative analysis.

[0109] After the constructed three-dimensional model of the digestive tract is displayed by the monitoring system, the suspected lesions, confirmed lesions and potential lesions of the subject to be tested can be marked on the three-dimensional model of the digestive tract. Different types of lesions, such as suspected lesions, confirmed lesions, lesions of different severity, and dangerous lesions that may break out, can be marked with different colors or shapes, such as a red cross-shaped mark to indicate a serious lesion, so that doctors can distinguish them. The lesion condition marking here can be done by the doctor manually marking each lesion location on the three-dimensional model of the digestive tract according to the marked lesion location, or by using lesion image feature detection to automatically identify and mark the lesion condition mark corresponding to each lesion location.

[0110] It can be seen that the above-mentioned endoscopic display inspection system can realize the real-time three-dimensional display of the actual state of the patient's digestive tract during the endoscopic examination process, and can actively mark and display suspicious lesions, thereby assisting doctors in performing three-dimensional light field display, facilitating doctors to quickly compare information, speeding up the efficiency of endoscopic examinations, and reducing the misdiagnosis rate. In addition, during postoperative follow-up examinations, the system can use the patient's last diagnosed three-dimensional model of the digestive tract to quickly locate the follow-up lesion and display it on the screen, helping doctors quickly check the recovery status, improving efficiency, reducing patient examination time, and enhancing the endoscopic treatment experience.

[0111] In one implementation of the embodiment of the present application, after constructing the three-dimensional digestive tract model with the lesion location marked, the method further includes:

[0112] (1) during a review of the subject to be tested, determining a second depth corresponding to a lesion location marked on the three-dimensional model of the digestive tract;

[0113] (2) The insertion portion is controlled to be inserted into the digestive tract to a position corresponding to the second depth.

[0114] When reviewing the object to be tested, a previously constructed three-dimensional model of the digestive tract of the object to be tested can be obtained, and the lesion position marked by the three-dimensional model of the digestive tract can be used to find the corresponding depth data, that is, the depth of the insertion part of the endoscope inserted into the digestive tract, which is recorded as the second depth. Then, the insertion part of the endoscope can be controlled to insert into the position corresponding to the second depth in the digestive tract through motor control or other methods, so as to find the corresponding lesion position for observation. Since the actual structure of the digestive tract of the object to be tested will be different at different times, the lesion position found in this way may not be very accurate. At this time, the position can be corrected in combination with the image data captured by the endoscope camera to improve the positioning accuracy of the lesion position.

[0115] During the review, in addition to the embodiment of using a motor to control the insertion of the insertion portion into the digestive tract to reach the lesion location, the doctor can also manually insert the insertion portion into the digestive tract. When the insertion portion extends into the marked lesion location, the system can output a certain form of feedback signal so that the doctor knows that the lesion location found during the last diagnosis has been reached. For example, an indicator light can be set, and the indicator light lights up when the insertion portion reaches the lesion location found last time. For another example, a buzzer can be set, and the buzzer works when the insertion portion reaches the lesion location found last time, generating sound feedback. For another example, a vibration feedback component can be set, and when the insertion portion reaches the lesion location found last time, the component can automatically lock by a knob, etc., to generate corresponding vibration feedback.

[0116] During a review, the endoscopic display inspection system can read the corresponding lesion location information based on the three-dimensional model of the digestive tract constructed during the patient's last examination, and quickly locate the previously discovered lesion. Through this setting, it can help doctors to review patients efficiently and quickly, shorten the patient's examination time, and thus improve the patient's medical experience. It should be noted that when the patient is reviewed, the accuracy of the three-dimensional model of the digestive tract constructed during the last examination will further decrease, and it can usually only help determine the approximate range of a lesion location. However, due to the high accuracy of the three-dimensional model of the digestive tract constructed in the embodiment of the present application, doctors can still use this three-dimensional model to achieve rapid lesion screening and positioning.

[0117] Considering that many existing endoscope systems are originally equipped with a position sensor at the head end of the endoscope body, if the digestive tract modeling method provided in the embodiment of the present application is adopted, it is only necessary to add two external cameras and corresponding software algorithms on the basis of the existing endoscope system, and there is no need to iteratively upgrade the endoscope system. It can be seen that this method has strong compatibility and scene adaptability, and has a high value for promotion and application.

[0118] In the embodiment of the present application, two external cameras are set, which are respectively recorded as the first external camera and the second external camera, wherein the first external camera is used to capture the depth data of the insertion portion of the endoscope inserted into the digestive tract, and the second external camera is used to capture the lesion event; in the process of using the endoscope to detect the digestive tract of the object to be tested, the depth data captured by the first external camera, the position data of the front end of the insertion portion in the digestive tract, and the lesion event identified based on the image captured by the second external camera are obtained, and then the lesion position of the digestive tract is determined based on the depth data, position data and lesion event, and finally a three-dimensional model of the digestive tract with the lesion position marked is constructed. In the above process, since the depth data can represent the true depth of the insertion portion of the endoscope inserted into the digestive tract, and the position data can reflect the true position reached by the insertion portion, the combination of the two can accurately reflect the true spatial movement of the head end, so the depth data and position data can be used to accurately quantify the distance and length of each position point of the digestive tract three-dimensional model, thereby realizing accurate three-dimensional modeling of the digestive tract. On this basis, the true position point of the lesion is found in combination with the identified lesion event, thereby improving the accuracy and credibility of lesion positioning. Finally, based on this authentic and reliable data, a 3D model of the digestive tract is constructed and the lesion locations are marked, enabling precise positioning and labeling of the lesions. This entire process requires no manual intervention from the doctor, as model building and lesion labeling are based on authentic and accurate data. Therefore, it offers excellent reference value and can more effectively assist doctors in their operations.

[0119] Furthermore, the depth data and lesion events in the embodiments of this application are sourced from external cameras outside the endoscope system, meaning that only the position data from the endoscope tip is required. Therefore, the software and hardware requirements for the endoscope system are extremely low, and no additional hardware is required. Therefore, the system is compatible with a wide range of endoscope systems from different manufacturers, types, and models, demonstrating strong compatibility and significant practical and promotional value.

[0120] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0121] The above mainly describes a digestive tract modeling method, and the following will describe a digestive tract modeling device.

[0122] See also Figure 6 In one embodiment of the present application, a digestive tract modeling device includes:

[0123] A data acquisition module 601 is configured to acquire, during the process of examining the digestive tract of a subject using an endoscope, data on the depth of insertion of the endoscope's insertion portion into the digestive tract captured by a first external camera, lesion events identified based on images captured by a second external camera, and position data on the tip of the insertion portion within the digestive tract;

[0124] a lesion location determination module 602, configured to determine the location of a lesion in the digestive tract based on the depth data, the location data, and the lesion event;

[0125] The model building module 603 is used to build a three-dimensional model of the digestive tract with the lesion location marked based on the depth data, the position data and the lesion location.

[0126] In one implementation of the embodiment of the present application, a scale mark for marking the insertion depth is provided on the insertion portion; and the data acquisition module includes:

[0127] a first image acquisition unit, configured to acquire a first image of the insertion portion inserted into the digestive tract, captured by a first external camera;

[0128] The depth data determining unit is configured to determine the depth data according to a scale line on the insertion portion in the first image that is closest to the insertion opening of the digestive tract.

[0129] In one implementation of the embodiment of the present application, the data acquisition module includes:

[0130] a second image acquisition unit, configured to acquire a second image of a display screen of the endoscope system captured by a second external camera;

[0131] The focus event determining unit is configured to determine that a focus event occurs if a display screen in the second image freezes or displays specified status information.

[0132] Furthermore, the focus event determination unit includes:

[0133] an image acquisition subunit, configured to acquire a third image of an operation action of an operator of the endoscope captured by a second external camera if a freeze or specified status information appears on the display screen in the second image;

[0134] The focus event determination subunit is configured to determine that a focus event occurs if the operation action in the third image is a designated action.

[0135] In one implementation of the embodiment of the present application, the lesion location determination module includes:

[0136] A lesion time determination unit is used to determine the target time for discovering the lesion according to the lesion event;

[0137] a data search unit, configured to search the first depth corresponding to the target time from the depth data, and to search the target position corresponding to the target time from the position data;

[0138] The lesion position determining unit is used to determine the lesion position according to the first depth and the target position.

[0139] In one implementation of the embodiment of the present application, the digestive tract modeling device further includes:

[0140] a depth determination module, configured to determine a second depth corresponding to a lesion position marked on the three-dimensional model of the digestive tract during a review of the subject to be tested;

[0141] The insertion control module is used to control the insertion portion to be inserted into the digestive tract at a position corresponding to the second depth.

[0142] In one implementation of the embodiment of the present application, the digestive tract modeling device further includes:

[0143] An endoscope image acquisition module is used to acquire image data of the digestive tract captured by an endoscope camera;

[0144] A digestive tract part recognition module is used to recognize information about various parts of the digestive tract based on image data;

[0145] Model building modules include:

[0146] The model construction unit is used to construct a three-dimensional model of the digestive tract that records the digestive tract location and marks the lesion location based on the location information, depth data and position data.

[0147] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the digestive tract modeling method represented by any of the above embodiments.

[0148] An embodiment of the present application also provides a computer program product, which, when executed on a modeling device, enables the modeling device to execute the digestive tract modeling method as represented by any of the above embodiments.

[0149] Figure 7 Schematic diagram of a modeling device provided in one embodiment of the present application. Figure 7 As shown, the modeling device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned embodiments of the digestive tract modeling method are implemented, for example Figure 2Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 6 Functions of modules 601 to 603 are shown.

[0150] The computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the modeling device 7.

[0151] The processor 70 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0152] The memory 71 may be an internal storage unit of the modeling device 7, such as a hard disk or memory of the modeling device 7. The memory 71 may also be an external storage device of the modeling device 7, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the modeling device 7. Furthermore, the memory 71 may include both an internal storage unit of the modeling device 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the modeling device 7. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0154] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0156] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0159] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0161] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A digestive tract modeling method, characterized in that: include: During the process of examining the digestive tract of a subject using an endoscope, obtaining data on the depth of the insertion portion of the endoscope inserted into the digestive tract, as captured by a first external camera, lesion events identified based on images captured by a second external camera, and position data on the tip of the insertion portion within the digestive tract; determining a lesion location in the digestive tract according to the depth data, the location data, and the lesion event; A three-dimensional model of the digestive tract with the lesion location marked is constructed based on the depth data, the position data and the lesion location.

2. The method according to claim 1, wherein The insertion portion is provided with a scale line for marking the insertion depth; the depth data is obtained by: acquiring a first image captured by the first external camera of the insertion portion inserted into the digestive tract; The depth data is determined based on a scale line on the insertion portion in the first image that is closest to the insertion opening of the digestive tract.

3. The method according to claim 1, wherein The focal events were determined by: Acquiring a second image of a display screen of the endoscope system captured by the second external camera; If the display screen in the second image shows a frozen screen or specified status information, it is determined that the lesion event occurs.

4. The method according to claim 3, wherein If the display screen in the second image shows a frozen screen or specified status information, determining that the lesion event occurs includes: If the display screen in the second image shows a frozen screen or a specified status information, acquiring a third image of the operation action of the operator of the endoscope captured by the second external camera; If the operation action in the third image is a designated action, it is determined that the lesion event occurs.

5. The method according to claim 1, wherein The determining the location of the lesion in the digestive tract according to the depth data, the location data, and the lesion event includes: Determining a target time for discovering a lesion based on the lesion event; searching for a first depth corresponding to the target moment from the depth data, and searching for a target position corresponding to the target moment from the position data; The lesion position is determined according to the first depth and the target position.

6. The method according to claim 1, wherein After constructing the three-dimensional digestive tract model with the lesion location marked, the method further includes: During the review of the subject to be tested, determining a second depth corresponding to the lesion location marked on the three-dimensional model of the digestive tract; The insertion portion is controlled to be inserted into the digestive tract to a position corresponding to the second depth.

7. The method according to any one of claims 1 to 6, wherein: Before constructing the three-dimensional digestive tract model with the lesion location marked, the method further includes: Acquiring image data of the digestive tract captured by a camera of the endoscope; identifying information of various parts of the digestive tract according to the image data; The constructing of a three-dimensional digestive tract model showing the location of the lesion includes: A three-dimensional model of the digestive tract is constructed based on the part information, the depth data, and the position data, which records the digestive tract part and marks the lesion position.

8. A digestive tract modeling system, characterized in that: include: a modeling device, a first external camera, and a second external camera; During the process of examining the digestive tract of a subject using an endoscope: The first external camera is used to capture a first image of the insertion portion of the endoscope inserted into the digestive tract; The second external camera is used to capture a second image of the display screen of the endoscope system; The modeling device is used to determine the depth data of the insertion portion into the digestive tract based on the first image, and to determine the lesion event based on the second image; The modeling device is also used to determine the location of the lesion in the digestive tract based on the depth data, the lesion event and the position data of the head end of the insertion part in the digestive tract; and to construct a three-dimensional model of the digestive tract that marks the lesion location based on the depth data, the position data and the lesion location.

9. A digestive tract modeling device, characterized in that: include: a data acquisition module for acquiring, during the process of examining the digestive tract of a subject using an endoscope, data on the depth of insertion of the insertion portion of the endoscope into the digestive tract captured by a first external camera, lesion events identified based on images captured by a second external camera, and position data on the tip of the insertion portion within the digestive tract; a lesion location determination module, configured to determine the lesion location in the digestive tract based on the depth data, the location data, and the lesion event; A model building module is used to build a three-dimensional model of the digestive tract with the lesion location marked based on the depth data, the position data and the lesion location.

10. A modeling device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the digestive tract modeling method according to any one of claims 1 to 7 is implemented.

Images