Endoscopic field of view expansion method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of endoscope field-of-view processing, and provides an endoscope field-of-view expansion method, device, equipment and storage medium, which can expand the endoscope field-of-view. A three-dimensional model of a target object is constructed according to a preoperative image; a type of mark points of an anatomical point of the target object are marked on the three-dimensional model; a second type of mark points of the anatomical point are marked on an under-mirror field-of-view picture of an endoscope; and the under-mirror field-of-view picture and the three-dimensional model are fused and displayed according to the first type of mark points of the anatomical point on the three-dimensional model and the second type of mark points of the anatomical point on the under-mirror field-of-view picture, so as to expand the endoscope field-of-view.

Description

Technical Field

[0001] This application relates to the field of endoscopic field of view processing technology, and in particular to an endoscopic field of view expansion method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] Neuroendoscopic surgery is a minimally invasive surgical technique used to treat a variety of neurological diseases and problems, such as intracranial tumors, spinal lesions, traumatic brain injury, and cerebrovascular diseases. Before the surgery, imaging examinations are required to accurately locate the lesion or target area. The surgeon needs to operate slowly and meticulously while maintaining the stability of the procedure. Each patient's neural structure and lesion may be different, so the surgery needs to be tailored to the individual patient's condition.

[0003] In traditional neuroendoscopic surgery, the proximity of the lens to the body tissue means the surgeon often only has a few centimeters of field of vision below the lens. Furthermore, to improve clarity, the image is usually magnified, further reducing the surgeon's field of vision. This makes it difficult for them to accurately assess the patient's overall condition, such as the size and relative location of lesions, and the relationship between surgical instruments and lesions. Similar problems arise in other minimally invasive surgeries, such as orthopedic procedures. The reduced field of vision during these procedures hinders the surgeon's ability to accurately assess the patient's overall condition, including the size and relative location of lesions and the relationship between surgical instruments and lesions, thus affecting the smooth progress of the surgery. Summary of the Invention

[0004] Therefore, it is necessary to provide an endoscopic field of view expansion method, device, computer equipment, storage medium, and computer program product to address the above-mentioned technical problems.

[0005] This application provides a method for expanding the endoscopic field of view, the method comprising:

[0006] Construct a 3D model of the target object based on preoperative images;

[0007] Obtain a class of marker points on the 3D model representing the anatomical points of the target object;

[0008] Obtain the Class II markers of the anatomical points on the endoscopic field of view;

[0009] Based on the anatomical points as first-class markers in the 3D model and second-class markers in the endoscopic field of view, the endoscopic field of view is fused with the 3D model and displayed to expand the endoscopic field of view.

[0010] In one embodiment, obtaining the two types of markers for the anatomical points marked on the endoscopic view includes:

[0011] Obtain the endoscopic view of the endoscope and the target 2D image corresponding to the 3D model from the target perspective;

[0012] Adjust the acquisition angle of the endoscope according to the target two-dimensional image until the content of the endoscopic field of view matches the content of the target two-dimensional image;

[0013] When the content of the microscopic field of view matches the content of the target two-dimensional image, the dot markers on the microscopic field of view are obtained to obtain the second type of markers for the anatomical points on the microscopic field of view.

[0014] In one embodiment, obtaining the target 2D image corresponding to the 3D model from the target's perspective includes:

[0015] Obtain multiple candidate viewpoints for viewing the target object using the endoscope;

[0016] Based on the two-dimensional images corresponding to the three-dimensional model under multiple candidate viewpoints, a target two-dimensional image corresponding to the three-dimensional model under the target viewpoint is obtained.

[0017] In one embodiment, based on the anatomical points as Class I markers in the 3D model and Class II markers in the endoscopic view, the endoscopic view is fused with the 3D model and displayed, including:

[0018] Among the several Class I marker points in the 3D model and the several Class II marker points in the endoscopic view, the Class I marker points and Class II marker points corresponding to the same anatomical point are determined to obtain marker point pairs;

[0019] Based on the marked point pairs, the lenticular view is fused with the 3D model and then displayed.

[0020] In one embodiment, the first point set includes several Class I marker points, and the second point set includes several Class II marker points; among the several Class I marker points in the 3D model and the several Class II marker points in the endoscopic view, Class I and Class II marker points corresponding to the same anatomical point are determined to obtain marker point pairs, including:

[0021] Obtain the marking order of each of the first-class markers in the first point set and the marking order of each of the second-class markers in the second point set;

[0022] According to the marking order, each of the first-class marker points and each of the second-class marker points are matched to obtain marker point pairs.

[0023] In one embodiment, obtaining the labeling order of each of the first-class markers in the first point set and the labeling order of each of the second-class markers in the second point set includes:

[0024] In the vision under the microscope and the target two-dimensional image corresponding to the three-dimensional model from the target's perspective, determine the image with the dots appearing first and the image with the dots appearing later.

[0025] Obtain the anatomical points from the prior marking points on the prior marking map;

[0026] According to the marking order of the prior marked points in the corresponding point set, a character or color is displayed at each prior marked point to represent the marking order; the character or color is used to instruct the endoscopic operator to perform the marking operation on the subsequent image according to the marking order represented by the character or color.

[0027] Based on the dotting operation performed by the endoscopic operator on the dotting diagram, obtain the subsequent markers of several anatomical points on the dotting diagram and the marking order of the subsequent markers in the corresponding point set;

[0028] Based on the marking order of the earlier marked points in the corresponding point set and the marking order of the later marked points in the corresponding point set, the marking order of each of the first-class marked points in the first point set and the marking order of each of the second-class marked points in the second point set are obtained.

[0029] In one embodiment, matching each of the first-class marker points and each of the second-class marker points according to the marking order to obtain marker point pairs includes:

[0030] When a match is determined between a Class I marker and a Class II marker based on the marking order, the position of the Class I marker relative to other points in the first point set and the position of the Class II marker relative to other points in the second point set are determined.

[0031] When the position of one of the Class I markers relative to other points in the first point set does not match the position of one of the Class II markers relative to other points in the second point set, a marker mismatch alarm message is issued.

[0032] When the endoscopy operator receives a marker mismatch alarm message and confirms that one of the Class I markers and one of the Class II markers do not match to the same marker pair, in response to the endoscopy operator's matching point selection operation, a marker pair is obtained based on at least one of the Class I and Class II markers selected by the endoscopy operator.

[0033] This application provides an endoscopic field-of-view expansion device, the device comprising:

[0034] The 3D model acquisition module is used to construct a 3D model of the target object based on preoperative images;

[0035] The marker point acquisition module is used to acquire a class of marker points on the three-dimensional model representing the anatomical points of the target object;

[0036] The marker acquisition module is also used to acquire the second type of markers of the anatomical points marked on the endoscopic field of view;

[0037] The fusion display module is used to fuse and display the endoscopic field of view with the three-dimensional model based on the first type of marker points of the anatomical points in the three-dimensional model and the second type of marker points in the endoscopic field of view, so as to expand the endoscopic field of view.

[0038] This application provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor executes the above-described method.

[0039] This application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor using the methods described above.

[0040] This application provides a computer program product having a computer program stored thereon, the computer program being executed by a processor using the above-described method.

[0041] In the aforementioned endoscopic field of view expansion methods, devices, computer equipment, storage media, and computer program products, a three-dimensional model of the target object is constructed based on preoperative images. First-class markers of the anatomical points of the target object are obtained on the three-dimensional model, and second-class markers of the anatomical points are obtained on the endoscopic field of view. Based on the first-class markers of the anatomical points on the three-dimensional model and the second-class markers on the endoscopic field of view, the endoscopic field of view is fused with the three-dimensional model and displayed. This achieves the expansion of the endoscopic field of view and improves the efficiency of minimally invasive surgery by recognizing the markers of the anatomical points of the target object on both the three-dimensional model and the endoscopic field of view. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating an endoscopic field-of-view expansion method in one embodiment;

[0044] Figure 2This is a diagram illustrating the application environment of an endoscopic field-of-view expansion method in one embodiment.

[0045] Figure 3 This is a schematic diagram illustrating content changes in one embodiment;

[0046] Figure 4 This is a structural block diagram of an endoscope field-of-view expansion device in one embodiment;

[0047] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0050] The endoscopic field of view expansion method provided in this application can be executed by a computer device, which can be a device in an endoscopic system or a device in a non-endoscopic system; the method includes, for example... Figure 1 The steps shown are as follows:

[0051] Step S101: Construct a three-dimensional model of the target object based on the preoperative images.

[0052] Minimally invasive surgery is a technique that uses endoscopes to perform surgery inside specific humans or animals. Depending on the surgical site or organ involved, minimally invasive surgery can include neuroendoscopic surgery, orthopedic minimally invasive surgery, and so on. The surgical site or organ involved in minimally invasive surgery can be considered the target object. Before the minimally invasive surgery begins, images of the target object can be acquired using methods such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) to obtain a three-dimensional model of the target object.

[0053] For example, in minimally invasive orthopedic surgery, the knee joint of a specific person or animal involved in the surgery can be used as the target object, and a three-dimensional model of the target object can be obtained through methods such as CT and MRI.

[0054] Step S102: Obtain a class of marker points on the 3D model representing the anatomical points of the target object;

[0055] Step S103: Obtain the Class II markers of the anatomical points on the endoscopic field of view.

[0056] The target object has some anatomically significant landmarks, which can be called anatomical points. After obtaining a three-dimensional model of the target object, this application can obtain the markers of the anatomical points of the target object on the three-dimensional model and the markers marked on the microscopic view. In order to distinguish the markers of the anatomical points on the three-dimensional model and the microscopic view, the markers of the anatomical points on the three-dimensional model are called Class I markers, and the markers of the anatomical points on the microscopic view are called Class II markers.

[0057] For example, the target object has anatomical points A, B, and C. One type of marker point can be obtained for each of the anatomical points A, B, and C on the 3D model. Additionally, two types of marker points can be obtained for each of the anatomical points A, B, and C on the endoscopic view.

[0058] S102 and S103 are mainly used to identify the corresponding steps, and do not indicate the order of execution between the two steps. In some embodiments, the step corresponding to S102 can be executed first and then the step corresponding to S103 can be executed, or the step corresponding to S103 can be executed first and then the step corresponding to S102 can be executed.

[0059] Step S104: Based on the Class I markers of the anatomical points in the 3D model and the Class II markers in the endoscopic field of view, the endoscopic field of view is fused with the 3D model and displayed to expand the endoscopic field of view.

[0060] For example, after obtaining the first-class and second-class markers of the anatomical points {A, B, C}, the endoscopic view can be fused with the 3D model and displayed according to the correspondence between the markers. Specifically, the fusion can be to overlay the endoscopic view with the 3D model. Content on the 3D model that does not overlap with the endoscopic view can be displayed with lower transparency, while content on the 3D model that overlaps with the endoscopic view can be displayed with higher transparency. This expands the endoscopic field of view without interfering with the doctor's viewing of the endoscopic view. Here, transparency below a preset value can be used as lower transparency, and transparency above a preset value can be used as higher transparency.

[0061] In the aforementioned method for expanding the endoscopic field of view, a three-dimensional model of the target object is constructed based on preoperative images. First-class markers of the anatomical points of the target object are obtained on the three-dimensional model, and second-class markers of the anatomical points are obtained on the endoscopic field of view. Based on the first-class markers of the anatomical points in the three-dimensional model and the second-class markers on the endoscopic field of view, the endoscopic field of view is fused with and displayed with the three-dimensional model. This method expands the endoscopic field of view and improves the efficiency of minimally invasive surgery by recognizing the markers of the anatomical points of the target object in both the three-dimensional model and the endoscopic field of view.

[0062] In one embodiment, step S104 may specifically include the following steps: among several Class I marker points in the 3D model and several Class II marker points in the endoscopic view, determine the Class I marker points and Class II marker points corresponding to the same anatomical point to obtain marker point pairs; based on the marker point pairs, fuse the endoscopic view with the 3D model and display it.

[0063] Specifically, after obtaining several Class I markers on the 3D model and several Class II markers in the endoscopic view, Class I and Class II markers corresponding to the same anatomical point can be identified, and these Class I and Class II markers corresponding to the same anatomical point can be regarded as marker pairs.

[0064] After obtaining multiple marker point pairs, the view under the microscope and the 3D model are fused and displayed based on these pairs. Specifically, the virtual camera's viewing angle can be adjusted multiple times. Each time the viewing angle is adjusted, a 2D image of the 3D model under that viewing angle is acquired through the virtual camera. If the position of a type I marker point in at least one marker point pair on the 2D image does not coincide with the position of a type II marker point in the view under the microscope, the viewing angle can be adjusted again. When the position of a type I marker point in each marker point pair on the 2D image coincides with the position of a type II marker point in the view under the microscope, the adjustment can be stopped, and the 2D image of the 3D model under that viewing angle and the view under the microscope are fused and displayed.

[0065] In the above embodiments, the first-class markers and second-class markers corresponding to the same anatomical point are regarded as a pair. Based on this, the marker pair can more accurately fuse the endoscopic view with the three-dimensional model and avoid misalignment.

[0066] Anatomical points of the target object can be obtained by the endoscopic operator through manual marking (a manual marking method), or by a model obtained through deep learning (an automatic marking method), or by a combination of automatic and manual marking methods.

[0067] The anatomical points of the target object can be manually marked as Class II markers in the endoscopic view. After obtaining the Class I markers of the anatomical points on the 3D model of the target object through manual marking, the process can proceed to step S103. Step S103 may include: acquiring the endoscopic view and acquiring the target 2D image corresponding to the 3D model from the target perspective; adjusting the acquisition angle of the endoscope according to the target 2D image until the content of the endoscopic view matches the content of the target 2D image; when the content of the endoscopic view matches the content of the target 2D image, acquiring the marking marks on the endoscopic view to obtain the Class II markers of the anatomical points on the endoscopic view.

[0068] One method for obtaining the target 2D image of the 3D model from the target perspective is to set a perspective, collect the 2D image of the 3D model from that perspective, use that perspective as the target perspective, and use the 2D image of the 3D model from that perspective as the target 2D image.

[0069] During the procedure, a two-dimensional image of the target and a real-time endoscopic view can be simultaneously displayed on the screen, such as... Figure 2 As shown, this allows the endoscopic operator to simultaneously view a two-dimensional image of the target and the endoscopic field of view.

[0070] Another way to obtain the target 2D image corresponding to the 3D model from the target perspective is to: obtain multiple candidate perspectives for viewing the target object with an endoscope; and obtain the target 2D image corresponding to the 3D model from the target perspective based on the 2D images corresponding to the 3D model from the multiple candidate perspectives.

[0071] Based on the surgical planning path of this minimally invasive surgery, several possible viewpoints of the target object during endoscopic viewing can be determined. These viewpoints are then used as candidate viewpoints, resulting in multiple candidate viewpoints. Next, a two-dimensional image corresponding to the 3D model of the target object can be obtained from each candidate viewpoint. For example, the candidate viewpoints include view_1, view_2, and view_3. The two-dimensional image corresponding to the 3D model of the target object can be obtained by viewing_1 (hereinafter referred to as view_1 2D image), view_2 (hereinafter referred to as view_2 2D image), and view_3 (hereinafter referred to as view_3 2D image). The two-dimensional images of view_1, view_2, and view_3 can be displayed together with the endoscopic field of view, such as... Figure 3 As shown.

[0072] The endoscopic operator can select a two-dimensional image whose content is close to the current content of the endoscopic field of view from the view_1, view_2, and view_3 two-dimensional images. For example, if the endoscopic operator selects view_1 two-dimensional image, the computer device can use view_1 two-dimensional image as the target two-dimensional image based on the endoscopic operator's selection operation of view_1 two-dimensional image, and correspondingly, view_1 is used as the target viewpoint.

[0073] After receiving the endoscopy operator's selection of the view_1 2D image, the computer equipment can cancel or reduce the display of the 2D image from a non-target viewpoint, and enlarge the view_1 2D image and the endoscopic field of view. (Refer to again...) Figure 3 When the endoscopic operator selects the 2D view_1 using the mouse, the computer responds by not displaying the 2D views_2 and_3, and instead enlarges the display of the 2D view_1 and the endoscopic field of view. This frees up more display area to show the 2D view from the target perspective and the endoscopic field of view when the display area is limited, facilitating manual marking.

[0074] The target 2D image can be used to instruct the endoscope operator or automatic control equipment to adjust the endoscope's acquisition angle, thereby changing the content of the endoscopic field of view. When the content of the endoscopic field of view matches the content of the target 2D image, the current acquisition angle of the endoscope can be locked, ensuring that the content of the endoscopic field of view matches the content of the 2D image of the 3D model from the target perspective.

[0075] One method to determine the match between the content of the under-lens view and the content of the target two-dimensional image is to obtain the similarity between the content of the under-lens view and the content of the target two-dimensional image. When the similarity is higher than the similarity threshold, the content of the under-lens view is determined to match the content of the target two-dimensional image.

[0076] When a class of markers is pre-determined through automatic marking, the target 2D image can include a class of markers on the 3D model representing anatomical points presented from the target perspective. These markers are then presented on the target 2D image from the target perspective. The endoscopic operator can directly mark the markers presented on the target 2D image in the endoscopic field of view, or they can first adjust the markers determined through automatic marking. After adjustment, the positions of the markers on the 3D model and the target 2D image may change. The endoscopic operator then marks the markers on the endoscopic field of view based on the adjusted markers presented on the target 2D image.

[0077] Specifically, the endoscopic operator can view the target two-dimensional image to determine the position of the first type of marker points on the target two-dimensional image, and mark the corresponding positions on the endoscopic field of view. The computer equipment can obtain the second type of marker points of the anatomical points on the endoscopic field of view based on the marked positions on the endoscopic field of view.

[0078] This embodiment mainly uses a user-interactive method to mark points, obtaining two types of marker points for the anatomical points in the endoscopic field of view. Before marking points, the target 2D image and the endoscopic field of view are displayed to instruct the endoscopist to adjust the acquisition angle of the endoscope until the content of the endoscopic field of view matches the content of the target 2D image. Thus, the endoscopist can mark points at the corresponding positions in the endoscopic field of view based on the position of the first type of marker points on the 2D image, thereby obtaining two types of marker points for the anatomical points in the endoscopic field of view.

[0079] The following details the implementation methods for obtaining Class I and Class II markers for anatomical points:

[0080] Method 1:

[0081] When the Class I markers of the anatomical point are determined by automatic marking, the Class II markers of the anatomical point can be obtained directly from the Class I markers determined by automatic marking. Alternatively, the endoscopic operator can adjust the Class I markers determined by automatic marking and obtain the Class II markers of the anatomical point based on the adjusted Class I markers.

[0082] The secondary markers for anatomical points are primarily obtained through manual marking. Specifically, a 2D view (view_1) and the endoscopic field of view are displayed on the screen. The primary markers are presented according to their corresponding positions on the 2D view (view_1). The 2D view (view_1) can be used to instruct the endoscopist to adjust the endoscope's acquisition angle, thus changing the content of the endoscopic field of view. When the content of the endoscopic field of view matches the content of the 2D view (view_1), the current acquisition angle of the endoscope can be locked to maintain this match. At this point, the endoscopist can observe the primary markers presented on the 2D view (view_1) and determine their positions. The endoscopist then marks these markers at the corresponding positions on the endoscopic field of view. The computer equipment can then determine the secondary markers for the anatomical points on the endoscopic field of view based on the marked positions.

[0083] Method 2:

[0084] If the Class I and Class II markers of the anatomical point are not determined automatically, they can be determined manually. Specifically, a 2D view (view_1) and a endoscopic view can be displayed on the screen. The 2D view (view_1) can be used to instruct the endoscopist to adjust the endoscope's acquisition angle, thus changing the content of the endoscopic view. When the content of the endoscopic view matches the content of the 2D view (view_1), the current acquisition angle of the endoscope can be locked to maintain a match between the endoscopic view and the 2D view (view_1). At this point, the endoscopist can mark points on both the 2D view (view_1) and the endoscopic view to obtain the Class I and Class II markers of the anatomical point.

[0085] Specifically, the endoscopic operator can mark points on the 2D view_1 image first and then mark points on the endoscopic field of view, or the endoscopic field of view can mark points on the 2D view_1 image first. The endoscopic operator can select either the 2D view_1 image or the endoscopic field of view. This selection operation is a first-selection operation. Based on the endoscopic operator's first-selection operation, the computer device can enlarge and display the first-selected image (hereinafter referred to as the mark-first image) so that the endoscopic operator can mark points on the mark-first image.

[0086] The computer device obtains the corresponding markers for anatomical points based on the marking operations performed by the endoscopist on the prior marking image. After completing the marking on the prior marking image, the endoscopist can select another image between view_1 (2D view) and the endoscopic field of view. This selection is a subsequent selection operation. In response to the endoscopist's subsequent selection operation, the computer device can zoom out on the prior marking image and zoom out on the subsequently selected image (hereinafter referred to as the subsequent marking image), allowing the endoscopist to mark points on the subsequent marking image. The computer device obtains the corresponding markers for anatomical points based on the endoscopist's marking operations on the subsequent marking image.

[0087] In this case, when the image with dots first is a 2D view_1, the image with dots second can be a view under the lens; when the image with dots first is a view under the lens, the image with dots second can be a 2D view_1.

[0088] In one embodiment, the first point set includes several Class I marker points, and the second point set includes several Class II marker points. Among the several Class I marker points in the 3D model and the several Class II marker points in the endoscopic view, the Class I marker points and Class II marker points corresponding to the same anatomical point are determined to obtain marker point pairs. Specifically, this may include: obtaining the marking order of each Class I marker point in the first point set and the marking order of each Class II marker point in the second point set; matching each Class I marker point and each Class II marker point according to the marking order to obtain marker point pairs.

[0089] For example, if a class of markers has 3 points, and the markers are named in order of their marking sequence, then the first set of markers includes the marker. Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 .

[0090] If there are 3 Class II markers, name them in order of marking sequence. In this case, the second set of markers includes the marker. Ⅱ_1 marker Ⅱ_2 and marker Ⅱ_3 .

[0091] Among them, the class of markers corresponding to the first marker order in the first point set is called marker. Ⅰ_1 In the second point set, the second type of marker point corresponding to the first marker order is called marker. Ⅱ_1 Therefore, the marker can be Ⅰ_1 and marker Ⅱ_1 Perform matching, and use the marker Ⅰ_1 and marker Ⅱ_1 Consider it as a pair of marker points, marker Ⅰ_1 and marker Ⅱ_1 They correspond to the same anatomical point.

[0092] Further, obtaining the marking order of each type I marker point in the first point set and the marking order of each type II marker point in the second point set includes: determining the marking-first image and the marking-later image in the endoscopic view and the target 2D image corresponding to the 3D model in the target view; obtaining the prior marker points of the anatomical points in the marking-first image; displaying characters or colors representing the marking order at each prior marker point according to the marking order of the prior marker points in the corresponding point set; the characters or colors are used to instruct the endoscopic operator to perform marking operations on the marking-later image according to the marking order represented by the characters or colors; obtaining the subsequent marker points of several anatomical points in the marking-later image and the marking order of the subsequent marker points in the corresponding point set based on the marking operations performed by the endoscopic operator on the marking-later image; and obtaining the marking order of each type I marker point in the first point set and the marking order of each type II marker point in the second point set based on the marking order of the prior marker points in the corresponding point set and the marking order of the subsequent marker points in the corresponding point set.

[0093] In the first image, when the markers are placed first, the first marker is a type of marker, and the corresponding point set to which the first marker belongs is the first point set. In the second image, when the markers are placed later, the target is a two-dimensional image, the later markers are type two, and the corresponding point set to which the later markers belong is the second point set. In the third image, when the markers are placed first, the target is a two-dimensional image, the first markers are type two, and the corresponding point set to which the first marker belongs is the second point set. In the fourth image, when the markers are placed later, the target is a two-dimensional image, the first markers are type two, and the corresponding point set to which the first marker belongs is the second point set. In the fifth image, when the markers are placed later, the target is a type of marker, and the corresponding point set to which the later marker belongs is the first point set.

[0094] Taking the image with dots placed first as the target 2D image, which is the aforementioned view_1 2D image, and the image with dots placed later as the view under the lens as an example:

[0095] If there are three anatomical points, and the three primary markers are determined using an automatic point marking method, the computer device can display a two-dimensional image (view_1). Clicking "view_1" will display the corresponding positions of these three primary markers on the view_1 image. The endoscopist can sequentially mark these three primary markers by clicking them with a mouse. Based on the endoscopist's sequential marking, the computer device can determine the marking order of these three primary markers within the first point set.

[0096] When using manual dotting to determine the first-class markers of the anatomical points, the endoscopic operator can mark the first-class markers corresponding to the three anatomical points on the view_1 two-dimensional image. Based on the operator's dotting operation, the computer device can determine the three first-class markers on the view_1 two-dimensional image and the marking order of each first-class marker in the first point set.

[0097] Following the order of marking from first to last, these three Class I markers are respectively designated as markers. Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 In this context, characters or colors representing the marking order are displayed at each type of marker point. For example, in the marker... Ⅰ_1 The number "1" is displayed at the marker. Ⅰ_2 The number "2" is displayed in the marker. Ⅰ_3 The number "3" is displayed in the area, or for example, a marker is displayed in blue. Ⅰ_1 The marker is displayed in yellow. Ⅰ_2 The marker is displayed in green. Ⅰ_3 In this sequence, blue corresponds to the first marking order, yellow corresponds to the second marking order, and green corresponds to the third marking order.

[0098] If in markerⅠ_1 marker Ⅰ_2 and marker Ⅰ_3 The markers display the numbers "1", "2", and "3" respectively. The endoscopic operator can mark the points on the endoscopic field of view according to the order represented by these numbers. Specifically, the endoscopic operator can first mark the points according to the markers... Ⅰ_1 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_1 The endoscopic operator can first follow the marker Ⅰ_2 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_2 The endoscopic operator can first follow the marker Ⅰ_3 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_3 .

[0099] Among them, the class of markers corresponding to the first marking order in the first point set is called marker. Ⅰ_1 In the second point set, the second type of marker point corresponding to the first marker order is called marker. Ⅱ_1 Therefore, the marker can be Ⅰ_1 and marker Ⅱ_1 Perform matching, and use the marker Ⅰ_1 and marker Ⅱ_1 Consider it as a pair of marker points, marker Ⅰ_1 and marker Ⅱ_1 They correspond to the same anatomical point.

[0100] A class of markers corresponding to the second marking order within the first point set is called a marker. Ⅰ_2 In the second point set, the type II marker corresponding to the second marking order is called marker. Ⅱ_2 Therefore, the marker can be Ⅰ_2 and marker Ⅱ_2 Perform matching, and use the marker Ⅰ_2 and marker Ⅱ_2 Consider it as a pair of marker points, marker Ⅰ_2 and marker Ⅱ_2 They correspond to the same anatomical point.

[0101] A class of markers corresponding to the third marker order within the first point set is called a marker. Ⅰ_3 In the second point set, the second type of marker corresponding to the third marker order is called marker. Ⅱ_3Therefore, the marker can be Ⅰ_3 and marker Ⅱ_3 Perform matching, and use the marker Ⅰ_3 and marker Ⅱ_3 Consider it as a pair of marker points, marker Ⅰ_3 and marker Ⅱ_3 They correspond to the same anatomical point.

[0102] This embodiment uses the marking order of the marker points in the corresponding point set to determine the first-class and second-class marker points corresponding to the same anatomical point. Based on the marker point pairs formed, the two-dimensional image of the three-dimensional model under the target view and the endoscopic view are fused and displayed to avoid misalignment.

[0103] To further avoid misalignment, this application also provides an embodiment that uses the relative position of a marker point with other marker points in the corresponding category point set for auxiliary verification. Specifically, according to the marking order, each Class I marker point and each Class II marker point are matched to obtain marker point pairs, which may include: when it is determined that one Class I marker point and one Class II marker point match according to the marking order, determining the position of one Class I marker point relative to other points in the first point set and the position of one Class II marker point relative to other points in the second point set; when the position of one Class I marker point relative to other points in the first point set and the position of one Class II marker point relative to other points in the second point set do not match, issuing a marker point mismatch alarm message; when the endoscopy operator receives the marker point mismatch alarm message and confirms that one Class I marker point and one Class II marker point do not match to the same marker point pair, in response to the endoscopy operator's matching point selection operation, obtaining a marker point pair based on at least one of the Class I and Class II marker points selected by the endoscopy operator.

[0104] For example, a class of markers is determined based on the marking order. Ⅰ_1 and Class II markers Ⅱ_1 After matching, a class of marker points can be obtained. Ⅰ_1 Compared to other points in the first point set (such as markers) Ⅰ_2 and marker Ⅰ_3 The location of the marker and the second-class marker. Ⅱ_1 Compared to other points in the second point set (such as markers) Ⅱ_2 and marker Ⅱ_3 The position of ).

[0105] For example, if based on a class of marker points Ⅰ_1 Relative to marker Ⅰ_2 and markerⅠ_3 The location can determine a class of marker points. Ⅰ_1 Located in marker Ⅰ_2 and marker Ⅰ_3 Between; if based on the two types of markers Ⅱ_1 Relative to marker Ⅱ_2 and marker Ⅱ_3 The location can determine the type II marker. Ⅱ_1 Located in marker Ⅱ_2 and marker Ⅱ_3 Between; at this point, a class of marker points can be identified. Ⅰ_1 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_1 Position matching relative to other points within the second point set.

[0106] Another example is based on a class of marker points. Ⅰ_1 Compared to marker Ⅰ_2 The location can determine a class of marker points. Ⅰ_1 In marker Ⅰ_2 Below; if based on the second type of marker Ⅱ_1 Compared to marker Ⅱ_2 The location can determine the type II marker. Ⅱ_1 In marker Ⅱ_2 Below; at this point, a class of marker points can be identified. Ⅰ_1 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_1 Position matching relative to other points within the second point set.

[0107] When a class of marker points Ⅰ_1 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_1 When matching the positions of other points within the second point set, a class of marker points can be determined. Ⅰ_1 and Class II markers Ⅱ_1 For the same anatomical point, they can be regarded as a pair of marker points.

[0108] When a class of marker points Ⅰ_1 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_1 When the position of a point does not match that of other points in the second point set, the computer device can issue a marker mismatch alarm message.

[0109] Marker mismatch alarm messages are used to instruct endoscopic operators to confirm a type of marker. Ⅰ_1 and Class II markers Ⅱ_1 Do they not correspond to the same anatomical point or are not the same marker pair? If the endoscopic operator identifies a type of marker... Ⅰ_1 and Class II markers Ⅱ_1 If the markers do not correspond to the same anatomical point and are not the same pair of markers, the endoscopic operator can use a marker in one class. Ⅰ_2 and marker Ⅰ_3 Select the marker with the second type of marker. Ⅱ_1 Matching markers; in response to the endoscope operator's matching point selection operation, the computer device can match the Class I markers selected by the endoscope operator with Class II markers. Ⅱ_1 A match is considered a pair of the same marker points.

[0110] Similarly, endoscopic operators can also use Class II markers. Ⅱ_2 and marker Ⅱ_3 Select a marker of a certain type Ⅰ_1 Matching markers; in response to the endoscope operator's matching point selection operation, the computer device can match the Class II markers selected by the endoscope operator with Class I markers. Ⅰ_1 A match is considered a pair of the same marker points.

[0111] Following the above method, when a class of marker points Ⅰ_2 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_2 When matching the positions of other points within the second point set, we can determine: a class of marker points. Ⅰ_2 and Class II markers Ⅱ_2 Corresponding to the same anatomical point, they can be considered as a pair of marker points. When a class of marker points... Ⅰ_2 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_2 When the positions of other points in the second point set do not match, the corresponding handling methods can be referred to the above content.

[0112] When a class of marker points is determined Ⅰ_3 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_3 When matching the positions of other points within the second point set, we can determine: a class of marker points. Ⅰ_3 and Class II markers Ⅱ_3Corresponding to the same anatomical point, they can be considered as a pair of marker points. When a class of marker points... Ⅰ_3 Relative to the positions of other points within the first point set and the second-class markers Ⅱ_3 When the positions of other points in the second point set do not match, the corresponding handling methods can be referred to the above content.

[0113] To better understand the above method, the following is a detailed application example of the endoscopic field of view expansion method of this application. The minimally invasive surgery in this application example is orthopedic minimally invasive surgery, with the knee joint as the target object and involving 3 anatomical points.

[0114] Part One:

[0115] After obtaining preoperative images through methods such as CT and MRI, a three-dimensional model of the knee joint can be generated based on these images. Specifically, a trained segmentation network can be used to segment the bones in the preoperative images to obtain the knee joint. The segmentation network can be a Unet network (also known as U-Net, a convolutional neural network that can be used for image segmentation).

[0116] A three-dimensional model of the knee joint can be reconstructed using various modal data, such as CT and MRI data. With MRI data, not only can a 3D skeletal model of the knee joint be extracted using a segmentation network, but also the soft tissues and bone marrow surrounding the knee joint can be extracted. Optionally, a 3D skeletal model of the knee joint can be segmented using CT data, and a 3D tissue model of the knee joint can be segmented using MRI data. The skeletal and tissue 3D models of the knee joint can then be fused to obtain a fused result.

[0117] Next, techniques such as surface reconstruction can be used to process the fusion results to obtain a three-dimensional model of the knee joint. Surface reconstruction is a commonly used three-dimensional reconstruction technique that reconstructs a three-dimensional model by extracting the surface geometry of the object, and then optimizes the reconstructed skeletal model, including removing unnecessary noise, smoothing the surface, and filling defects, to further improve the accuracy and integrity of the model.

[0118] Optionally, the sources of the three-dimensional model of the knee joint include, but are not limited to, models reconstructed from preoperative images of the patient, standard bone models, and parametric bone models.

[0119] Part Two: Obtaining Class I markers of the anatomical points of the knee joint in the 3D model and Class II markers in the endoscopic view.

[0120] Specifically, based on the surgical planning path of this minimally invasive orthopedic surgery, several possible viewpoints (such as view_1, view_2 and view_3) that may appear during the operation can be used as candidate viewpoints, and two-dimensional images of the three-dimensional model of the knee joint under view_1, view_2 and view_3 can be obtained.

[0121] During minimally invasive orthopedic surgery, the system can display the endoscopic view, as well as three 2D images of the knee joint: view_1, view_2, and view_3. The endoscopic operator can select an image from these three images that closely matches the current content of the endoscopic view, such as view_1. Based on this selection, the computer system can use view_1 as the target image.

[0122] After receiving the endoscopy operator's selection of the view_1 2D image, the computer equipment can cancel or reduce the display of the view_2 and view_3 2D images, and enlarge the view_1 2D image and the endoscopic field of view. Thus, when the display area is limited, more display area can be freed up to display the view_1 2D image and the endoscopic field of view for manual marking.

[0123] The first method: Both Class I and Class II marker points are determined manually by marking points.

[0124] The endoscopic operator can mark the Class I markers corresponding to the three anatomical points on the view_1 two-dimensional image. Based on the operator's marking operation, the computer device can determine the three Class I markers on the view_1 two-dimensional image and the marking order between each Class I marker.

[0125] Following the order of marking from first to last, these three Class I markers are respectively designated as markers. Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 In this context, characters or colors representing the marking order are displayed at each type of marker point. For example, in the marker... Ⅰ_1 The number "1" is displayed at the marker. Ⅰ_2 The number "2" is displayed in the marker. Ⅰ_3 The number "3" is displayed in the area, or for example, a marker is displayed in blue. Ⅰ_1 The marker is displayed in yellow. Ⅰ_2 The marker is displayed in green. Ⅰ_3 In this sequence, blue corresponds to the first marking order, yellow corresponds to the second marking order, and green corresponds to the third marking order.

[0126] If in marker Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 The markers display the numbers "1", "2", and "3" respectively. The endoscopic operator can mark the points on the endoscopic field of view according to the order represented by these numbers. Specifically, the endoscopic operator can first mark the points according to the markers... Ⅰ_1 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_1 The endoscopic operator can first follow the marker Ⅰ_2 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_2 The endoscopic operator can first follow the marker Ⅰ_3 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_3 .

[0127] The second method: Class I markers are determined automatically, and Class II markers are determined manually.

[0128] The computer device can display a two-dimensional view (view_1). Clicking on view_1 will display the corresponding positions of the three anatomical points' Class I markers on the view_1 two-dimensional view. The endoscopist can sequentially mark these three Class I markers by clicking them in sequence, similar to using a mouse. Based on the endoscopist's sequential marking operation, the computer device can determine the marking order among the three Class I markers.

[0129] Following the order of marking from first to last, these three Class I markers are respectively designated as markers. Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 In this context, characters or colors representing the marking order are displayed at each type of marker point. For example, in the marker... Ⅰ_1 The number "1" is displayed at the marker. Ⅰ_2 The number "2" is displayed in the marker. Ⅰ_3 The number "3" is displayed in the area, or for example, a marker is displayed in blue. Ⅰ_1 The marker is displayed in yellow. Ⅰ_2 The marker is displayed in green. Ⅰ_3 In this sequence, blue corresponds to the first marking order, yellow corresponds to the second marking order, and green corresponds to the third marking order.

[0130] If in marker Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 The markers display the numbers "1", "2", and "3" respectively. The endoscopic operator can mark the points on the endoscopic field of view according to the order represented by these numbers. Specifically, the endoscopic operator can first mark the points according to the markers... Ⅰ_1 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_1 The endoscopic operator can first follow the marker Ⅰ_2 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_2 The endoscopic operator can first follow the marker Ⅰ_3 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_3 .

[0131] The third method: Class I markers are determined using both automatic and manual marking methods; Class II markers are determined using manual marking methods.

[0132] The computer device can display a 2D view (view_1). Clicking "view_1" will display the corresponding positions of the three anatomical markers on the view_1 2D image. The endoscopist can adjust the markers determined by the automatic marking method, and the computer device will display the adjusted markers on the view_1 2D image. The endoscopist can sequentially mark these three markers by clicking them with a mouse, and the computer device can determine the marking order of the three markers based on the endoscopist's sequential marking operation.

[0133] Following the order of marking from first to last, these three Class I markers are respectively designated as markers. Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 In this context, characters or colors representing the marking order are displayed at each type of marker point. For example, in the marker... Ⅰ_1 The number "1" is displayed at the marker. Ⅰ_2 The number "2" is displayed in the marker. Ⅰ_3 The number "3" is displayed in the area, or for example, a marker is displayed in blue. Ⅰ_1 The marker is displayed in yellow. Ⅰ_2 The marker is displayed in green.Ⅰ_3 In this sequence, blue corresponds to the first marking order, yellow corresponds to the second marking order, and green corresponds to the third marking order.

[0134] If in marker Ⅰ_1 marker Ⅰ_2 and marker Ⅰ_3 The markers display the numbers "1", "2", and "3" respectively. The endoscopic operator can mark the points on the endoscopic field of view according to the order represented by these numbers. Specifically, the endoscopic operator can first mark the points according to the markers... Ⅰ_1 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_1 The endoscopic operator can first follow the marker Ⅰ_2 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_2 The endoscopic operator can first follow the marker Ⅰ_3 On the 2D image of view_1, points are marked at the corresponding positions on the view under the microscope, and the computer device obtains Class II markers. Ⅱ_3 .

[0135] To avoid fusion errors, the three selected anatomical points are not on the same straight line. Thus, the three Class I markers are not on the same straight line, and the three Class II markers are not on the same straight line.

[0136] Among the three Class I markers, the Class I marker corresponding to the first marker in the marking order is called marker. Ⅰ_1 Among the three binary markers, the binary marker corresponding to the first marker in the marking order is marker. Ⅱ_1 Therefore, the marker can be Ⅰ_1 and marker Ⅱ_1 Perform matching, and use the marker Ⅰ_1 and marker Ⅱ_1 Consider it as a pair of marker points, marker Ⅰ_1 and marker Ⅱ_1 They correspond to the same anatomical point.

[0137] Among the three Class I markers, the Class I marker corresponding to the second marker in the marking sequence is marker. Ⅰ_2 Among the three binary markers, the binary marker corresponding to the second marking order is marker. Ⅱ_2 Therefore, the marker can be Ⅰ_2 and marker Ⅱ_2 Perform matching, and use the marker Ⅰ_2and marker Ⅱ_2 Consider it as a pair of marker points, marker Ⅰ_2 and marker Ⅱ_2 They correspond to the same anatomical point.

[0138] Among the three Class I markers, the Class I marker corresponding to the third marker in the sequence is called marker. Ⅰ_3 Among the three class II markers, the class II marker corresponding to the third marker in the sequence is marker. Ⅱ_3 Therefore, the marker can be Ⅰ_3 and marker Ⅱ_3 Perform matching, and use the marker Ⅰ_3 and marker Ⅱ_3 Consider it as a pair of marker points, marker Ⅰ_3 and marker Ⅱ_3 They correspond to the same anatomical point.

[0139] Part Three:

[0140] After acquiring multiple marker pairs, the computer device can adjust the virtual camera's viewing angle multiple times. Each time the viewing angle is adjusted, a 2D image of the 3D model from that angle is acquired via the virtual camera. When the position of a Class I marker within at least one marker pair on the 2D image does not coincide with the position of a Class II marker within the endoscopic view, the viewing angle can be adjusted again. Adjustment stops when the positions of Class I markers within each marker pair on the 2D image coincide with the positions of Class II markers within the endoscopic view. The 2D image of the 3D model from that viewing angle is then fused with the endoscopic view, and the fused image is displayed. Furthermore, the computer device can display the pre-operative planned path on the fused image to guide the endoscopic operator in subsequent procedures.

[0141] The above embodiments expand the endoscopic field of view and improve the efficiency of minimally invasive surgery by marking the anatomical points of the target object in the three-dimensional model and the endoscopic field of view. Furthermore, by treating the first-class and second-class markers corresponding to the same anatomical point as a pair, the marker pairs obtained can more accurately fuse the endoscopic field of view with the three-dimensional model and avoid misalignment.

[0142] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0143] Based on the same inventive concept, this application also provides an endoscopic field-of-view expansion device for implementing the endoscopic field-of-view expansion method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the endoscopic field-of-view expansion device provided below can be found in the limitations of the endoscopic field-of-view expansion method described above, and will not be repeated here.

[0144] In one embodiment, such as Figure 4 As shown, an endoscopic field-of-view expansion device is provided, comprising:

[0145] The 3D model acquisition module 401 is used to construct a 3D model of the target object based on preoperative images.

[0146] The marker point acquisition module 402 is used to acquire a type of marker points on the three-dimensional model marking the anatomical points of the target object;

[0147] The marker acquisition module 402 is also used to acquire the second type of markers of the anatomical points marked on the endoscopic field of view.

[0148] The fusion display module 403 is used to fuse and display the endoscopic field of view with the three-dimensional model based on the first type of marker points of the anatomical points in the three-dimensional model and the second type of marker points in the endoscopic field of view, so as to expand the endoscopic field of view.

[0149] In one embodiment, the marker acquisition module 402 is further configured to: acquire the endoscopic field of view of the endoscope and acquire the target two-dimensional image corresponding to the three-dimensional model from the target perspective; adjust the acquisition angle of the endoscope according to the target two-dimensional image until the content of the endoscopic field of view matches the content of the target two-dimensional image; when the content of the endoscopic field of view matches the content of the target two-dimensional image, acquire the dot markings on the endoscopic field of view to obtain the second type of marker points marked by the anatomical points on the endoscopic field of view.

[0150] In one embodiment, the marker acquisition module 402 is further configured to: acquire multiple candidate viewpoints of the target object viewed by the endoscope; and obtain a target two-dimensional image corresponding to the three-dimensional model in the target viewpoint based on the two-dimensional images corresponding to the three-dimensional model in the multiple candidate viewpoints.

[0151] In one embodiment, the fusion display module 403 is further configured to: determine, from among a plurality of first-class marker points in the three-dimensional model and a plurality of second-class marker points in the endoscopic view, a first-class marker point and a second-class marker point corresponding to the same anatomical point, to obtain a marker point pair; and based on the marker point pair, fuse the endoscopic view with the three-dimensional model and display it.

[0152] In one embodiment, the first point set includes a plurality of Class I marker points, and the second point set includes a plurality of Class II marker points; the fusion display module 403 is further configured to: obtain the marking order of each of the Class I marker points in the first point set and the marking order of each of the Class II marker points in the second point set; and match each of the Class I marker points and each of the Class II marker points according to the marking order to obtain marker point pairs.

[0153] In one embodiment, the fusion display module 403 is further configured to: determine a prior marking image and a subsequent marking image in the endoscopic field of view and the target two-dimensional image corresponding to the three-dimensional model in the target view; obtain the prior marking points of the anatomical points in the prior marking image; display characters or colors representing the marking order at each prior marking point according to the marking order of the prior marking points in the corresponding point set; the characters or colors are used to instruct the endoscopic operator to perform marking operations on the subsequent marking image according to the marking order represented by the characters or colors; obtain the subsequent marking points of several anatomical points in the subsequent marking image and the marking order of the subsequent marking points in the corresponding point set according to the marking operations performed by the endoscopic operator on the subsequent marking image; and obtain the marking order of each type of marking point in the first point set and the marking order of each type of marking point in the second point set according to the marking order of the prior marking points in the corresponding point set and the marking order of each type of marking point in the second point set.

[0154] In one embodiment, the fusion display module 403 is further configured to: when a match is determined between one type I marker point and one type II marker point according to the marking order, determine the position of the one type I marker point relative to other points in the first point set and the position of the one type II marker point relative to other points in the second point set; when the position of the one type I marker point relative to other points in the first point set and the position of the one type II marker point relative to other points in the second point set do not match, issue a marker point mismatch alarm message; and, in response to the endoscopy operator receiving the marker point mismatch alarm message and confirming that the one type I marker point and the one type II marker point do not match to the same marker point pair, obtain a marker point pair based on at least one of the type I and type II marker points selected by the endoscopy operator in response to the matching point selection operation of the endoscopy operator.

[0155] Each module in the aforementioned endoscopic field-of-view expansion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0156] In one exemplary embodiment, a computer device is provided, the internal structure of which can be as shown in the figure. Figure 5 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the methods described above. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for expanding the endoscopic field of view.

[0157] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the various method embodiments described above.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various method embodiments described above.

[0160] In one embodiment, a computer program product is provided having a computer program stored thereon, the computer program being executed by a processor of the steps described in the various method embodiments above.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0162] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for expanding the field of view in an endoscope, characterized in that, The method includes: Construct a 3D model of the target object based on preoperative images; Obtain a class of marker points on the 3D model representing the anatomical points of the target object; Obtain the Class II markers of the anatomical points on the endoscopic field of view; Among the several Class I marker points in the 3D model and the several Class II marker points in the endoscopic view, the Class I marker points and Class II marker points corresponding to the same anatomical point are determined to obtain marker point pairs; Determine a two-dimensional image of the three-dimensional model from a certain perspective. The position of the first type of marker points in each marker point pair on the two-dimensional image of the three-dimensional model from that perspective coincides with the position of the second type of marker points on the view under the lens. The two-dimensional image of the three-dimensional model from the stated viewpoint is fused with the endoscopic view and displayed to expand the endoscopic field of view.

2. The method according to claim 1, characterized in that, Obtain the Class II markers of the anatomical points on the endoscopic view, including: Obtain the endoscopic view of the endoscope and the target 2D image corresponding to the 3D model from the target perspective; Adjust the acquisition angle of the endoscope according to the target two-dimensional image until the content of the endoscopic field of view matches the content of the target two-dimensional image; When the content of the microscopic field of view matches the content of the target two-dimensional image, the dot markers on the microscopic field of view are obtained to obtain the second type of markers for the anatomical points on the microscopic field of view.

3. The method according to claim 2, characterized in that, Obtaining the target 2D image corresponding to the 3D model from the target's perspective includes: Obtain multiple candidate viewpoints for viewing the target object using the endoscope; Based on the two-dimensional images corresponding to the three-dimensional model under multiple candidate viewpoints, a target two-dimensional image corresponding to the three-dimensional model under the target viewpoint is obtained.

4. The method according to claim 1, characterized in that, The first point set includes several Class I marker points, and the second point set includes several Class II marker points. From the several Class I marker points in the 3D model and the several Class II marker points in the endoscopic view, Class I and Class II marker points corresponding to the same anatomical point are determined to obtain marker point pairs, including: Obtain the marking order of each of the first-class markers in the first point set and the marking order of each of the second-class markers in the second point set; According to the marking order, each of the first-class marker points and each of the second-class marker points are matched to obtain marker point pairs.

5. The method according to claim 4, characterized in that, Obtaining the labeling order of each of the first-class markers in the first point set and the labeling order of each of the second-class markers in the second point set includes: In the vision under the microscope and the target two-dimensional image corresponding to the three-dimensional model from the target's perspective, determine the image with the dots appearing first and the image with the dots appearing later. Obtain the anatomical points from the prior marking points on the prior marking map; According to the marking order of the prior marked points in the corresponding point set, a character or color is displayed at each prior marked point to represent the marking order; the character or color is used to instruct the endoscopic operator to perform the marking operation on the subsequent image according to the marking order represented by the character or color. Based on the dotting operation performed by the endoscopic operator on the dotting diagram, obtain the subsequent markers of several anatomical points on the dotting diagram and the marking order of the subsequent markers in the corresponding point set; Based on the marking order of the earlier marked points in the corresponding point set and the marking order of the later marked points in the corresponding point set, the marking order of each of the first-class marked points in the first point set and the marking order of each of the second-class marked points in the second point set are obtained.

6. The method according to claim 4, characterized in that, According to the labeling order, each of the first-class label points and each of the second-class label points are matched to obtain label point pairs, including: When a match is determined between a Class I marker and a Class II marker based on the marking order, the position of the Class I marker relative to other points in the first point set and the position of the Class II marker relative to other points in the second point set are determined. When the position of one of the Class I markers relative to other points in the first point set does not match the position of one of the Class II markers relative to other points in the second point set, a marker mismatch alarm message is issued. When the endoscopy operator receives a marker mismatch alarm message and confirms that one of the Class I markers and one of the Class II markers do not match to the same marker pair, in response to the endoscopy operator's matching point selection operation, a marker pair is obtained based on at least one of the Class I and Class II markers selected by the endoscopy operator.

7. An endoscopic field-of-view expansion device, characterized in that, The device includes: The 3D model acquisition module is used to construct a 3D model of the target object based on preoperative images; The marker point acquisition module is used to acquire a class of marker points on the three-dimensional model representing the anatomical points of the target object; The marker acquisition module is also used to acquire the second type of markers of the anatomical points marked on the endoscopic field of view; The fusion display module is used to determine, from among several Class I markers in the 3D model and several Class II markers in the endoscopic view, a Class I marker and a Class II marker corresponding to the same anatomical point, to obtain marker pair; to determine a 2D image of the 3D model from a certain perspective, wherein the position of the Class I marker in each marker pair on the 2D image of the 3D model from that perspective coincides with the position of the Class II marker on the endoscopic view; and to fuse and display the 2D image of the 3D model from that perspective and the endoscopic view to expand the endoscopic field of view.

8. The apparatus according to claim 7, characterized in that, The marker point acquisition module is also used for: Obtain the endoscopic view of the endoscope and the target 2D image corresponding to the 3D model from the target perspective; Adjust the acquisition angle of the endoscope according to the target two-dimensional image until the content of the endoscopic field of view matches the content of the target two-dimensional image; When the content of the microscopic field of view matches the content of the target two-dimensional image, the dot markers on the microscopic field of view are obtained to obtain the second type of markers for the anatomical points on the microscopic field of view.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images