Endoscope visual field expansion method, device and equipment and storage medium

Abstract

The invention relates to the technical field of endoscope view field processing, provides an endoscope view field expansion method, device and equipment and a storage medium, and can expand the endoscope view field. The method comprises the following steps: constructing a three-dimensional model of a target object according to a preoperative image; obtaining a first-class mark point marked on the three-dimensional model by the anatomical point of the target object; acquiring second-class mark points marked on the under-mirror view picture of the endoscope by the anatomical points; and according to the first-class mark points of the anatomical points in the three-dimensional model and the second-class mark points of the anatomical points in the under-mirror view image, fusing and displaying the under-mirror view image and the three-dimensional model so as to expand the view of the endoscope.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of endoscope field of view processing, in particular to an endoscope field of view expansion method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] Neuroendoscopic surgery is a minimally invasive surgical technique used to treat various neurological diseases and problems, such as intracranial tumors, spinal lesions, craniocerebral injuries, cerebrovascular lesions, etc. Precise positioning of the lesion or target area is required before surgery through imaging examination. The doctor needs to operate in a slow and meticulous manner while maintaining the stability of the surgery. The neural structure and lesion condition of each patient may be different, so the surgery needs to develop a corresponding treatment plan according to the individual situation.

[0003] In traditional neuroendoscopic surgery, because the lens is close to the human tissue, the doctor often has only a few centimeters of field of view below the lens during the operation, and in order to improve the clarity, the lens picture is usually magnified, further reducing the doctor's field of view, making the doctor unable to accurately judge the overall condition of the patient, such as the size of the lesion, the relative position of the lesion, the relative relationship between the surgical instrument and the lesion, etc. The same problem also occurs in other minimally invasive surgeries, such as orthopedic minimally invasive surgery. Because the doctor's field of view is reduced during the operation, the doctor cannot accurately judge the overall condition of the patient, such as the size of the lesion, the relative position of the lesion, the relative relationship between the surgical instrument and the lesion, etc., affecting the normal progress of the surgery. SUMMARY

[0004] Therefore, it is necessary to provide an endoscope field of view expansion method, device, computer equipment, storage medium and computer program product to solve the above technical problems.

[0005] The present application provides an endoscope field of view expansion method, which comprises:

[0006] constructing a three-dimensional model of a target object according to a preoperative image;

[0007] obtaining a first type of marker point marked by an anatomical point of the target object on the three-dimensional model;

[0008] obtaining a second type of marker point marked by the anatomical point on an endoscopic under-lens field of view picture;

[0009] fusing and displaying the under-lens field of view picture and the three-dimensional model according to the first type of marker point of the anatomical point on the three-dimensional model and the second type of marker point on the under-lens field of view picture, to expand the endoscope field of view.

[0010] In one embodiment, obtaining the second type of marker point marked by the anatomical point on the endoscopic under-lens field of view picture comprises:

[0011] obtaining a target two-dimensional image corresponding to the three-dimensional model under a target view angle;

[0012] adjusting the collection angle of the endoscope according to the target two-dimensional image until the content of the endoscopic view image matches the content of the target two-dimensional image;

[0013] when the content of the endoscopic view image matches the content of the target two-dimensional image, obtaining a dot marking on the endoscopic view image to obtain a second type of marking point marked by the anatomical point on the endoscopic view image.

[0014] In one embodiment, obtaining a target two-dimensional image corresponding to the three-dimensional model under a target view angle comprises:

[0015] obtaining a plurality of candidate view angles of the endoscope viewing the target object;

[0016] obtaining a target two-dimensional image corresponding to the three-dimensional model under a target view angle according to two-dimensional images corresponding to the three-dimensional model under a plurality of candidate view angles.

[0017] In one embodiment, fusing and displaying the endoscopic view image and the three-dimensional model according to the first type of marking point of the anatomical point in the three-dimensional model and the second type of marking point in the endoscopic view image comprises:

[0018] determining a first type of marking point and a second type of marking point corresponding to the same anatomical point from a plurality of first type of marking points of the three-dimensional model and a plurality of second type of marking points of the endoscopic view image to obtain a marking point pair;

[0019] fusing and displaying the endoscopic view image and the three-dimensional model based on the marking point pair.

[0020] In one embodiment, the first point set comprises a plurality of first type of marking points, and the second point set comprises a plurality of second type of marking points; determining a first type of marking point and a second type of marking point corresponding to the same anatomical point from a plurality of first type of marking points of the three-dimensional model and a plurality of second type of marking points of the endoscopic view image to obtain a marking point pair comprises:

[0021] obtaining a marking order of each first type of marking point in the first point set and a marking order of each second type of marking point in the second point set;

[0022] matching each first type of marking point and each second type of marking point according to the marking order to obtain a marking point pair.

[0023] In an embodiment, the marking order of each of the first type of marking points in the first point set and the marking order of each of the second type of marking points in the second point set are obtained, comprising:

[0024] In the under-mirror view field picture and the target two-dimensional diagram corresponding to the three-dimensional model under the target view angle, a dotting-first diagram and a dotting-later diagram are determined;

[0025] The anatomic points in the first marking points in the dotting-first diagram are obtained;

[0026] According to the marking order of the first marking points in the corresponding point set, characters or colors representing the marking order are displayed at each of the first marking points; the characters or colors are used to instruct the endoscope operator to perform dotting operation on the dotting-later diagram according to the marking order represented by the characters or colors;

[0027] According to the dotting operation performed by the endoscope operator on the dotting-later diagram, the anatomic points in the second marking points in the dotting-later diagram and the marking order of the second marking points in the corresponding point set are obtained;

[0028] According to the marking order of the first marking points in the corresponding point set and the marking order of the second marking points in the corresponding point set, the marking order of each of the first type of marking points in the first point set and the marking order of each of the second type of marking points in the second point set are obtained.

[0029] In an embodiment, according to the marking order, each of the first type of marking points and each of the second type of marking points are matched to obtain a marking point pair, comprising:

[0030] When it is determined according to the marking order that one of the first type of marking points and one of the second type of marking points are matched, the position of the one of the first type of marking points relative to other points in the first point set and the position of the one of the second type of marking points relative to other points in the second point set are determined;

[0031] When the position of the one of the first type of marking points relative to other points in the first point set and the position of the one of the second type of marking points relative to other points in the second point set are not matched, a marking point mismatch warning message is issued;

[0032] In the case that the endoscope operator receives the marking point mismatch warning message and confirms that the one of the first type of marking points and the one of the second type of marking points are not matched to the same marking point pair, a marking point pair is obtained based on at least one of the first type of marking points and the second type of marking points selected by the endoscope operator in response to a matching point selection operation of the endoscope operator.

[0033] The present application provides an endoscope view field extension device, which comprises:

[0034] The three-dimensional model acquisition module is configured to construct a three-dimensional model of the target object according to preoperative images.

[0035] The marker point acquisition module is configured to acquire a first type of marker point of an anatomical point of the target object marked on the three-dimensional model.

[0036] The marker point acquisition module is further configured to acquire a second type of marker point of the anatomical point marked on an endoscopic under-mirror field of view image.

[0037] The fusion display module is configured to fuse and display the under-mirror field of view image and the three-dimensional model according to the first type of marker point of the anatomical point on the three-dimensional model and the second type of marker point of the anatomical point on the under-mirror field of view image, so as to expand the endoscopic field of view.

[0038] The present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor executes the above method.

[0039] The present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to execute the above method.

[0040] The present application provides a computer program product, which stores a computer program, and the computer program is executed by a processor to execute the above method.

[0041] In the above endoscopic field of view expansion method, device, computer device, storage medium and computer program product, a three-dimensional model of a target object is constructed according to preoperative images, a first type of marker point of an anatomical point of the target object marked on the three-dimensional model is acquired, a second type of marker point of the anatomical point marked on an endoscopic under-mirror field of view image is acquired, the under-mirror field of view image and the three-dimensional model are fused and displayed according to the first type of marker point of the anatomical point on the three-dimensional model and the second type of marker point of the anatomical point on the under-mirror field of view image, and the endoscopic field of view is expanded based on the marker points of the anatomical point of the target object on the three-dimensional model and the under-mirror field of view image, so as to improve the efficiency of minimally invasive surgery. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative effort.

[0043] Figure 1 It is a flowchart of the endoscopic field of view expansion method in one embodiment;

[0044] Figure 2An application environment diagram of the endoscope field of view expansion method in one embodiment;

[0045] Figure 3 A schematic diagram of content change display in one embodiment;

[0046] Figure 4 A structural block diagram of the endoscope field of view expansion device in one embodiment;

[0047] Figure 5 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0049] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0050] The endoscope field of view expansion method provided by the present application can be executed by a computer device, which can be a device in an endoscope system or a device that is not in an endoscope system. The method includes the steps shown in Figure 1 The steps are as follows:

[0051] In step S101, a three-dimensional model of a target object is constructed according to a preoperative image.

[0052] Minimally invasive surgery is a technique of performing surgery in a specific human or a specific animal using an endoscope. According to the surgical site or organ involved, minimally invasive surgery can include neuroendoscopic surgery, orthopedic minimally invasive surgery, etc. The surgical site or organ involved in minimally invasive surgery can be regarded as a target object. Before the minimally invasive surgery begins, an image of the target object can be acquired by CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) and the like to obtain a three-dimensional model of the target object.

[0053] For example, in orthopedic minimally invasive surgery, the knee joint of a specific human or a specific animal involved in the surgery can be regarded as a target object, and a three-dimensional model of the target object can be obtained by CT and MRI and the like.

[0054] Step S102, obtaining a type of marking points where anatomical points of the target object are marked on the three-dimensional model;

[0055] Step S103: obtaining the second type of marking points of the anatomical points on the endoscope visual field screen.

[0056] The target object has some anatomically significant landmarks, which can be referred to as anatomical points. After obtaining a three-dimensional model of the target object, the present application can obtain the markers of the target object's anatomical points marked on the three-dimensional model and on the image under the microscope. To distinguish the markers of the anatomical points on the three-dimensional model from those on the image under the microscope, the markers marked on the three-dimensional model are referred to as first-class markers, and the markers marked on the image under the microscope are referred to as second-class markers.

[0057] For example, the target object has anatomical points A, B, and C. A first type of marker point can be obtained, which is a marker for each of the anatomical points A, B, and C on the three-dimensional model. Alternatively, a second type of marker point can be obtained, which is a marker for each of the anatomical points A, B, and C on the endoscope's visual field.

[0058] S102 and S103 are mainly used to identify the corresponding steps, and do not indicate the execution order of the two steps. In some embodiments, the step corresponding to S102 may be executed first and then the step corresponding to S103, or the step corresponding to S103 may be executed first and then the step corresponding to S102.

[0059] In step S104 , based on the first type of marker points of the anatomical points on the three-dimensional model and the second type of marker points on the endoscopic visual field, the endoscopic visual field image is fused and displayed with the three-dimensional model to expand the endoscopic visual field.

[0060] Exemplarily, after obtaining the first-class marker points and the second-class marker points of the anatomical points {A, B, C}, the endoscopic field of view and the three-dimensional model can be fused and displayed according to the correspondence between the marker points. Specifically, the fusion can be to superimpose the endoscopic field of view and the three-dimensional model, and the content on the three-dimensional model that does not overlap with the endoscopic field of view can be displayed with a lower transparency, and the content on the three-dimensional model that overlaps with the endoscopic field of view can be displayed with a higher transparency. The endoscopic field of view is expanded while avoiding interfering with the doctor's viewing of the endoscopic field of view. Among them, the transparency lower than the preset value can be used as the lower transparency, and the transparency higher than the preset value can be used as the higher transparency.

[0061] In the endoscopic field of view expansion method, a three-dimensional model of the target object is constructed according to a preoperative image, a first type of marking point of an anatomical point of the target object marked on the three-dimensional model is obtained, a second type of marking point of the anatomical point marked in an endoscopic field of view picture is obtained, the field of view picture and the three-dimensional model are fused and displayed according to the first type of marking point of the anatomical point on the three-dimensional model and the second type of marking point of the anatomical point in the field of view picture, and the field of view picture is expanded and the efficiency of minimally invasive surgery is improved based on the marking points of the anatomical point of the target object in the three-dimensional model and the field of view picture of the endoscope.

[0062] In one embodiment, step S104 can specifically include the following steps: among the plurality of first type of marking points of the three-dimensional model and the plurality of second type of marking points of the field of view picture, a first type of marking point and a second type of marking point corresponding to the same anatomical point are determined to obtain a marking point pair; and the field of view picture and the three-dimensional model are fused and displayed based on the marking point pair.

[0063] Specifically, after obtaining the plurality of first type of marking points on the three-dimensional model and the plurality of second type of marking points of the field of view picture, the first type of marking point and the second type of marking point corresponding to the same anatomical point are determined, and the first type of marking point and the second type of marking point corresponding to the same anatomical point are regarded as a marking point pair.

[0064] After obtaining the plurality of marking point pairs, the field of view picture and the three-dimensional model are fused and displayed based on the plurality of marking point pairs. Specifically, the angle of view of the virtual camera can be adjusted multiple times; each time the angle of view is adjusted, a two-dimensional image of the three-dimensional model at the angle of view can be acquired by the virtual camera, if the position of the first type of marking point in at least one marking point pair on the two-dimensional image does not coincide with the position of the second type of marking point on the field of view picture, the next angle of view adjustment can be performed, and when the position of the first type of marking point in each marking point pair on the two-dimensional image coincides with the position of the second type of marking point on the field of view picture, the adjustment can be stopped, and the two-dimensional image of the three-dimensional model at the angle of view and the field of view picture are fused and displayed.

[0065] In the above embodiment, the first type of marking point and the second type of marking point corresponding to the same anatomical point are regarded as a pair, and the marking point pair obtained based thereon can more accurately fuse and display the field of view picture and the three-dimensional model, avoiding misalignment.

[0066] The first type of marking point of the anatomical point of the target object on the three-dimensional model of the target object can be obtained by a marking method of an endoscopic operator (belonging to a manual marking method), can be obtained by a model obtained by deep learning (belonging to an automatic marking method), or can be obtained by a combination of the automatic marking method and the manual marking method.

[0067] The second-type mark points of the anatomical points of the target object in the endoscopic view can be obtained by manual marking. After obtaining the first-type mark points of the anatomical points on the three-dimensional model of the target object, the method can proceed to step S103. Step S103 can include: obtaining an endoscopic view and a target two-dimensional image corresponding to the three-dimensional model at a target view angle; adjusting the collection angle of the endoscope according to the target two-dimensional image until the content of the endoscopic view matches the content of the target two-dimensional image; and obtaining the mark points of the anatomical points in the endoscopic view when the content of the endoscopic view matches the content of the target two-dimensional image.

[0068] The target two-dimensional image corresponding to the three-dimensional model at the target view angle can be obtained by setting a view angle, collecting a two-dimensional image of the three-dimensional model at the view angle, taking the view angle as the target view angle, and taking the two-dimensional image of the three-dimensional model at the view angle as the target two-dimensional image.

[0069] During the surgery, the target two-dimensional image and the real-time endoscopic view of the endoscope can be displayed on the screen at the same time, as shown in FIG. 6, so that the endoscope operator can simultaneously view the target two-dimensional image and the endoscopic view. Figure 2

[0070] The target two-dimensional image corresponding to the three-dimensional model at the target view angle can also be obtained by: obtaining a plurality of candidate view angles of the endoscope viewing the target object; and obtaining the target two-dimensional image corresponding to the three-dimensional model at the target view angle according to the two-dimensional images corresponding to the three-dimensional model at the plurality of candidate view angles.

[0071] According to the surgical planning path of the present minimally invasive surgery, a plurality of view angles of the endoscope viewing the target object that can appear during the surgery can be determined, and the view angles are taken as candidate view angles to obtain the plurality of candidate view angles. Then, the two-dimensional image corresponding to the three-dimensional model of the target object at each candidate view angle can be obtained. For example, the candidate view angles include view_1, view_2, and view_3. The two-dimensional image corresponding to the three-dimensional model of the target object at view_1 (hereinafter referred to as view_1 two-dimensional image), the two-dimensional image corresponding to the three-dimensional model of the target object at view_2 (hereinafter referred to as view_2 two-dimensional image), and the two-dimensional image corresponding to the three-dimensional model of the target object at view_3 (hereinafter referred to as view_3 two-dimensional image) can be obtained. The view_1 two-dimensional image, the view_2 two-dimensional image, and the view_3 two-dimensional image can be displayed together with the endoscopic view, as shown in FIG. 6. Figure 3

[0072] ​​The endoscope operator can select a two-dimensional view in which the image content is close to the current content of the endoscopic view in the view_1 two-dimensional view, the view_2 two-dimensional view, and the view_3 two-dimensional view, for example, the endoscope operator selects the view_1 two-dimensional view, and the computer device can take the view_1 two-dimensional view as the target two-dimensional view based on the selection operation of the endoscope operator on the view_1 two-dimensional view, and accordingly, the view_1 as the target view angle.

[0073] After the computer device receives the selection operation of the endoscope operator on the view_1 two-dimensional view, the computer device can cancel the display or reduce the display of the two-dimensional view under the non-target view angle, and enlarge the display of the view_1 two-dimensional view and the endoscopic view. Referring again to Figure 3 , after the endoscope operator selects the view_1 two-dimensional view by the mouse, the computer device can not display the view_2 two-dimensional view and the view_3 two-dimensional view, and enlarge the display of the view_1 two-dimensional view and the endoscopic view in response to the operation. Thus, in the case of limited display area, more display area can be used to display the two-dimensional view under the target view angle and the endoscopic view to facilitate manual dotting.

[0074] The target two-dimensional view can be used to instruct the endoscope operator or the automatic control device to adjust the collection angle of the endoscope, so that the content of the endoscopic view changes; when the content of the endoscopic view matches the content of the target two-dimensional view, the current collection angle of the endoscope can be locked, so that the content of the endoscopic view matches the content of the two-dimensional view of the three-dimensional model under the target view angle.

[0075] The way to determine that the content of the endoscopic view matches the content of the target two-dimensional view can be: obtaining the similarity between the content of the endoscopic view and the content of the target two-dimensional view, and when the similarity is higher than a similarity threshold, it is determined that the content of the endoscopic view matches the content of the target two-dimensional view.

[0076] In the case where a type of marker points are determined by automatic dotting in advance, the target two-dimensional view can include a type of marker points of the three-dimensional model presented under the target view angle, and the type of marker points are presented on the target two-dimensional view under the target view angle; the endoscope operator can directly dot on the endoscopic view based on the type of marker points presented on the target two-dimensional view, or first adjust the type of marker points determined by automatic dotting, and after adjustment, the positions of the type of marker points on the three-dimensional model and the target two-dimensional view can change, and the endoscope operator dots on the endoscopic view based on the type of marker points presented on the adjusted target two-dimensional view.

[0077] Specifically, the endoscope operator can watch the type of marker points presented by the target two-dimensional graph, determine the position of the type of marker points on the target two-dimensional graph, and mark the corresponding position on the under-scope view image. The computer device can obtain the type two marker points of the anatomical point on the under-scope view image according to the marked position on the under-scope view image.

[0078] The embodiment mainly performs marking based on the user interaction mode to obtain the type two marker points of the anatomical point on the under-scope view image. Before marking, the target two-dimensional graph and the under-scope view image are displayed to instruct the endoscope operator to adjust the collection angle of the endoscope until the content of the under-scope view image matches the content of the target two-dimensional graph. Thus, the endoscope operator can mark the corresponding position on the under-scope view image according to the position of the type of marker points presented on the two-dimensional graph, thereby obtaining the type two marker points of the anatomical point on the under-scope view image.

[0079] The following specifically introduces the implementation of obtaining the type one marker points and the type two marker points of the anatomical point:

[0080] Method one:

[0081] In the case that the type one marker points of the anatomical point are determined by the automatic marking mode, the type two marker points of the anatomical point can be directly obtained according to the type one marker points determined by the automatic marking mode, or the type one marker points determined by the automatic marking mode can be adjusted by the endoscope operator, and the type two marker points of the anatomical point are obtained based on the adjusted type one marker points.

[0082] The type two marker points of the anatomical point are mainly obtained by the manual marking mode. Specifically, the view_1 two-dimensional graph and the under-scope view image can be displayed on the display screen, and the type one marker points are presented on the corresponding position of the view_1 two-dimensional graph. The view_1 two-dimensional graph can be used to instruct the endoscope operator to adjust the collection angle of the endoscope so that the content of the under-scope view image changes. When the content of the under-scope view image matches the content of the view_1 two-dimensional graph, the current collection angle of the endoscope can be locked so that the content of the under-scope view image remains matched with the content of the view_1 two-dimensional graph. At this time, the endoscope operator can watch the type of marker points presented by the view_1 two-dimensional graph, determine the position of the type of marker points on the view_1 two-dimensional graph, and mark the corresponding position on the under-scope view image. The computer device can obtain the type two marker points of the anatomical point on the under-scope view image according to the marked position on the under-scope view image.

[0083] Method two:

[0084] In the case that the one type of marking points of the anatomical point are not determined by the automatic marking manner, the one type of marking points and the two types of marking points of the anatomical point can be determined by the manual marking manner. Specifically, the view_1 two-dimensional graph and the in-vivo field of view picture can be displayed on the display screen. The view_1 two-dimensional graph can be used to instruct the endoscope operator to adjust the collection angle of the endoscope, so that the content of the in-vivo field of view picture changes; when the content of the in-vivo field of view picture matches the content of the view_1 two-dimensional graph, the current collection angle of the endoscope can be locked, so that the content of the in-vivo field of view picture remains matched with the content of the view_1 two-dimensional graph. At this time, the endoscope operator can mark on the view_1 two-dimensional graph and the in-vivo field of view picture to obtain the one type of marking points and the two types of marking points of the anatomical point.

[0085] Specifically, the endoscope operator can mark on the view_1 two-dimensional graph first and then mark on the in-vivo field of view picture, or mark on the in-vivo field of view picture first and then mark on the view_1 two-dimensional graph. The endoscope operator can select one of the view_1 two-dimensional graph and the in-vivo field of view picture, and the selection operation at this time is the first selection operation. Based on the first selection operation of the endoscope operator, the computer device can enlarge the display of the selected graph (hereinafter referred to as the first marking graph) to enable the endoscope operator to mark on the first marking graph.

[0086] Based on the marking operation of the endoscope operator on the first marking graph, the computer device can obtain the corresponding marking points of the anatomical point. After the endoscope operator completes the marking on the first marking graph, the endoscope operator can select the other of the view_1 two-dimensional graph and the in-vivo field of view picture, and the selection operation at this time is the second selection operation. In response to the second selection operation of the endoscope operator, the computer device can reduce the display of the first marking graph and enlarge the display of the selected graph (hereinafter referred to as the second marking graph) to enable the endoscope operator to mark on the second marking graph. Based on the marking operation of the endoscope operator on the second marking graph, the computer device can obtain the corresponding marking points of the anatomical point.

[0087] Specifically, when the first marking graph is the view_1 two-dimensional graph, the second marking graph can be the in-vivo field of view picture; when the first marking graph is the in-vivo field of view picture, the second marking graph can be the view_1 two-dimensional graph.

[0088] In one embodiment, the first point set includes a plurality of one type of marking points, and the second point set includes a plurality of two types of marking points. Among the plurality of one type of marking points of the three-dimensional model and the plurality of two types of marking points of the in-vivo field of view picture, the one type of marking points and the two types of marking points corresponding to the same anatomical point are determined to obtain a marking point pair. Specifically, the marking order of each one type of marking point in the first point set and the marking order of each two type of marking point in the second point set are obtained; and the marking point pair is obtained by matching each one type of marking point and each two type of marking point according to the marking order.

[0089] Exemplarily, if the number of the first type of marker points is 3, the first type of marker points are named in the order of marker Ⅰ_1 , marker Ⅰ_2 and marker Ⅰ_3 .

[0090] If the number of the second type of marker points is 3, the second type of marker points are named in the order of marker Ⅱ_1 , marker Ⅱ_2 and marker Ⅱ_3 .

[0091] wherein the first type of marker point corresponding to the first marker order in the first point set is marker Ⅰ_1 , and the second type of marker point corresponding to the first marker order in the second point set is marker Ⅱ_1 , thus, marker Ⅰ_1 and marker Ⅱ_1 can be matched, marker Ⅰ_1 and marker Ⅱ_1 are regarded as a marker point pair, and marker Ⅰ_1 and marker Ⅱ_1 correspond to the same anatomical point.

[0092] Further, the marker orders of the first type of marker points in the first point set and the marker orders of the second type of marker points in the second point set are obtained by: determining the first drawing graph and the second drawing graph in the field of view under the microscope and in the target two-dimensional graph corresponding to the three-dimensional model under the target view angle; obtaining the first type of marker points in the first drawing graph; displaying characters or colors representing the marker orders at the first type of marker points according to the marker orders of the first type of marker points in the corresponding point set; the characters or colors are used to instruct the endoscopic operator to perform the marking operation on the second drawing graph according to the marker orders represented by the characters or colors; obtaining the second type of marker points in the second drawing graph and the marker orders of the second type of marker points in the corresponding point set according to the marking operation performed by the endoscopic operator on the second drawing graph; and obtaining the marker orders of the first type of marker points in the first point set and the marker orders of the second type of marker points in the second point set according to the marker orders of the first type of marker points in the corresponding point set and the marker orders of the second type of marker points in the corresponding point set.

[0093] Among them, when the first image with the dots is the field of view under the microscope, the first marked point is a first-class marked point, and the corresponding point set to which the first marked point belongs is the first point set; when the second image with the dots is the target two-dimensional image, the second-class marked point is a second-class marked point, and the corresponding point set to which the second marked point belongs is the second point set; when the first image with the dots is the target two-dimensional image, the first marked point is a second-class marked point, and the corresponding point set to which the first marked point belongs is the second point set; when the second image with the dots is the field of view under the microscope, the second-class marked point is a first-class marked point, and the corresponding point set to which the second marked point belongs is the first point set.

[0094] As an example, the target two-dimensional image is the first dot image, the target two-dimensional image is the view_1 two-dimensional image, and the second dot image is the field of view under the microscope:

[0095] If the number of anatomical points is three, and the first-class marker points are determined for the anatomical points using an automatic marking method, the number of the first-class marker points is three. The computer device can display the view_1 two-dimensional image, and according to view_1, the corresponding positions of the three first-class marker points on the view_1 two-dimensional image are presented. The endoscope operator can sequentially mark the three first-class marker points by, for example, clicking on the three first-class marker points in sequence with a mouse. The computer device can determine the marking order of the three first-class marker points within the first point set based on the sequential marking operation of the endoscope operator.

[0096] When determining the first-class marking points of the anatomical points by manual marking, the endoscopic operator can mark the first-class marking points corresponding to the three anatomical points on the view_1 two-dimensional image. Based on the marking operation of the endoscopic operator, the computer device can determine the three first-class marking points on the view_1 two-dimensional image and the marking order of each first-class marking point in the first point set.

[0097] According to the marking order from first to last, these three first-class marking points are recorded as markers Ⅰ_1 、marker Ⅰ_2 and marker Ⅰ_3 , where characters or colors representing the marking order are displayed at each type of marking point, for example, Ⅰ_1 The number "1" is displayed at the marker. Ⅰ_2 The number "2" is displayed at the marker Ⅰ_3 The number "3" is displayed at the same place, and for example, the marker is displayed in blue. Ⅰ_1 , display the marker in yellow Ⅰ_2 , display the marker in green Ⅰ_3 , where blue corresponds to the first marking order, yellow corresponds to the second marking order, and green corresponds to the third marking order.

[0098] If the markerⅠ_1 , marker Ⅰ_2 , and marker Ⅰ_3 , the endoscope operator can mark on the endoscopic view according to the marker order represented by the numbers; specifically, the endoscope operator can first mark on the endoscopic view according to the position of marker Ⅰ_1 on the view_1 two-dimensional graph, and the computer device obtains the second-type marker point marker Ⅱ_1 , the endoscope operator can first mark on the endoscopic view according to the position of marker Ⅰ_2 on the view_1 two-dimensional graph, and the computer device obtains the second-type marker point marker Ⅱ_2 , the endoscope operator can first mark on the endoscopic view according to the position of marker Ⅰ_3 on the view_1 two-dimensional graph, and the computer device obtains the second-type marker point marker Ⅱ_3 .

[0099] In the first point set, the first-type marker point corresponding to the first marker order is marker Ⅰ_1 , and in the second point set, the second-type marker point corresponding to the first marker order is marker Ⅱ_1 , thus, marker Ⅰ_1 and marker Ⅱ_1 can be matched, marker Ⅰ_1 and marker Ⅱ_1 are regarded as a marker point pair, and marker Ⅰ_1 and marker Ⅱ_1 correspond to the same anatomical point.

[0100] In the first point set, the first-type marker point corresponding to the second marker order is marker Ⅰ_2 , and in the second point set, the second-type marker point corresponding to the second marker order is marker Ⅱ_2 , thus, marker Ⅰ_2 and marker Ⅱ_2 can be matched, marker Ⅰ_2 and marker Ⅱ_2 are regarded as a marker point pair, and marker Ⅰ_2 and marker Ⅱ_2 correspond to the same anatomical point.

[0101] In the first point set, the first-type marker point corresponding to the third marker order is marker Ⅰ_3 , and in the second point set, the second-type marker point corresponding to the third marker order is marker Ⅱ_3, so the marker can be Ⅰ_3 and marker Ⅱ_3 Match and mark Ⅰ_3 and marker Ⅱ_3 Considered as a marker point pair, marker Ⅰ_3 and marker Ⅱ_3 Corresponding to the same anatomical point.

[0102] This embodiment uses the marking order of the marking points in the corresponding point set to determine the first-class marking points and the second-class marking points corresponding to the same anatomical point. Based on the marking point pairs formed in this way, the two-dimensional image of the three-dimensional model under the target perspective and the microscopic field of view are fused and displayed to avoid misalignment.

[0103] In order to further avoid the occurrence of misalignment, the present application also provides an embodiment, which uses the relative position of the marker point and other marker points in the corresponding category point set for auxiliary verification. Specifically, according to the marking order, each Class I marker point and each Class II marker point are matched to obtain a marker point pair, which can include: when it is determined that one of the Class I marker points and one of the Class II marker points match according to the marking order, determining the position of one of the Class I marker points relative to other points in the first point set and the position of one of the Class II marker points relative to other points in the second point set; when the position of one of the Class I marker points relative to other points in the first point set and the position of one of the Class II marker points relative to other points in the second point set do not match, issuing a marker point mismatch alarm message; when the endoscope operator receives the marker point mismatch alarm message and confirms that one of the Class I marker points and one of the Class II marker points do not match to the same marker point pair, in response to the endoscope operator's matching point selection operation, obtaining a marker point pair based on at least one of the Class I marker points and the Class II marker points selected by the endoscope operator.

[0104] For example, according to the marking order, a type of marker point marker is determined Ⅰ_1 And the second type of marker Ⅱ_1 After matching, you can get a type of marker Ⅰ_1 Relative to other points in the first point set (such as marker Ⅰ_2 and marker Ⅰ_3 ) and the second type of markers Ⅱ_1 Relative to other points in the second point set (such as marker Ⅱ_2 and marker Ⅱ_3 ) location.

[0105] For example, if according to a type of marker Ⅰ_1 Relative to marker Ⅰ_2 and markerⅠ_3 The position of the first type of marker marker Ⅰ_1 is determined according to the position of marker Ⅰ_2 and the position of the second type of marker marker Ⅰ_3 is determined according to the position of marker Ⅱ_1 and the position of marker Ⅱ_2 . Ⅱ_3 The position of the second type of marker marker Ⅱ_1 is determined according to the position of marker Ⅱ_2 and the position of marker Ⅱ_3 . At this time, the position of the first type of marker marker Ⅰ_1 relative to other points in the first point set matches the position of the second type of marker marker Ⅱ_1 relative to other points in the second point set.

[0106] Further, the position of the first type of marker marker Ⅰ_1 is determined according to the position of marker Ⅰ_2 . The position of the first type of marker marker Ⅰ_1 is determined to be below marker Ⅰ_2 . The position of the second type of marker marker Ⅱ_1 is determined according to the position of marker Ⅱ_2 . The position of the second type of marker marker Ⅱ_1 is determined to be below marker Ⅱ_2 . At this time, the position of the first type of marker marker Ⅰ_1 relative to other points in the first point set matches the position of the second type of marker marker Ⅱ_1 relative to other points in the second point set.

[0107] When the position of the first type of marker marker Ⅰ_1 relative to other points in the first point set matches the position of the second type of marker marker Ⅱ_1 relative to other points in the second point set, the first type of marker marker Ⅰ_1 and the second type of marker marker Ⅱ_1 are determined to correspond to the same anatomical point, and can be regarded as a marker pair.

[0108] When the position of the first type of marker marker Ⅰ_1 relative to other points in the first point set does not match the position of the second type of marker marker Ⅱ_1 relative to other points in the second point set, the computer device can issue a marker mismatch warning message.

[0109] The marker point mismatch alarm message is used to indicate the endoscope operator to confirm whether a first type marker point marker Ⅰ_1 and a second type marker point marker Ⅱ_1 correspond to the same anatomical point or are the same marker point pair; if the endoscope operator determines that the first type marker point marker Ⅰ_1 and the second type marker point marker Ⅱ_1 do not correspond to the same anatomical point or are not the same marker point pair, the endoscope operator can select a marker point that matches the second type marker point marker Ⅰ_2 and marker Ⅰ_3 from the first type marker point marker Ⅱ_1 ; in response to the matching point selection operation of the endoscope operator, the computer device can match the first type marker point selected by the endoscope operator with the second type marker point marker Ⅱ_1 , and regard them as the same marker point pair.

[0110] Similarly, the endoscope operator can also select a marker point that matches the first type marker point marker Ⅱ_2 from the second type marker point marker Ⅱ_3 and marker Ⅰ_1 ; in response to the matching point selection operation of the endoscope operator, the computer device can match the second type marker point selected by the endoscope operator with the first type marker point marker Ⅰ_1 , and regard them as the same marker point pair.

[0111] In the above manner, when the position of the first type marker point marker Ⅰ_2 relative to other points in the first point set matches the position of the second type marker point marker Ⅱ_2 relative to other points in the second point set, it can be determined that the first type marker point marker Ⅰ_2 and the second type marker point marker Ⅱ_2 correspond to the same anatomical point, and can be regarded as a marker point pair. When the position of the first type marker point marker Ⅰ_2 relative to other points in the first point set does not match the position of the second type marker point marker Ⅱ_2 relative to other points in the second point set, the corresponding processing manner can refer to the above content.

[0112] When it is determined that the position of the first type marker point marker Ⅰ_3 relative to other points in the first point set matches the position of the second type marker point marker Ⅱ_3 relative to other points in the second point set, it can be determined that the first type marker point marker Ⅰ_3 and the second type marker point marker Ⅱ_3The same anatomical point can be considered as a marker point pair. When a type of marker point marker Ⅰ_3 The position relative to other points in the first point set does not match the position of the second type of marker point marker Ⅱ_3 When the position relative to other points in the second point set does not match the position of the second type of marker point marker

[0113] In order to better understand the above method, the following detailed description of an application example of the endoscopic field of view expansion method in the present application, the minimally invasive surgery of the application example is orthopedic minimally invasive surgery, and the target object is the knee joint, and the number of anatomical points involved is three.

[0114] First part:

[0115] After obtaining the preoperative image by CT and MRI, a three-dimensional model of the knee joint can be obtained based on the preoperative image. Specifically, a trained segmentation network can be used to segment the preoperative image to obtain the knee joint. The segmentation network can be a Unet network (also known as U-Net, a convolutional neural network that can be used for image segmentation).

[0116] The three-dimensional model of the knee joint can be reconstructed with various modalities of data, such as CT data and MRI data. With the help of MRI data, not only the three-dimensional model of the bone of the knee joint can be extracted by the segmentation network, but also the soft tissue and bone marrow around the knee joint can be extracted. Optionally, the three-dimensional model of the bone of the knee joint is segmented by CT data, and the three-dimensional model of the tissue of the knee joint is segmented by MRI data, and the fusion result is obtained by fusing the three-dimensional model of the bone of the knee joint and the three-dimensional model of the tissue of the knee joint.

[0117] Then, the three-dimensional model of the knee joint can be obtained by processing the fusion result by surface reconstruction and other techniques. Surface reconstruction is a commonly used three-dimensional reconstruction technique that reconstructs a three-dimensional model by extracting the surface geometry of an object, and optimizes the reconstructed bone model, including removing redundant noise, smoothing the surface, filling defects, etc. to further improve the accuracy and integrity of the model.

[0118] Optionally, the source of the three-dimensional model of the knee joint includes but is not limited to a model reconstructed based on preoperative images of the patient, a standard bone model, and a parameterized bone model.

[0119] Second part: Obtain the one type of marker point of the anatomical point of the knee joint in the three-dimensional model and the two types of marker points in the endoscopic field of view picture.

[0120] Specifically, according to the surgical planning path of the present orthopedic minimally invasive surgery, several view angles (such as view_1, view_2 and view_3) that may appear during the surgery can be taken as candidate view angles, and the two-dimensional graphs of the knee joint three-dimensional model under view_1, view_2 and view_3 can be obtained.

[0121] During the orthopedic minimally invasive surgery, the endoscope under-view field picture, the view_1 two-dimensional graph, the view_2 two-dimensional graph and the view_3 two-dimensional graph of the knee joint three-dimensional model can be displayed. The endoscope operator can select a two-dimensional graph in the view_1 two-dimensional graph, the view_2 two-dimensional graph and the view_3 two-dimensional graph, which has a content close to the current content of the endoscope under-view field picture, for example, the view_1 two-dimensional graph is selected. Based on the selection operation of the endoscope operator on the view_1 two-dimensional graph, the computer device can take the view_1 two-dimensional graph as the target two-dimensional graph.

[0122] After the computer device receives the selection operation of the endoscope operator on the view_1 two-dimensional graph, the view_2 two-dimensional graph and the view_3 two-dimensional graph can be cancelled or displayed in a reduced manner, and the view_1 two-dimensional graph and the endoscope under-view field picture can be displayed in an enlarged manner. Thus, in the case of limited display area, more display area can be released to display the view_1 two-dimensional graph and the endoscope under-view field picture for manual dotting.

[0123] The first way: both the first type of marker points and the second type of marker points are determined by manual dotting:

[0124] The endoscope operator can dot the first type of marker points corresponding to the three anatomical points on the view_1 two-dimensional graph. Based on the dotting operation of the endoscope operator, the computer device can determine the three first type of marker points on the view_1 two-dimensional graph and the marker order between the first type of marker points.

[0125] According to the marker order from the first to the last, the three first type of marker points are respectively marker Ⅰ_1 , marker Ⅰ_2 and marker Ⅰ_3 , wherein the characters or colors representing the marker order are displayed at each first type of marker point, for example, the number "1" is displayed at marker Ⅰ_1 , the number "2" is displayed at marker Ⅰ_2 , and the number "3" is displayed at marker Ⅰ_3 , and for another example, marker Ⅰ_1 is displayed in blue, marker Ⅰ_2 is displayed in yellow, and marker Ⅰ_3 is displayed in green, wherein the blue corresponds to the first marker order, the yellow corresponds to the second marker order, and the green corresponds to the third marker order.

[0126] If the numbers "1", "2" and "3" are displayed at marker Ⅰ_1 , marker Ⅰ_2 and marker Ⅰ_3 respectively, the endoscope operator can mark points on the endoscopic view according to the marker order represented by the numbers; specifically, the endoscope operator can first mark a point on the endoscopic view at the position corresponding to the position of marker Ⅰ_1 on the view_1 two-dimensional image, and the computer device obtains the second type of marker point marker Ⅱ_1 , the endoscope operator can first mark a point on the endoscopic view at the position corresponding to the position of marker Ⅰ_2 on the view_1 two-dimensional image, and the computer device obtains the second type of marker point marker Ⅱ_2 , the endoscope operator can first mark a point on the endoscopic view at the position corresponding to the position of marker Ⅰ_3 on the view_1 two-dimensional image, and the computer device obtains the second type of marker point marker Ⅱ_3 .

[0127] The second way: the first type of marker point is determined by automatic marking, and the second type of marker point is determined by manual marking:

[0128] The computer device can display the view_1 two-dimensional image, and present the first type of marker points of the three anatomical points on the view_1 two-dimensional image according to view_1. The endoscope operator can sequentially mark the three first type of marker points by, for example, clicking the three first type of marker points in sequence with a mouse, and the computer device can determine the marker order among the three first type of marker points based on the sequential marking operation of the endoscope operator.

[0129] According to the marker order from first to last, the three first type of marker points are marker Ⅰ_1 , marker Ⅰ_2 and marker Ⅰ_3 , wherein a character or color representing the marker order is displayed at each first type of marker point, for example, the number "1" is displayed at marker Ⅰ_1 , the number "2" is displayed at marker Ⅰ_2 , and the number "3" is displayed at marker Ⅰ_3 , and for another example, marker Ⅰ_1 is displayed in blue, marker Ⅰ_2 is displayed in yellow, and marker Ⅰ_3 is displayed in green, wherein blue corresponds to the first marker order, yellow corresponds to the second marker order, and green corresponds to the third marker order.

[0130] If the marker Ⅰ_1 、marker Ⅰ_2 and marker Ⅰ_3 The numbers "1", "2" and "3" are displayed at the ends respectively. The endoscope operator can mark the dots on the visual field under the endoscope in the order represented by the numbers. Specifically, the endoscope operator can first mark the dots according to the markers. Ⅰ_1 At the position on the view_1 two-dimensional map, a dot is placed at the corresponding position on the field of view under the microscope, and the computer device obtains the second type of marker Ⅱ_1 , the endoscope operator can first follow the marker Ⅰ_2 At the position on the view_1 two-dimensional map, a dot is placed at the corresponding position on the field of view under the microscope, and the computer device obtains the second type of marker Ⅱ_2 , the endoscope operator can first follow the marker Ⅰ_3 At the position on the view_1 two-dimensional map, a dot is placed at the corresponding position on the field of view under the microscope, and the computer device obtains the second type of marker Ⅱ_3 .

[0131] The third method: The first type of marking points are determined by automatic marking and manual marking, and the second type of marking points are determined by manual marking:

[0132] The computer device can display the view_1 two-dimensional image. By pressing view_1, the corresponding positions of the first-class markers of the three anatomical points on the view_1 two-dimensional image can be displayed. The endoscopist adjusts the first-class markers determined by the automatic marking method, and the computer device can display the adjusted first-class markers on the view_1 two-dimensional image. The endoscopist can sequentially calibrate the three first-class markers by, for example, clicking on the three first-class markers in sequence with a mouse. The computer device can determine the marking order of the three first-class markers based on the endoscopist's sequential calibration operations.

[0133] According to the marking order from first to last, these three first-class marking points are recorded as markers Ⅰ_1 、marker Ⅰ_2 and marker Ⅰ_3 , where characters or colors representing the marking order are displayed at each type of marking point, for example, Ⅰ_1 The number "1" is displayed at the marker. Ⅰ_2 The number "2" is displayed at the marker Ⅰ_3 The number "3" is displayed at the same place, and for example, the marker is displayed in blue. Ⅰ_1 , display the marker in yellow Ⅰ_2 , display the marker in greenⅠ_3 wherein the blue color corresponds to the first marker order, the yellow color corresponds to the second marker order, and the green color corresponds to the third marker order.

[0134] If the numbers "1", "2" and "3" are displayed at marker Ⅰ_1 , marker Ⅰ_2 and marker Ⅰ_3 respectively, the endoscope operator can mark the in-vivo view according to the marker order represented by the numbers; specifically, the endoscope operator can first mark the in-vivo view at the position corresponding to the position of marker Ⅰ_1 on the view_1 two-dimensional image, and the computer device obtains the second type of marker point marker Ⅱ_1 , the endoscope operator can first mark the in-vivo view at the position corresponding to the position of marker Ⅰ_2 on the view_1 two-dimensional image, and the computer device obtains the second type of marker point marker Ⅱ_2 , the endoscope operator can first mark the in-vivo view at the position corresponding to the position of marker Ⅰ_3 on the view_1 two-dimensional image, and the computer device obtains the second type of marker point marker Ⅱ_3 .

[0135] To avoid fusion errors, the selected three anatomical points are not on the same straight line, and thus the three first type of marker points are not on the same straight line and the three second type of marker points are not on the same straight line.

[0136] Among the three first type of marker points, the first type of marker point corresponding to the first marker order is marker Ⅰ_1 , and among the three second type of marker points, the second type of marker point corresponding to the first marker order is marker Ⅱ_1 , thus marker Ⅰ_1 and marker Ⅱ_1 can be matched, marker Ⅰ_1 and marker Ⅱ_1 are regarded as a marker point pair, and marker Ⅰ_1 and marker Ⅱ_1 correspond to the same anatomical point.

[0137] Among the three first type of marker points, the first type of marker point corresponding to the second marker order is marker Ⅰ_2 , and among the three second type of marker points, the second type of marker point corresponding to the second marker order is marker Ⅱ_2 , thus marker Ⅰ_2 and marker Ⅱ_2 can be matched, marker Ⅰ_2and marker Ⅱ_2 and marker Ⅰ_2 and marker Ⅱ_2 correspond to the same anatomical point.

[0138] In the three first-type marker points, the first-type marker point corresponding to the third marker order is marker Ⅰ_3 In the three second-type marker points, the second-type marker point corresponding to the third marker order is marker Ⅱ_3 Therefore, marker Ⅰ_3 and marker Ⅱ_3 can be matched, and marker Ⅰ_3 and marker Ⅱ_3 correspond to the same anatomical point. Ⅰ_3 and marker Ⅱ_3 correspond to the same anatomical point.

[0139] The third part:

[0140] After obtaining the plurality of marker point pairs, the computer device can adjust the perspective of the virtual camera multiple times; each time the perspective is adjusted, a two-dimensional image of the three-dimensional model under the perspective can be acquired by the virtual camera, when the position of the first-type marker point in the two-dimensional image does not coincide with the position of the second-type marker point in the field-of-view image under the endoscope in at least one marker point pair, the next perspective adjustment can be performed, when the position of the first-type marker point in the two-dimensional image coincides with the position of the second-type marker point in the field-of-view image under the endoscope in each marker point pair, the adjustment can be stopped, and the two-dimensional image of the three-dimensional model under the perspective and the field-of-view image under the endoscope can be fused to display a fused image. In addition, the computer device can also display the preoperative planning path on the fused image to guide the endoscopic operator to perform subsequent operations.

[0141] The above embodiments are based on the marker points of the anatomical points of the target object in the three-dimensional model and the field-of-view image under the endoscope, expand the field of view of the endoscope, and improve the efficiency of minimally invasive surgery; and the first-type marker point and the second-type marker point corresponding to the same anatomical point are regarded as a pair, and the marker point pair obtained based thereon can more accurately fuse and display the field-of-view image under the endoscope and the three-dimensional model, and avoid misalignment.

[0142] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiments of the present application also provide an endoscope field of view expansion device for implementing the above-mentioned endoscope field of view expansion method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more endoscope field of view expansion device embodiments provided below can refer to the limitations of the endoscope field of view expansion method described above, which will not be repeated here.

[0144] In one embodiment, as shown in Figure 4 An endoscope field of view expansion device is provided, comprising:

[0145] A three-dimensional model acquisition module 401 is configured to construct a three-dimensional model of a target object according to preoperative images;

[0146] A marker point acquisition module 402 is configured to acquire a first type of marker point marked by an anatomical point of the target object on the three-dimensional model;

[0147] The marker point acquisition module 402 is further configured to acquire a second type of marker point marked by the anatomical point on an endoscopic field of view picture;

[0148] A fusion display module 403 is configured to fuse and display the endoscopic field of view picture and the three-dimensional model according to the first type of marker point of the anatomical point on the three-dimensional model and the second type of marker point on the endoscopic field of view picture, so as to expand the endoscopic field of view.

[0149] In one embodiment, the marker point acquisition module 402 is further configured to: acquire an endoscopic field of view picture of the endoscope and a target two-dimensional picture corresponding to the three-dimensional model at a target viewing angle; adjust the acquisition angle of the endoscope according to the target two-dimensional picture until the content of the endoscopic field of view picture matches the content of the target two-dimensional picture; and when the content of the endoscopic field of view picture matches the content of the target two-dimensional picture, acquire a dot marking on the endoscopic field of view picture to obtain the second type of marker point marked by the anatomical point on the endoscopic field of view picture.

[0150] In an embodiment, the marker point acquisition module 402 is further configured to: acquire a plurality of candidate view angles of the endoscope viewing target object; and obtain a target two-dimensional image corresponding to the three-dimensional model at a target view angle according to two-dimensional images corresponding to the three-dimensional model at the plurality of candidate view angles.

[0151] In an embodiment, the fusion display module 403 is further configured to: determine a marker point pair corresponding to the same anatomical point from a plurality of first marker points of the three-dimensional model and a plurality of second marker points of the endoscopic view image; and fuse and display the endoscopic view image and the three-dimensional model based on the marker point pair.

[0152] In an embodiment, the first point set includes a plurality of first marker points, and the second point set includes a plurality of second marker points. The fusion display module 403 is further configured to: acquire a marker order of each of the first marker points in the first point set and a marker order of each of the second marker points in the second point set; and match each of the first marker points and each of the second marker points according to the marker orders to obtain a marker point pair.

[0153] In an embodiment, the fusion display module 403 is further configured to: determine a first image in which a marker point is marked first and a second image in which a marker point is marked second from the endoscopic view image and a target two-dimensional image corresponding to the three-dimensional model at a target view angle; acquire a first marker point of an anatomical point in the first image in which the marker point is marked first; display a character or color representing a marker order at each first marker point according to a marker order of the first marker point in a corresponding point set; the character or color is used to instruct an endoscope operator to mark a marker point on the second image in which the marker point is marked second according to the marker order represented by the character or color; and acquire a second marker point of the anatomical point in the second image in which the marker point is marked second and a marker order of the second marker point in the corresponding point set according to a marking operation of the endoscope operator on the second image in which the marker point is marked second. The marker order of each of the first marker points in the first point set and the marker order of each of the second marker points in the second point set are obtained according to the marker order of the first marker point in the corresponding point set and the marker order of the second marker point in the corresponding point set.

[0154] In an embodiment, the fusion display module 403 is further configured to: when it is determined that one of the first type of marker points and one of the second type of marker points match according to the marker order, determine the position of the one of the first type of marker points relative to other points in the first point set and the position of the one of the second type of marker points relative to other points in the second point set; when the position of the one of the first type of marker points relative to other points in the first point set and the position of the one of the second type of marker points relative to other points in the second point set do not match, send a marker point mismatch warning message; and in the case that the endoscope operator receives the marker point mismatch warning message and confirms that the one of the first type of marker points and the one of the second type of marker points do not match to the same marker point pair, in response to a matching point selection operation of the endoscope operator, obtain a marker point pair based on at least one of the first type of marker point and the second type of marker point selected by the endoscope operator.

[0155] The above-mentioned modules in the endoscope field of view expansion device can be implemented wholly or partially by software, hardware, and combinations thereof. The above-mentioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the above-mentioned modules.

[0156] In an exemplary embodiment, a computer device is provided, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data related to the above-mentioned method. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement an endoscope field of view expansion method.

[0157] Those skilled in the art can understand that Figure 5 The structure shown in the above-mentioned figures is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

[0158] In an embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps in each of the above method embodiments when executing the computer program.

[0159] In an embodiment, a computer readable storage medium is provided, storing a computer program, the computer program implementing the steps in each of the above method embodiments when executed by a processor.

[0160] In an embodiment, a computer program product is provided, storing a computer program, the computer program implementing the steps in each of the above method embodiments when executed by a processor.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0162] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0163] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0164] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An endoscopic field of view extension method, characterized by, The method comprises: constructing a three-dimensional model of a target object according to a preoperative image; acquiring a first type of marking point of an anatomical point marked on the three-dimensional model of the target object; acquiring a second type of marking point of the anatomical point marked on an endoscopic under-mirror field of view image; fusing and displaying the under-mirror field of view image and the three-dimensional model according to the first type of marking point of the anatomical point on the three-dimensional model and the second type of marking point of the anatomical point on the under-mirror field of view image, so as to expand the endoscopic field of view.

2. The method of claim 1, wherein, The acquiring of the second type of marking point of the anatomical point marked on the under-mirror field of view image comprises: acquiring an under-mirror field of view image of the endoscope and a target two-dimensional image corresponding to the three-dimensional model under a target view angle; adjusting the acquisition angle of the endoscope according to the target two-dimensional image until the content of the under-mirror field of view image matches the content of the target two-dimensional image; when the content of the under-mirror field of view image matches the content of the target two-dimensional image, acquiring the dot marking on the under-mirror field of view image to obtain the second type of marking point of the anatomical point marked on the under-mirror field of view image.

3. The method of claim 2, wherein, The acquiring of the target two-dimensional image corresponding to the three-dimensional model under a target view angle comprises: acquiring a plurality of candidate view angles of the endoscope viewing the target object; acquiring a target two-dimensional image corresponding to the three-dimensional model under a target view angle according to the two-dimensional images corresponding to the three-dimensional model under a plurality of candidate view angles.

4. The method of claim 1, wherein, The fusing and displaying of the under-mirror field of view image and the three-dimensional model according to the first type of marking point of the anatomical point on the three-dimensional model and the second type of marking point of the anatomical point on the under-mirror field of view image comprises: determining the first type of marking point and the second type of marking point corresponding to the same anatomical point from a plurality of first type of marking points on the three-dimensional model and a plurality of second type of marking points on the under-mirror field of view image to obtain a marking point pair; fusing and displaying the under-mirror field of view image and the three-dimensional model based on the marking point pair.

5. The method of claim 4, wherein, The first point set comprises a plurality of first type of marking points, and the second point set comprises a plurality of second type of marking points; the determining of the first type of marking point and the second type of marking point corresponding to the same anatomical point from a plurality of first type of marking points on the three-dimensional model and a plurality of second type of marking points on the under-mirror field of view image to obtain a marking point pair comprises: acquiring a marking order of each first type of marking point in the first point set and a marking order of each second type of marking point in the second point set; matching each first type of marking point and each second type of marking point according to the marking order to obtain a marking point pair.

6. The method of claim 5, wherein, The acquiring of the marking order of each first type of marking point in the first point set and the marking order of each second type of marking point in the second point set comprises: determining a dot marking first image and a dot marking second image from the under-mirror field of view image and the target two-dimensional image corresponding to the three-dimensional model under a target view angle; acquiring a first marking point of the anatomical point on the dot marking first image; displaying a character or color representing the marking order at each first marking point according to the marking order of the first marking point in the corresponding point set; the character or color is used to instruct an endoscopic operator to perform a dot marking operation on the dot marking second image according to the marking order represented by the character or color; According to the dotting operation of the endoscope operator on the dotting-after picture, the marking order of the several anatomical points in the after-marking points of the dotting-after picture and in the corresponding point set is obtained; According to the marking order of the before-marking points in the corresponding point set and the marking order of the after-marking points in the corresponding point set, the marking order of each of the first-type marking points in the first point set and the marking order of each of the second-type marking points in the second point set are obtained.

7. The method of claim 5, wherein, According to the marking order, each of the first-type marking points and each of the second-type marking points are matched to obtain a marking point pair, including: When it is determined according to the marking order that one of the first-type marking points and one of the second-type marking points are matched, the position of the one of the first-type marking points relative to other points in the first point set and the position of the one of the second-type marking points relative to other points in the second point set are determined; When the position of the one of the first-type marking points relative to other points in the first point set and the position of the one of the second-type marking points relative to other points in the second point set are not matched, a marking point mismatch warning message is sent out; In the case that the endoscope operator receives the marking point mismatch warning message and confirms that the one of the first-type marking points and the one of the second-type marking points are not matched to the same marking point pair, a marking point pair is obtained based on at least one of the first-type marking points and the second-type marking points selected by the endoscope operator in response to a matching point selection operation of the endoscope operator.

8. An endoscopic field of view expansion device, characterized by, The device comprises: a three-dimensional model acquisition module configured to construct a three-dimensional model of a target object according to preoperative images; a marking point acquisition module configured to acquire first-type marking points of anatomical points of the target object marked on the three-dimensional model; the marking point acquisition module is further configured to acquire second-type marking points of anatomical points marked on an endoscopic under-scope view; a fusion display module configured to fuse and display the under-scope view and the three-dimensional model according to the first-type marking points of the anatomical points on the three-dimensional model and the second-type marking points of the anatomical points on the under-scope view, so as to expand the endoscopic view. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

Images