Image processing method and endoscope equipment

Abstract

The invention relates to the technical field of medical treatment, in particular to an image processing method and endoscope equipment. According to the embodiment of the invention, the method comprises the steps: obtaining endoscope image data which is collected through an endoscope camera and comprises a target detection part in an operation process, and determining the first coordinate information of the target detection part in a first coordinate system corresponding to the endoscope camera; based on the current pose state information of the endoscope camera determined by the optical positioning camera and a transformation matrix between a second coordinate system corresponding to the optical positioning camera and a third coordinate system corresponding to the CT image data set, performing transformation processing on the first coordinate information corresponding to the target detection part, cT coordinate information corresponding to the target detection part is determined, and target CT image data are determined in a CT image data set which is collected by CT image collection equipment and contains the target detection part; and displaying the fused target image based on the data obtained by fusing the endoscope image data and the target CT image data.

Description

Technical Field

[0001] This application relates to the field of medical technology, and particularly to an image processing method and an endoscope device. Background Art

[0002] At present, the treatment method for primary liver cancer is still mainly surgical. Laparoscopic liver resection belongs to minimally invasive surgery, which has the advantages of small incision, less pain, fast recovery, and small scar compared with traditional open surgery, and has quickly become the current development trend of surgery. However, at the same time, due to the lack of depth perception information and tactile feedback information, it poses a greater challenge to surgeons. The augmented reality technology that has developed rapidly in recent years can fuse and display the model data reconstructed based on preoperative medical images with the endoscopic images collected during the operation, which is a way of applying intraoperative navigation technology, enabling doctors to intuitively see the structures below the organ surface and effectively solving the problem of lack of depth perception information to help doctors complete laparoscopic surgery more accurately.

[0003] However, as the endoscopic camera moves during the operation, the image of the organ part shown in the endoscopic image will change accordingly. If a doctor needs to obtain the depth perception information of the organ part in the current endoscopic image, he needs to manually adjust the organ part in the display screen of the model data reconstructed from preoperative medical images to be consistent with the organ part in the current endoscopic image display screen. Therefore, the existing augmented reality technology adds more operation steps for doctors, is more cumbersome to use, and reduces the efficiency of the doctor during the operation. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method and an endoscope device to provide an efficient image processing solution and provide real-time intraoperative navigation for doctors.

[0005] In a first aspect, this application provides an endoscope device, which includes an endoscopic camera, a display, and a processor, and the endoscope device is connected to an optical positioning camera;

[0006] The endoscopic camera is used to collect endoscopic image data including a target detection part during the operation and transmit the endoscopic image data to the processor;

[0007] The optical positioning camera is used to determine the current pose state information of the endoscopic camera and transmit the pose state information to the processor;

[0008] The processor is configured to determine first coordinate information of the target detection site in a first coordinate system corresponding to the endoscopic camera based on the endoscopic image data, and perform transformation processing on the first coordinate information corresponding to the target detection site based on the current pose state information of the endoscopic camera and a transformation matrix between a second coordinate system corresponding to the optical positioning camera and a third coordinate system corresponding to the CT image dataset, so as to determine CT coordinate information corresponding to the target detection site; obtain a CT image dataset including the target detection site collected by a CT image acquisition device, where the CT image dataset includes a plurality of CT image data; determine target CT image data matching the endoscopic image data in the CT image dataset based on the CT coordinate information, and transmit data obtained by fusing the endoscopic image data and the target CT image data to the display;

[0009] The display is configured to display a target image obtained by fusing an endoscopic image corresponding to the endoscopic image data and a CT image corresponding to the target CT image data based on data obtained by fusing the endoscopic image data and the target CT image data.

[0010] Since, in the embodiment of the present application, endoscopic image data including a target detection site is collected during a surgical procedure by an endoscopic camera, and based on the current pose state information of the endoscopic camera determined by an optical positioning camera and a transformation matrix between a second coordinate system corresponding to the optical positioning camera and a third coordinate system corresponding to the CT image dataset, transformation processing is performed on the first coordinate information of the target detection site in a first coordinate system corresponding to the endoscopic camera to determine CT coordinate information corresponding to the target detection site, it is possible to obtain target CT image data matching the CT coordinate information in a pre-collected CT image dataset, so that the organ part in the CT image corresponding to the displayed target CT image data is consistent with the organ part in the currently collected endoscopic image. Then, by displaying a target image obtained by fusing an endoscopic image corresponding to the endoscopic image data and a CT image corresponding to the target CT image data, when a doctor moves the endoscopic camera to change the organ part displayed in the endoscopic image, the present application can determine a CT image corresponding to the same organ part, and fuse and display the endoscopic image and the CT image, helping the doctor obtain depth perception information of the organ part and improving the efficiency of the doctor during the surgical procedure.

[0011] In a possible embodiment, the processor is specifically configured to:

[0012] Perform transformation processing on the first coordinate information based on the current pose state information of the endoscopic camera to determine second coordinate information of the target detection site in a second coordinate system corresponding to the optical positioning camera;

[0013] Based on the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset, perform transformation processing on the second coordinate information to determine the CT coordinate information corresponding to the target detection part.

[0014] In a possible embodiment, the endoscope device further includes a positioning marker, the positioning marker is arranged at a specific position of the endoscope device, and the relative position between the positioning marker and the endoscope camera remains unchanged during the use of the endoscope device; the processor is specifically configured to:

[0015] Based on the current pose state information of the endoscope camera and the pose transformation matrix of the endoscope camera, determine the transformation matrix between the first coordinate system and the second coordinate system; wherein, the current pose state information of the endoscope camera is determined by the optical positioning camera based on the coordinate information of the positioning marker collected currently in the second coordinate system corresponding to the optical positioning camera; the pose transformation matrix of the endoscope camera is used to characterize the relative position relationship between the endoscope camera and the positioning marker;

[0016] Based on the transformation matrix between the first coordinate system and the second coordinate system, perform transformation processing on the first coordinate information of the target detection part to determine the second coordinate information of the target detection part.

[0017] In a possible embodiment, the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset is determined by the following method:

[0018] Obtain the coordinate information of multiple marked points on the reference part model collected by the optical positioning camera in the second coordinate system corresponding to the optical positioning camera;

[0019] Based on the coordinate information of the multiple marked points corresponding in the second coordinate system and the reference CT coordinate information of the multiple marked points collected by the CT image acquisition device, determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0020] In a possible embodiment, the processor is further configured to:

[0021] Collect reference endoscope image data including multiple marked points on the reference part model through the endoscope camera; and obtain the reference CT coordinate information including the multiple marked points collected by the CT image acquisition device;

[0022] Based on the reference endoscopic image data, determine the test CT coordinate information corresponding to the multiple marker points through the pose transformation matrix of the endoscopic camera and the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset;

[0023] Determine that the error between the test CT coordinate information corresponding to the multiple marker points and the reference CT coordinate information satisfies a preset error condition.

[0024] In a second aspect, the present application provides an image processing method applied to an endoscopic device, and the method includes:

[0025] Obtain endoscopic image data including a target detection part collected by an endoscopic camera during a surgical procedure, and determine first coordinate information of the target detection part in a first coordinate system corresponding to the endoscopic camera;

[0026] Obtain the current pose state information of the endoscopic camera determined by an optical positioning camera, and based on the current pose state information of the endoscopic camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image dataset, perform transformation processing on the first coordinate information corresponding to the target detection part to determine the CT coordinate information corresponding to the target detection part;

[0027] Obtain a CT image dataset including a target detection part collected by a CT image acquisition device, where the CT image dataset includes multiple CT image data; based on the CT coordinate information corresponding to the target detection part, determine target CT image data matching the endoscopic image data in the CT image dataset;

[0028] Based on the data after fusion processing of the endoscopic image data and the target CT image data, display a target image obtained by fusing the endoscopic image corresponding to the endoscopic image data and the CT image corresponding to the target CT image data.

[0029] In a possible embodiment, the performing transformation processing on the first coordinate information corresponding to the target detection part based on the current pose state information of the endoscopic camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image dataset to determine the CT coordinate information corresponding to the target detection part includes:

[0030] Perform transformation processing on the first coordinate information based on the current pose state information of the endoscopic camera to determine second coordinate information of the target detection part in the second coordinate system corresponding to the optical positioning camera;

[0031] Based on the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset, perform transformation processing on the second coordinate information to determine the CT coordinate information corresponding to the target detection site.

[0032] In a possible embodiment, the endoscopic device further includes a positioning marker, which is arranged at a specific position of the endoscopic device, and the relative position between the positioning marker and the endoscopic camera remains unchanged during the use of the endoscopic device;

[0033] The transformation processing of the first coordinate information based on the current pose state information of the endoscopic camera to determine the second coordinate information of the target detection site in the second coordinate system corresponding to the optical positioning camera includes:

[0034] Based on the current pose state information of the endoscopic camera and the pose transformation matrix of the endoscopic camera, determine the transformation matrix between the first coordinate system and the second coordinate system; wherein, the current pose state information of the endoscopic camera is determined by the optical positioning camera based on the coordinate information of the positioning marker collected currently in the second coordinate system corresponding to the optical positioning camera; the pose transformation matrix of the endoscopic camera is used to represent the relative position relationship between the endoscopic camera and the positioning marker;

[0035] Based on the transformation matrix between the first coordinate system and the second coordinate system, perform transformation processing on the first coordinate information of the target detection site to determine the second coordinate information of the target detection site.

[0036] In a possible embodiment, the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset is determined by the following method:

[0037] Obtain the coordinate information of multiple marker points on the reference part model collected by the optical positioning camera in the second coordinate system corresponding to the optical positioning camera;

[0038] Based on the coordinate information of the multiple marker points corresponding in the second coordinate system and the reference CT coordinate information of the multiple marker points collected by the CT image acquisition device, determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0039] In a possible embodiment, before obtaining the endoscopic image data containing the target detection site collected by the endoscopic camera during the operation, it further includes:

[0040] Collect reference endoscopic image data including a plurality of marked points on the reference part model through the endoscopic camera; and obtain reference CT coordinate information corresponding to the plurality of marked points collected by a CT image acquisition device.

[0041] Based on the reference endoscopic image data, determine the test CT coordinate information corresponding to the plurality of marked points through the pose transformation matrix of the endoscopic camera and the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image data set.

[0042] Determine that the error between the test CT coordinate information corresponding to the plurality of marked points and the reference CT coordinate information meets a preset error condition.

[0043] In a third aspect, the present application provides an image processing device, including:

[0044] An acquisition module, configured to acquire endoscopic image data including a target detection part collected by an endoscopic camera during a surgical procedure, and determine first coordinate information of the target detection part in a first coordinate system corresponding to the endoscopic camera.

[0045] A transformation module, configured to acquire current pose state information of the endoscopic camera determined by an optical positioning camera, and based on the current pose state information of the endoscopic camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image data set, perform transformation processing on the first coordinate information corresponding to the target detection part to determine CT coordinate information corresponding to the target detection part.

[0046] A determination module, configured to acquire a CT image data set including a plurality of CT image data and including the target detection part collected by a CT image acquisition device; based on the CT coordinate information corresponding to the target detection part, determine target CT image data matching the endoscopic image data in the CT image data set.

[0047] A display module, configured to display a target image obtained by fusing an endoscopic image corresponding to the endoscopic image data and a CT image corresponding to the target CT image data based on data obtained by fusing the endoscopic image data and the target CT image data.

[0048] In a possible embodiment, the transformation module is specifically configured to:

[0049] Perform transformation processing on the first coordinate information based on the current pose state information of the endoscopic camera to determine second coordinate information of the target detection part in a second coordinate system corresponding to the optical positioning camera.

[0050] Based on the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset, perform transformation processing on the second coordinate information to determine the CT coordinate information corresponding to the target detection part.

[0051] In a possible embodiment, the endoscopic device further includes a positioning marker, the positioning marker is disposed at a specific position of the endoscopic device, and the relative position between the positioning marker and the endoscopic camera remains unchanged during the use of the endoscopic device;

[0052] The transformation module is specifically configured to:

[0053] Based on the current pose state information of the endoscopic camera and the pose transformation matrix of the endoscopic camera, determine the transformation matrix between the first coordinate system and the second coordinate system; wherein, the current pose state information of the endoscopic camera is determined by the optical positioning camera based on the coordinate information of the positioning marker collected currently in the second coordinate system corresponding to the optical positioning camera; the pose transformation matrix of the endoscopic camera is used to represent the relative position relationship between the endoscopic camera and the positioning marker;

[0054] Based on the transformation matrix between the first coordinate system and the second coordinate system, perform transformation processing on the first coordinate information of the target detection part to determine the second coordinate information of the target detection part.

[0055] In a possible embodiment, the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset is determined by the following method:

[0056] Obtain the coordinate information of multiple marker points on the reference part model collected by the optical positioning camera in the second coordinate system corresponding to the optical positioning camera;

[0057] Based on the coordinate information of the multiple marker points corresponding in the second coordinate system and the reference CT coordinate information of the multiple marker points collected by the CT image acquisition device, determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0058] In a possible embodiment, the acquisition module is further configured to:

[0059] Collect reference endoscopic image data including multiple marker points on the reference part model through the endoscopic camera; and obtain the reference CT coordinate information including the multiple marker points collected by the CT image acquisition device;

[0060] Based on the reference endoscopic image data, determine the test CT coordinate information corresponding to the multiple marker points through the pose transformation matrix of the endoscopic camera and the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset;

[0061] Determine that the error between the test CT coordinate information corresponding to the multiple marker points and the reference CT coordinate information meets a preset error condition.

[0062] In a fourth aspect, the present application provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute the image processing method as described in the second aspect above.

[0063] In a fifth aspect, the present application provides a computer program product, including a computer program:

[0064] When the computer program is executed by a processor, it implements the image processing method as described in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. Obviously, the following introduced drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0066] Figure 1 It is an application scenario diagram of an optional image processing method for an embodiment of the present application;

[0067] Figure 2 It is an application scenario diagram of an optional image processing method for an embodiment of the present application;

[0068] Figure 3 It is a schematic diagram of an intraoperative application scenario for an embodiment of the present application;

[0069] Figure 4 It is a flowchart of an image processing method for an embodiment of the present application;

[0070] Figure 5 It is a flowchart of determining the current pose state information of an endoscopic camera for an embodiment of the present application;

[0071] Figure 6 It is a schematic diagram of a rigid body sphere for an embodiment of the present application;

[0072] Figure 7 It is an implementation flowchart of determining the CT coordinate information corresponding to a target detection part for an embodiment of the present application;

[0073] Figure 8 It is a flowchart of an embodiment for determining the second coordinate information in the embodiments of the present application;

[0074] Figure 9 It is a flowchart of an embodiment for determining the pose transformation matrix of an endoscopic camera in the embodiments of the present application;

[0075] Figure 10 It is an application scenario diagram of an embodiment for determining the coordinate information of an endoscopic camera in a second coordinate system in the embodiments of the present application;

[0076] Figure 11 It is a schematic diagram of an embodiment for determining the relative pose between an endoscopic camera and an optical positioning camera in the embodiments of the present application;

[0077] Figure 12 It is a flowchart of an embodiment for determining the transformation matrix between a second coordinate system and a third coordinate system corresponding to a CT image dataset in the embodiments of the present application; Figure 13 It is a schematic diagram of a reference part model in the embodiments of the present application;

[0078] Figure 14 It is a schematic diagram of a reference CT image corresponding to reference CT image data in the embodiments of the present application;

[0079] Figure 15 It is a flowchart of an embodiment for checking errors in the embodiments of the present application;

[0080] Figure 16 It is a schematic diagram of test CT coordinate information corresponding to multiple marker points in the embodiments of the present application;

[0081] Figure 17 It is a schematic diagram of test CT coordinate information corresponding to multiple marker points that meet the error conditions in the embodiments of the present application;

[0082] Figure 18 It is a schematic diagram of an overall view in the embodiments of the present application. Detailed implementation manners

[0083] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Among them, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0084] Moreover, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; "and / or" in the text is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. Additionally, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0085] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and should not be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0086] The application scenarios of the image processing method provided by the embodiments of the present application are introduced below with reference to the accompanying drawings.

[0087] The embodiments of the present application provide an image processing method, which can be applied to an endoscope device. The following combines Figure 1 to introduce the structure of the endoscope device.

[0088] As Figure 1 shown, a diagram of an application scenario of an embodiment of the present application, which includes an endoscope device 10 and an optical positioning camera 20 connected to the endoscope device 10. The image processing method of the present application is implemented through the endoscope device 10 and the optical positioning camera 20.

[0089] Among them, the endoscope device 10 includes a cold light source and an imaging component.

[0090] The cold light source is a device that provides illumination light for the endoscope during endoscopy and surgery. Its key components usually include: a cold light source host, a light guide beam, and a cold light source light output interface. The imaging component is a device that captures the images of endoscopy and surgery to collect the image data of the examination and surgery. Its key components include a camera, buttons, an imaging host, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a video cable, an optical adapter, a lens rod, a communication port, and a video output interface. The imaging host can be connected to a display through the video output interface and connected to the cold light source host through the communication port.

[0091] The objective field of view, lens rod, optical adapter, buttons, CMOS, and camera in the key components of the imaging component are collectively referred to as the endoscope camera.

[0092] Among them, the present application introduces the image processing method of the embodiments of the present application based on the endoscope camera 101, the processor 102, the display 103, and the optical positioning camera 20 included in the endoscope device 10.

[0093] It should be noted that the processor 102 in the embodiments of the present application may be located in the imaging host.

[0094] The endoscope camera 101 is configured to collect endoscope image data including the target detection site during the operation and transmit the endoscope image data to the processor 102;

[0095] The optical positioning camera 20 is configured to determine the current pose state information of the endoscope camera 101 and transmit the pose state information to the processor 102;

[0096] The processor 102 is configured to determine the first coordinate information of the target detection site in the first coordinate system corresponding to the endoscope camera 101 based on the endoscope image data, and perform transformation processing on the first coordinate information corresponding to the target detection site based on the current pose state information of the endoscope camera 101 and the transformation matrix between the second coordinate system corresponding to the optical positioning camera 20 and the third coordinate system corresponding to the CT image data set to determine the CT coordinate information corresponding to the target detection site; obtain the CT image data set including the target detection site collected by the CT image acquisition device, where the CT image data set includes a plurality of CT image data; determine the target CT image data matching the endoscope image data in the CT image data set based on the CT coordinate information, and transmit the data obtained by fusing the endoscope image data and the target CT image data to the display 103;

[0097] The display 103 is configured to display the target image obtained by fusing the endoscope image corresponding to the endoscope image data and the CT image corresponding to the target CT image data based on the data obtained by fusing the endoscope image data and the target CT image data.

[0098] It should be noted that the optical positioning camera 20 in the embodiments of the present application may be connected to the endoscope device 10 in a wired manner.

[0099] In another alternative embodiment, the image processing method provided by the present application may also be applied to an endoscope device as Figure 2 shown. As Figure 2 the processor 102 in is located in another host other than the imaging host, and the present application places no restrictions on this host. For the specific implementation process principle, refer to the introduction of Figure 1 and the present application will not elaborate here.

[0100] Exemplarily, as Figure 3As shown in the figure, it is a schematic diagram of an intraoperative application scenario in an embodiment of the present application. Among them, the doctor 30 holds the handheld part of the endoscope device, collects the endoscope image data of the detected organ in the patient 31's body, the optical positioning camera 20 collects the pose state information of the endoscope device, and the display 103 shows the endoscope image corresponding to the endoscope image data and the CT image determined based on the endoscope image data to the doctor 30.

[0101] Of course, the method provided in the embodiments of the present application is not limited to Figure 1 , Figure 2 and Figure 3 the application scenarios shown, and can also be used in other possible application scenarios, which are not limited in the embodiments of the present application.

[0102] For example, Figure 4 as shown in the figure, it is a flowchart of an image processing method in an embodiment of the present application, which can be applied to an endoscope device, and specifically may include the following steps:

[0103] Step S401: The endoscope device obtains the endoscope image data including the target detection part collected by the endoscope camera during the operation, and determines the first coordinate information of the target detection part in the first coordinate system corresponding to the endoscope camera.

[0104] Optionally, in the embodiment of the present application, based on the endoscope coordinate information of the target detection part in the endoscope image coordinate system corresponding to the endoscope image data, the internal parameters of the endoscope camera are used to perform transformation processing on the endoscope coordinate information corresponding to the target detection part, and the first coordinate information of the target detection part in the first coordinate system corresponding to the endoscope camera is determined.

[0105] Among them, the first coordinate system corresponding to the endoscope camera is the endoscope camera coordinate system corresponding to the endoscope camera.

[0106] It should be noted that before applying step S401, the internal parameters of the endoscope camera need to be determined in advance in the present application.

[0107] Exemplarily, in Formula 1, K is the internal parameter of the endoscope camera:

[0108]

[0109] Among them, represents the endoscope coordinate information of the target detection part in the endoscope image coordinate system corresponding to the endoscope image data, represents the first coordinate information of the target detection part in the first coordinate system corresponding to the endoscope camera.

[0110] Based on Formula 1, it can be known that the internal parameter of the endoscope camera is used to obtain the coordinate information of the target detection part in the endoscope image data.

[0111] It should be noted that the process of obtaining the internal parameters of the endoscopic camera is a conventional technical means in the art and will not be elaborated here.

[0112] In implementation, based on the internal parameters of the endoscopic camera, the endoscopic coordinate information in the endoscopic image coordinate system corresponding to the target detection site is transformed into the first coordinate information in the first coordinate system corresponding to the endoscopic camera.

[0113] In the embodiment of the present application, an inverse matrix operation is performed on the internal parameters of the endoscopic camera to obtain a coordinate transformation matrix for transforming the endoscopic coordinate information into the first coordinate information in the first coordinate system corresponding to the endoscopic camera.

[0114] Exemplarily, as shown in Formula 2:

[0115] R 12 = invers(K) —— Formula 2

[0116] Wherein, R 12 represents the coordinate transformation matrix for transforming the endoscopic coordinate information into the first coordinate information in the first coordinate system corresponding to the endoscopic camera; K represents the internal parameters of the endoscopic camera; invers represents the inverse matrix operation.

[0117] Based on the R determined above 12 , after obtaining the endoscopic image data including the target detection site during the operation, use R 12 to perform a transformation process on the endoscopic coordinate information of the target detection site in the endoscopic image coordinate system corresponding to the endoscopic image data, and determine the first coordinate information of the target detection site in the first coordinate system corresponding to the endoscopic camera.

[0118] Step S402: The endoscopic device acquires the current pose state information of the endoscopic camera determined by the optical positioning camera, and based on the current pose state information of the endoscopic camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image dataset, performs a transformation process on the first coordinate information corresponding to the target detection site to determine the CT coordinate information corresponding to the target detection site;

[0119] Optionally, in the embodiment of the present application, the endoscopic device further includes a positioning marker, then the embodiment of the present application provides a flowchart for determining the current pose state information of the endoscopic camera as shown in Figure 5 as follows:

[0120] Step S501: Through the optical positioning camera, collect the coordinate information of the positioning marker on the endoscopic device in the second coordinate system corresponding to the optical positioning camera;

[0121] Among them, the positioning marker is set at a specific position of the endoscopic device, and the relative position between the positioning marker and the endoscopic camera remains unchanged during the use of the endoscopic device;

[0122] Optionally, in the embodiments of the present application, a rigid body sphere can be set on the endoscopic device as the positioning marker.

[0123] As Figure 6 shown, it is a schematic diagram of a rigid body sphere in the embodiments of the present application. Among them, before the operation, the rigid body sphere is set on the mirror rod of the endoscopic device. During the operation, the part of the mirror rod in the endoscopic camera is inserted into the human body, and the endoscopic image data of the target detection part is collected by the endoscopic camera, and the rigid body sphere is located outside the human body.

[0124] It should be noted that the specific position where the rigid body sphere is set on the endoscopic device is the mirror rod of the endoscopic device.

[0125] In practice, since the rigid body sphere is located outside the human body, the coordinate information of the rigid body sphere in the second coordinate system corresponding to the optical positioning camera is collected by the optical positioning camera.

[0126] Step S502: The optical positioning camera determines the pose state information of the endoscopic device according to the coordinate information of the positioning marker in the second coordinate system.

[0127] Optionally, in the embodiments of the present application, the coordinate information of the rigid body sphere in the second coordinate system is used as the pose state information of the endoscopic device.

[0128] It should be noted that since most of the endoscopic camera is located inside the human body during the operation, the pose state information of the endoscopic camera cannot be directly determined. However, in the present application, a rigid body sphere is set on the mirror rod of the endoscopic device, and the coordinate information of the rigid body sphere in the second coordinate system corresponding to the optical positioning camera is collected by the optical positioning camera. Combining the fixed relative position between the endoscopic camera and the rigid body sphere, the pose state information of the endoscopic device can be characterized by the coordinate information of the positioning marker in the second coordinate system.

[0129] Based on the above determined pose state information of the endoscopic camera, the present application performs transformation processing on the first coordinate information corresponding to the target detection part based on the current pose state information of the endoscopic camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image dataset, and determines the CT coordinate information corresponding to the target detection part.

[0130] Specifically, as Figure 7 shown, it is a flowchart of an embodiment for determining the CT coordinate information corresponding to the target detection part in the embodiments of the present application. The specific steps are as follows:

[0131] Step S701: The endoscope device processes and transforms the first coordinate information based on the current pose state information of the endoscope camera to determine the second coordinate information of the target detection part in the second coordinate system corresponding to the optical positioning camera.

[0132] It should be noted that in the embodiment of the present application, the second coordinate system corresponding to the optical positioning camera is the optical positioning camera coordinate system.

[0133] As Figure 8 shown, the flowchart of an embodiment for determining the second coordinate information in the embodiment of the present application is as follows:

[0134] Step S801: The endoscope device determines the transformation matrix between the first coordinate system and the second coordinate system based on the current pose state information of the endoscope camera and the pose transformation matrix of the endoscope camera.

[0135] Among them, the current pose state information of the endoscope camera is determined by the optical positioning camera based on the coordinate information of the positioning marker collected currently in the second coordinate system corresponding to the optical positioning camera; the pose transformation matrix of the endoscope camera is used to represent the relative position relationship between the endoscope camera and the positioning marker.

[0136] It should be noted that the pose transformation matrix of the endoscope camera is determined in advance before executing the image processing method of the present application.

[0137] As Figure 9 shown, the flowchart of an embodiment for determining the pose transformation matrix of the endoscope camera in the embodiment of the present application is as follows:

[0138] Step S901: The optical positioning camera collects the coordinate information of the positioning marker in the second coordinate system corresponding to the optical positioning camera.

[0139] Step S902: Based on the coordinate information of the positioning marker in the second coordinate system corresponding to the optical positioning camera, determine the coordinate information of the end of the endoscope camera in the second coordinate system.

[0140] Optionally, an application scenario diagram for determining the coordinate information of the endoscope camera in the second coordinate system as Figure 10 shown in the embodiment of the present application. Among them, the present application is based on an endoscope camera distance measurement system that combines an optical positioning camera, a reference component, and an endoscope camera. Using visual positioning technology, the relative pose between the end of the endoscope camera and the optical positioning camera can be automatically determined. Exemplarily, the reference component can be an isosceles triangular prism. The A surface of the isosceles triangular prism has a calibration pattern, and a plurality of reference markers 901 are fixed on the B surface of the isosceles triangular prism. The A surface and the B surface are symmetric with respect to the isosceles triangular prism.

[0141] When the lens of the endoscope (the end of the endoscope camera), the positioning marker, and the two symmetric points located on the A surface and the B surface are collinear, the distance from the end of the endoscope camera to the symmetric point A on the A surface is obtained through the endoscope camera, and the external parameters of the endoscope camera are determined.

[0142] In the process of image measurement and machine vision applications, in order to determine the mutual relationship between the three-dimensional geometric position of a certain point on the surface of a spatial object and its corresponding point in the image, it is necessary to establish a geometric model of camera imaging, and these geometric model parameters are camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations, and this process of solving the parameters is called camera calibration.

[0143] Such as Figure 11 shown, a schematic diagram of the relative pose between an endoscope camera and an optical positioning camera in an embodiment of the present application. When the linear relationship as shown in Figure 11 is satisfied, the endoscope captures a checkerboard pattern, obtains the information in the photo and uses it to correct the camera parameters of the endoscope. Through the calibrated endoscope, the external parameters of the endoscope camera can be obtained, that is, the external parameters characterize the position and orientation of the endoscope camera in three-dimensional space.

[0144] In the embodiment of the present application, the optical positioning camera uses a visual positioning method to identify the three-dimensional coordinates of the positioning marker and the three-dimensional coordinates of the reference markers located at the four corners of the reference component, and then determines the three-dimensional coordinates of the symmetric point B.

[0145] Optionally, the embodiment of the present application determines the coordinate information of the end of the endoscope camera in the second coordinate system based on the three-dimensional coordinates of the symmetric point B and the external parameters of the endoscope camera.

[0146] In an alternative embodiment, the optical positioning camera can send the three-dimensional coordinates of the symmetric point B to the processor in the endoscope device, and the processor in the endoscope device determines the coordinate information of the end of the endoscope camera in the second coordinate system based on the three-dimensional coordinates of the symmetric point B and the external parameters of the endoscope camera.

[0147] In another alternative embodiment, the optical positioning camera determines the coordinate information of the end of the endoscope camera in the second coordinate system based on the three-dimensional coordinates of the symmetric point B and the external parameters of the endoscope camera.

[0148] Step S903: The endoscope device determines the pose transformation matrix of the endoscope camera based on the coordinate information of the positioning marker in the second coordinate system and the coordinate information of the end of the endoscope camera in the second coordinate system.

[0149] It should be noted that the pose transformation matrix of the endoscope camera characterizes the relative pose between the end of the endoscope camera and the rigid body ball, and the pose transformation matrix of the endoscope camera can be expressed as: R 刚体球到相机 .

[0150] In the embodiments of the present application, the relative pose between the end of the endoscopic camera and the rigid body sphere represents the relative distance and the relative orientation between the end of the endoscopic camera and the rigid body sphere.

[0151] In an alternative embodiment, if the coordinate information of the end of the endoscopic camera in the second coordinate system is determined by the optical positioning camera, the endoscopic device determines the pose transformation matrix of the endoscopic camera based on the coordinate information of the positioning marker in the second coordinate system sent by the optical positioning camera and the coordinate information of the end of the endoscopic camera in the second coordinate system.

[0152] In another alternative embodiment, if the coordinate information of the end of the endoscopic camera in the second coordinate system is determined by the endoscopic device, the endoscopic device determines the pose transformation matrix of the endoscopic camera based on the determined coordinate information of the end of the endoscopic camera in the second coordinate system and the coordinate information of the positioning marker in the second coordinate system sent by the optical positioning camera.

[0153] Wherein, the pose transformation matrix of the endoscopic camera is: the transformation matrix between the coordinate information of the positioning marker in the second coordinate system and the coordinate information of the endoscopic camera in the second coordinate system. Then, in the embodiments of the present application, by performing coordinate transformation processing on the coordinate information of the positioning marker in the second coordinate system and the coordinate information of the endoscopic camera in the second coordinate system, the pose transformation matrix of the endoscopic camera can be determined.

[0154] Exemplarily, the transformation matrix between the first coordinate system and the second coordinate system is determined by Equation 3:

[0155] R 23 = R 刚体球到相机 * R 光学定位相机下刚体球 ——Equation 3

[0156] Wherein, R 23 represents the transformation matrix between the first coordinate system and the second coordinate system, R 刚体球到相机 represents the pose transformation matrix of the endoscopic camera, and R 光学定位相机下刚体球 represents the pose state information.

[0157] Step S802: The endoscopic device performs transformation processing on the first coordinate information of the target detection part based on the transformation matrix between the first coordinate system and the second coordinate system to determine the second coordinate information of the target detection part.

[0158] Based on the above, the second coordinate information of the target detection part in the second coordinate system corresponding to the optical positioning camera can be obtained.

[0159] Step S702: The endoscopic device performs a transformation process on the second coordinate information based on the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset, and determines the CT coordinate information corresponding to the target detection site.

[0160] Among them, in the embodiments of the present application, the third coordinate system corresponding to the CT image dataset is the CT image coordinate system.

[0161] It should be noted that before applying the image processing method of the present application, it is necessary to pre-determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0162] Optionally, as Figure 12 shown, the embodiments of the present application provide a flowchart for determining the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset. The specific steps are as follows:

[0163] Step S1201: The endoscopic device acquires the coordinate information of multiple marker points on the reference site model collected by the optical positioning camera in the second coordinate system corresponding to the optical positioning camera;

[0164] As Figure 13 shown, the embodiments of the present application pre-prepare a reference site model with multiple marker points.

[0165] Optionally, the reference site model in the embodiments of the present application can be a three-dimensional liver model.

[0166] It should be noted that the coordinate information of multiple marker points on the reference site model in the second coordinate system corresponding to the optical positioning camera in the embodiments of the present application is determined by the optical positioning camera obtaining the point cloud data on the surface of the reference site model through a probe.

[0167] The present application acquires through the optical positioning camera Figure 13 the coordinate information of multiple marker points in the reference site model shown in the corresponding second coordinate system of the optical positioning camera.

[0168] Step S1202: The endoscopic device determines the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset based on the coordinate information of multiple marker points corresponding in the second coordinate system and the reference CT coordinate information of multiple marker points collected by the CT image acquisition device.

[0169] In implementation, the embodiments of the present application use the CT image acquisition device to acquire the reference CT image dataset corresponding to the reference site model. Among them, the reference CT image corresponding to any reference CT image data in the reference CT image dataset can be as Figure 14 shown.

[0170] Among them, the reference CT image dataset includes reference CT coordinate information corresponding to multiple marker points.

[0171] Based on the coordinate information of each marker point on the reference part model corresponding to the second coordinate system of the optical positioning camera and the reference CT coordinate information, the embodiment of the present application determines the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0172] Optionally, the embodiment of the present application can calculate the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset through a point cloud registration algorithm.

[0173] Before applying the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset to the image processing method of the present application, the embodiment of the present application can also check the pre-determined transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0174] Optionally, as Figure 15 shown, a flowchart of a verification error in the embodiment of the present application is as follows:

[0175] Step S1501, the endoscope device collects reference endoscope image data including multiple marker points on the reference part model through the endoscope camera; and obtains the reference CT coordinate information corresponding to the multiple marker points collected by the CT image acquisition device;

[0176] Among them, the reference endoscope image data of the multiple marker points at least includes the reference coordinate information of the multiple marker points in the endoscope image.

[0177] Step S1502, based on the reference endoscope image data, the endoscope device determines the test CT coordinate information corresponding to the multiple marker points through the pose transformation matrix of the endoscope camera and the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset;

[0178] Optionally, the present application determines the test CT coordinate information through Formula Four:

[0179] P Ri =P i *R 12 *R 23 ’ *R 34 ——Formula Four

[0180] Among them, P Ri represents the test CT coordinate information corresponding to the marker point, P i represents the reference coordinate information of the multiple marker points, and R 12represents a pre-determined coordinate transformation matrix for transforming coordinate information in the endoscopic image into first coordinate information in the first coordinate system corresponding to the endoscopic camera, R 23 ’ represents the transformation matrix between the current first coordinate system and the second coordinate system, R 34 represents a pre-determined transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

[0181] It should be noted that, as can be seen from Equation 3, the transformation matrix between the first coordinate system and the second coordinate system is determined according to the pose transformation matrix of the endoscopic camera and the pose state information of the endoscopic camera. The pose transformation matrix of the endoscopic camera is pre-determined, while the pose state information of the endoscopic camera is collected in real time. Therefore, based on the current pose state information of the endoscopic camera and the pre-determined pose transformation matrix of the endoscopic camera, R can be obtained 23 ’ .

[0182] Among them, the specific method and principle for determining R 12 refer to Equation 2. The specific method and principle for determining the transformation matrix R 34 between the second coordinate system and the third coordinate system corresponding to the CT image dataset refer to Step S1202.

[0183] Exemplarily, as Figure 16 shown, it is a schematic diagram of the test CT coordinate information corresponding to multiple marker points in an embodiment of the present application. Among them, as Figure 16 the black dots in are displayed based on the determined test CT coordinate information corresponding to multiple marker points.

[0184] It should be noted that, for the convenience of observing the differences between the coordinate information corresponding to multiple marker points on the reference model in the endoscopic image data and the determined test CT coordinate information corresponding to multiple marker points, the present application fuses and displays the corresponding three-dimensional model determined based on the CT image data and the endoscopic image data collected by the endoscopic camera. Among them, as Figure 16 the black dots in are displayed on the three-dimensional model.

[0185] Step S1503: The endoscopic device determines that the error between the test CT coordinate information corresponding to multiple marker points and the reference CT coordinate information satisfies a preset error condition.

[0186] Optionally, the present application determines the error between the test CT coordinate information corresponding to multiple marker points and the reference CT coordinate information through Equation 5:

[0187]

[0188] Among them, Error represents the error, Represents the reference CT coordinate information corresponding to the marked points, P Ri Represents the test CT coordinate information corresponding to the marked points, and N represents the number of marked points.

[0189] Optionally, based on the error between the test CT coordinate information and the reference CT coordinate information corresponding to the determined multiple marked points, this application compares the error with an error threshold. When it is determined that the error is not greater than the error threshold, it is determined that the preset error condition is satisfied.

[0190] Exemplarily, as Figure 17 shown, it is a schematic diagram of the test CT coordinate information corresponding to multiple marked points that satisfy the error condition in an embodiment of this application.

[0191] After determining that the error condition is satisfied in an embodiment of this application, the transformation matrix R between the third coordinate systems corresponding to the pre-determined CT image dataset is used 34 to perform transformation processing on the second coordinate information to determine the CT coordinate information corresponding to the target detection part.

[0192] It should be noted that after determining that the error satisfies the preset error condition, when performing the image processing method of this application, the pre-determined R can also be used 12 , and the pre-determined pose transformation matrix of the endoscope camera to determine the CT coordinate information corresponding to the target detection part.

[0193] Among them, if the position of the positioning marker on the endoscope device is changed, the pose transformation matrix of the endoscope camera needs to be re-determined.

[0194] Optionally, if it is determined in an embodiment of this application that the preset error condition is not satisfied, then R needs to be re-determined 12 , the pose transformation matrix of the endoscope camera, and the transformation matrix R between the second coordinate system and the third coordinate system corresponding to the CT image dataset 34 .

[0195] Exemplarily, this application determines the CT coordinate information corresponding to the target detection part through Formula Six:

[0196] P Ri ’ = P i ’ * R 12 * R 23 * R 34 ——Formula Six

[0197] Among them, P Ri ’ represents the CT coordinate information corresponding to the target detection part, and P i ’ represents the coordinate information corresponding to the target detection part in the endoscope image data including the target detection part.

[0198] After the CT coordinate information corresponding to the target detection part is determined in the embodiment of the present application, step S403 is executed.

[0199] Step S403: The endoscope device acquires a CT image dataset including the target detection part collected by the CT image acquisition device. The CT image dataset includes multiple CT image data; based on the CT coordinate information corresponding to the target detection part, the target CT image data matching the endoscope image data is determined in the CT image dataset.

[0200] It should be noted that before executing the image processing method of the present application, a CT image dataset including the target detection part needs to be collected from the patient in advance by the CT image acquisition device.

[0201] Optionally, if the original coordinate information included in the CT image data and the CT coordinate information corresponding to the target detection part meet the matching condition, then the CT image data is determined as the target CT image data, and it is determined that the target CT image data matches the CT coordinate information corresponding to the target detection part.

[0202] Optionally, the following several methods are used in the embodiment of the present application to determine that the original coordinate information included in the CT image data and the CT coordinate information corresponding to the target detection part meet the matching condition:

[0203] Determination method 1: If the original coordinate information included in the CT image data and the CT coordinate information corresponding to the target detection part are both the same, then it is determined that the CT image data matches the CT coordinate information corresponding to the target detection part.

[0204] Determination method 2: If the number of the same coordinate information between the original coordinate information included in the CT image data and the CT coordinate information corresponding to the target detection part exceeds the preset quantity threshold, then it is determined that the CT image data matches the CT coordinate information corresponding to the target detection part.

[0205] Determination method 3: If the error between the original coordinate information included in the CT image data and the CT coordinate information corresponding to the target detection part is less than the preset threshold, then it is determined that the CT image data matches the CT coordinate information corresponding to the target detection part.

[0206] Based on the process of obtaining the target CT image data by acquiring the endoscope image data as described above, such as Figure 18As shown in the figure, it is an overall schematic diagram of an embodiment of the present application. Among them, after collecting endoscope image data through an endoscope device, the endoscope coordinate information of the endoscope image data in the endoscope image coordinate system is transformed into the first coordinate information in the endoscope camera coordinate system; then the determined first coordinate information in the endoscope camera coordinate system is transformed into the second coordinate information in the optical positioning camera coordinate system; the determined second coordinate information in the optical positioning camera coordinate system is transformed into the CT coordinate information in the CT image coordinate system, and then based on the determined CT coordinate information, the matching target CT image data is obtained from the pre-collected CT image dataset.

[0207] Step S404: The endoscope device displays the target image after fusing the endoscope image corresponding to the endoscope image data and the CT image corresponding to the target CT image data based on the data after the fusion processing of the endoscope image data and the target CT image data.

[0208] Optionally, in the embodiment of the present application, the endoscope image data and the target CT image data are fused, and the target image after fusing the endoscope image corresponding to the endoscope image data and the CT image corresponding to the target CT image data is displayed.

[0209] It should be noted that since the CT image corresponding to the target CT image data is three-dimensional, the displayed target image is the image after fusing the two-dimensional endoscope image and the three-dimensional CT image.

[0210] Based on the same inventive concept, the above-mentioned image processing method of the present invention can also be implemented by an image processing device. The effect of this image processing device is similar to that of the foregoing method, and will not be elaborated here.

[0211] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. The principle of the electronic device to solve problems is similar to that of the method in the above embodiment. Therefore, the implementation of this electronic device can refer to the implementation of the above method, and the repeated parts will not be elaborated.

[0212] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0214] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0216] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. An endoscope device, characterized in that, The endoscope device includes an endoscope camera, a display, and a processor, and the endoscope device is connected to an optical positioning camera; The endoscope camera is configured to collect endoscope image data including a target detection site during a surgical procedure and transmit the endoscope image data to the processor; The optical positioning camera is configured to determine the current pose state information of the endoscope camera and transmit the pose state information to the processor; The processor is configured to, based on the endoscope image data, determine the first coordinate information of the target detection site in a first coordinate system corresponding to the endoscope camera, and perform transformation processing on the first coordinate information corresponding to the target detection site based on the current pose state information of the endoscope camera and a transformation matrix between a second coordinate system corresponding to the optical positioning camera and a third coordinate system corresponding to the CT image data set to determine the CT coordinate information corresponding to the target detection site; Obtain a CT image data set including the target detection site collected by a CT image acquisition device, where the CT image data set includes a plurality of CT image data; based on the CT coordinate information, determine target CT image data matching the endoscope image data in the CT image data set, and transmit the data obtained by fusing the endoscope image data and the target CT image data to the display; The display is configured to, based on the data obtained by fusing the endoscope image data and the target CT image data, display a target image obtained by fusing the endoscope image corresponding to the endoscope image data and the CT image corresponding to the target CT image data.

2. The endoscope device according to claim 1, characterized in that, Specifically, the processor is configured to: Perform transformation processing on the first coordinate information based on the current pose state information of the endoscope camera to determine the second coordinate information of the target detection site in a second coordinate system corresponding to the optical positioning camera; Perform transformation processing on the second coordinate information based on a transformation matrix between the second coordinate system and a third coordinate system corresponding to the CT image data set to determine the CT coordinate information corresponding to the target detection site.

3. The endoscopic device according to claim 2, characterized in that The endoscope device further includes a positioning marker, the positioning marker is disposed at a specific position of the endoscope device, and the relative position between the positioning marker and the endoscope camera remains unchanged during the use of the endoscope device; specifically, the processor is configured to: Determine a transformation matrix between the first coordinate system and the second coordinate system based on the current pose state information of the endoscope camera and a pose transformation matrix of the endoscope camera; wherein, the current pose state information of the endoscope camera is determined by the optical positioning camera based on the coordinate information of the positioning marker collected currently in a second coordinate system corresponding to the optical positioning camera; the pose transformation matrix of the endoscope camera is used to represent the relative position relationship between the endoscope camera and the positioning marker; Based on the transformation matrix between the first coordinate system and the second coordinate system, perform transformation processing on the first coordinate information of the target detection part to determine the second coordinate information of the target detection part.

4. The endoscopic device according to any one of claims 1 to 3, characterized in that, Determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset in the following manner: Obtain the coordinate information of multiple marker points on the reference part model collected by the optical positioning camera in the second coordinate system corresponding to the optical positioning camera; Based on the coordinate information of the multiple marker points in the second coordinate system and the reference CT coordinate information of the multiple marker points collected by the CT image acquisition device, determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

5. The endoscope device according to claim 3, characterized in that, The processor is further configured to: Collect reference endoscope image data including multiple marker points on the reference part model through the endoscope camera; and obtain reference CT coordinate information including the multiple marker points collected by the CT image acquisition device; Based on the reference endoscope image data, determine the test CT coordinate information corresponding to the multiple marker points through the pose transformation matrix of the endoscope camera and the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset; Determine that the error between the test CT coordinate information corresponding to the multiple marker points and the reference CT coordinate information satisfies a preset error condition.

6. An image processing method, characterized in that, Applied to an endoscope device, the method includes: Obtain endoscope image data including a target detection part collected by the endoscope camera during a surgical procedure, and determine the first coordinate information of the target detection part in the first coordinate system corresponding to the endoscope camera; Obtain the current pose state information of the endoscope camera determined by the optical positioning camera, and based on the current pose state information of the endoscope camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image dataset, perform transformation processing on the first coordinate information corresponding to the target detection part to determine the CT coordinate information corresponding to the target detection part; Obtain a CT image dataset including the target detection part collected by the CT image acquisition device, where the CT image dataset includes multiple CT image data; based on the CT coordinate information corresponding to the target detection part, determine target CT image data that matches the endoscope image data in the CT image dataset; Based on the data after fusion processing of the endoscope image data and the target CT image data, display a target image after fusing the endoscope image corresponding to the endoscope image data and the CT image corresponding to the target CT image data.

7. The method according to claim 6, wherein The performing transformation processing on the first coordinate information corresponding to the target detection part based on the current pose state information of the endoscope camera and the transformation matrix between the second coordinate system corresponding to the optical positioning camera and the third coordinate system corresponding to the CT image dataset to determine the CT coordinate information corresponding to the target detection part includes: Perform a transformation process on the first coordinate information based on the current pose state information of the endoscopic camera to determine the second coordinate information of the target detection part in the second coordinate system corresponding to the optical positioning camera; Based on the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset, perform a transformation process on the second coordinate information to determine the CT coordinate information corresponding to the target detection part.

8. The method according to claim 7, wherein The endoscopic device further includes a positioning marker, which is arranged at a specific position of the endoscopic device, and the relative position between the positioning marker and the endoscopic camera remains unchanged during the use of the endoscopic device; The performing a transformation process on the first coordinate information based on the current pose state information of the endoscopic camera to determine the second coordinate information of the target detection part in the second coordinate system corresponding to the optical positioning camera includes: Based on the current pose state information of the endoscopic camera and the pose transformation matrix of the endoscopic camera, determine the transformation matrix between the first coordinate system and the second coordinate system; wherein, the current pose state information of the endoscopic camera is determined by the optical positioning camera based on the coordinate information of the positioning marker collected currently in the second coordinate system corresponding to the optical positioning camera; the pose transformation matrix of the endoscopic camera is used to represent the relative position relationship between the endoscopic camera and the positioning marker; Based on the transformation matrix between the first coordinate system and the second coordinate system, perform a transformation process on the first coordinate information of the target detection part to determine the second coordinate information of the target detection part.

9. The method according to any one of claims 6-8, characterized in that Determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset in the following manner: Obtain the coordinate information of multiple marked points on the reference part model collected by the optical positioning camera in the second coordinate system corresponding to the optical positioning camera; Based on the coordinate information of the multiple marked points corresponding in the second coordinate system and the reference CT coordinate information corresponding to the multiple marked points collected by the CT image acquisition device, determine the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset.

10. The method according to claim 8, wherein Before obtaining the endoscopic image data containing the target detection part collected by the endoscopic camera during the operation, it further includes: Collect reference endoscopic image data including multiple marked points on the reference part model through the endoscopic camera; and obtain the reference CT coordinate information corresponding to the multiple marked points collected by the CT image acquisition device; Based on the reference endoscopic image data, determine the test CT coordinate information corresponding to the multiple marked points through the pose transformation matrix of the endoscopic camera and the transformation matrix between the second coordinate system and the third coordinate system corresponding to the CT image dataset; Determine that the error between the test CT coordinate information corresponding to the multiple marked points and the reference CT coordinate information meets the preset error condition.

Images