Thoracoscope device and method, apparatus and product for determining an observation area image thereof

Abstract

The embodiment of the application discloses a thoracoscope device and a method, device and product for determining an observation area image thereof, and belongs to the technical field of medical image processing. The method comprises the following steps: acquiring an observation direction of an endoscope, a preoperative three-dimensional image of a target object, and virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image; for each surgical instrument, determining an in-vivo working length of the surgical instrument, wherein the in-vivo working length is determined based on a skin contact line of the surgical instrument and an instrument model, and the instrument model comprises a plurality of markers and lengths from each marker to a distal end of the surgical instrument; and determining an observation area image corresponding to a current observation area in the preoperative three-dimensional image and a pose of each surgical instrument in the observation area image according to a combination of the virtual access points corresponding to each surgical instrument, the in-vivo working length, the instrument model and in-vivo markers in an intraoperative video. The technical scheme provided by the embodiment of the application can significantly improve the operation experience of a user.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a thoracoscopic device and a method, apparatus and product for determining the image of the observation area. Background Technology

[0002] When performing surgery with existing thoracoscopic equipment, surgeons can only rely on intraoperative video images transmitted from the endoscope to carry out all operations. This operating mode is highly dependent on the surgeon's clinical experience, has a high threshold for clinical expertise, and offers a relatively low level of surgical experience. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, storage medium, and program product for determining an observation area image, in order to improve the user's surgical operation experience.

[0004] According to one aspect of the present invention, a method for determining an observation area image is provided, applied to a processor of a thoracoscopic device, the method comprising: In response to a predetermined trigger operation, the endoscope's observation direction, the preoperative three-dimensional image of the target object, and the virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image are acquired. For each of the surgical instruments, the in vivo working length of the surgical instrument is determined. The in vivo working length is determined based on the skin contact line of the surgical instrument and the instrument model, which includes multiple marks and the length of each mark to the end of the surgical instrument. Based on the virtual access point corresponding to each surgical instrument, the in vivo working length, the instrument model, and the combination of in vivo markers in the intraoperative video, the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image are determined. The image of the observation area and the pose of each surgical instrument in the image of the observation area are displayed from the observation direction.

[0005] According to another aspect of the present invention, an observation area image determination apparatus is provided, the apparatus comprising: The acquisition module is used to acquire the observation direction of the endoscope, the preoperative three-dimensional image of the target object, and the virtual access point corresponding to each surgical instrument in the preoperative three-dimensional image in response to a predetermined trigger operation. A length module is used to determine the in vivo working length of each of the surgical instruments, the in vivo working length being determined based on the skin contact line of the surgical instrument and an instrument model, the instrument model including multiple markers and the length of each marker to the end of the surgical instrument; The determination module is used to determine the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image, based on the virtual access point corresponding to each surgical instrument, the in vivo working length, the instrument model, and the combination of in vivo markers in the intraoperative video. The display module is used to display the image of the observation area and the pose of each surgical instrument in the image of the observation area from the observation direction.

[0006] According to another aspect of the present invention, a thoracoscopy device is provided, the thoracoscopy device comprising: A first surgical instrument and a second surgical instrument, wherein portions of the first surgical instrument and the second surgical instrument extend into the body of the target object to perform surgical operations, and include markings distributed along the axial direction, wherein the lengths of the different markings to the ends of the surgical instruments are different and known. Endoscopes are used to provide intraoperative video. A processor for executing the observation area image determination method described in any embodiment.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement the observation area image determination method as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the observation area image determination method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the observation area image determination method as described in any embodiment of the present invention.

[0010] The technical solution provided by this invention predetermines the instrument models of each surgical instrument and determines the in vivo working length of the surgical instrument based on the skin contact line and the instrument model. This achieves the technical effect of accurately determining the in vivo working length of the surgical instrument without relying on complex sensors or electromagnetic positioning systems. By using the virtual approach points corresponding to each surgical instrument in the preoperative 3D image, the intraoperative video, and the instrument models and in vivo working lengths of each surgical instrument, the current spatial pose information of each surgical instrument during the operation is uniformly mapped to the preoperative 3D image coordinate system, thereby determining the observation area image affected by the surgical instrument, i.e., the observation area image corresponding to the current observation area. Along the observation direction, the observation area image and the pose of each surgical instrument in the observation area image are displayed, realizing real-time or near real-time display of the observation area image and the pose of each surgical instrument in the observation area image. This provides a unified and quantifiable spatial reference benchmark for intraoperative video understanding, operation area analysis, and instrument behavior analysis, which is conducive to improving the intelligence level of surgical navigation, operational safety, and user surgical experience.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1A This is a schematic diagram of the thoracoscopic device provided in an embodiment of the present invention; Figure 1B This is another structural schematic diagram of the thoracoscopic device provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the method for determining the observation area image provided in an embodiment of the present invention. Figure 3 This is another flowchart illustrating the method for determining the observation area image provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the observation area image determination device provided in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Figure 1A This is a schematic diagram of the structure of a thoracoscopic device provided in an embodiment of the present invention. As can be seen from the figure, the thoracoscopic device 10 includes: A first surgical instrument 101 and a second surgical instrument 102, wherein a portion of the first surgical instrument and the second surgical instrument extends into the body of the target object to perform a surgical operation, and include markings distributed along the axial direction, wherein the lengths of the different markings to the ends of the surgical instruments are different and known. Endoscope 103 is used to provide intraoperative video; Processor 11 is configured to execute the observation area image determination method described in any of the following embodiments.

[0017] The first and second surgical instruments can be selected from scissors, needle holders, electrocautery hooks, or ultrasonic scalpels, and are usually different from each other. Both the first and second surgical instruments are partially inserted into the target body to perform surgical operations such as cutting, suturing, clamping, or electrocautery.

[0018] The endoscope is used to acquire real-time images of the surgical area and transmits the acquired image data to the image processing unit via signal transmission cable or wireless communication module. After processing, the image processing unit displays the data on the display device of the doctor's console so that the operator can observe the surgical field in real time.

[0019] During preoperative planning, users select the planned approach area and virtual approach point based on preoperative 3D images, the location of the target lesion, the target lung segment, the chest wall structure, the location of the intercostal spaces, and conventional thoracoscopic approach habits.

[0020] Surgical instruments enter the target body through actual entry points such as surface incisions or cannulas. Matching the actual entry point to the patient's body coordinate system is primarily based on the chest wall, ribs, and intercostal spaces. Specifically, the corresponding chest wall region can be located in the patient's body coordinate system based on the intercostal space where the virtual entry point is located, its relative position to adjacent ribs, and its proximity to the anterior, mid, or posterior axillary lines. After the chest wall region is determined, the actual entry point is determined based on the specific virtual entry point specified by the surgeon in the 3D model. In this embodiment, this actual entry point can be optionally used as the starting point for the surgical instruments in the patient's body coordinate system.

[0021] It should be noted that in actual surgical scenarios, during preoperative planning, it is usually necessary to determine the virtual entry point for the endoscope; during the actual surgery, it is necessary to determine the actual entry point for the endoscope and insert the endoscope into the target body from that actual entry point. However, the data processing flow involved in this application only considers the coordinates of the virtual entry point for the surgical instruments; therefore, the virtual entry points mentioned below are all the planned entry points for the surgical instruments.

[0022] like Figure 1B As shown, the thoracoscopic device 10 also includes a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, which are communicatively connected to the processor 11. The memory stores computer programs executable by at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from the storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0023] Multiple components in the thoracoscopic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0024] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described below, such as the observation area image determination method.

[0025] Figure 2 This is a flowchart illustrating the observation area image determination method provided in an embodiment of the present invention. This embodiment is applicable to situations where, during surgery, the observation area image corresponding to the current observation area is automatically determined in a preoperative three-dimensional image. This method can be executed by an observation area image determination device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 2 As shown, the method in this embodiment includes: S110. In response to a predetermined trigger operation, acquire the endoscope's observation direction, the preoperative three-dimensional image of the target object, and the virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image.

[0026] The scheduled trigger action can be a foot-triggered action or other trigger actions, such as a voice-triggered action.

[0027] The direction of observation of the endoscope can be determined based on existing technology, such as by a position sensor set at the end of the endoscope.

[0028] The target group is patients undergoing surgery.

[0029] Preoperative 3D images are reconstructed based on 3D image data acquired preoperatively for the target subject. The 3D image data can be selected from 3D CT data, MRI data, etc. Specifically, the 3D CT data can be selected from 3D thin-slice CT data, 3D enhanced CT data, etc.

[0030] The virtual entry point is the position on the body surface where the surgical instrument enters the target anatomical region in the preoperative 3D image. This virtual entry point corresponds to the actual entry point of the surgical instrument on the target object.

[0031] It should be noted that the surgical instruments in this embodiment are rigid surgical instruments, which will not deform during the operation, especially not along the axial direction.

[0032] S120. For each surgical instrument, determine the working length inside the surgical instrument. The working length inside the surgical instrument is determined based on the skin contact line of the surgical instrument and the instrument model. The instrument model includes multiple marks and the length of each mark to the end of the surgical instrument.

[0033] The working length inside the body refers to the length of the surgical instrument extending into the target body from the actual entry point along its own shaft.

[0034] The skin contact line can be understood as the contact point on the body surface of a surgical instrument. The skin contact line of a surgical instrument is a dynamic value; one side extends into the target body, becoming the effective working section; the other side is exposed outside the target body, becoming the ineffective working section.

[0035] An instrument model can be understood as a pre-defined geometric model of a surgical instrument, including multiple markers and the length of each marker to the end of the instrument. Of course, an instrument model may also include attribute data such as the total length of the surgical instrument, its model number, the number of markers, the marker spacing, and the marker encoding method.

[0036] In one embodiment, for each surgical instrument, the user determines the skin contact line of the surgical instrument and the mark corresponding to the skin contact line; then determines the length corresponding to the mark in the instrument model and uses the length as the in-body working length of the surgical instrument; then inputs the in-body working length into the interactive interface and associates the in-body working length with the instrument identifier of the surgical instrument.

[0037] In one embodiment, for each surgical instrument, a target marker corresponding to the surgical instrument is determined in the external video, and the in vivo working length corresponding to the target marker is determined according to the instrument model; wherein, the target marker is the marker corresponding to the skin contact line.

[0038] The external video can be surgical monitoring video or video captured by a specialized image acquisition device. The former is surgical monitoring video, while the latter is video showing the external status of surgical instruments.

[0039] For each surgical instrument, the mark corresponding to the skin contact line of the surgical instrument is extracted from the external video and used as the target mark; then, based on the instrument model of the surgical instrument, the length corresponding to the target mark is determined, which is the length of the surgical instrument left in the target body, i.e., the in-body working length.

[0040] The marker combination includes at least two ring structures distributed along the axial direction, and the attribute data of different ring structures are different.

[0041] In one embodiment, the attribute data can be selected as color. The surgical instrument has different color rings arranged sequentially from its starting point to its ending point, with each color ring representing a fixed distance interval. Based on the color ring corresponding to the skin contact line of the surgical instrument, the current exposed length and in-body working length of the surgical instrument can be determined.

[0042] In one embodiment, the mark may also be a scale line, a ring structure of different widths, a digital code mark, a raised structure, a groove structure, a stepped structure, a mark with different spacing combinations, a fluorescent mark, or a composite mark with a unique coding feature, etc.

[0043] The working length inside the body and the current exposed length can be either exact values ​​or estimates. For example, if marked as scale lines, the working length inside the body is an exact value within an acceptable error range; if marked as color rings of a certain width, the working length inside the body corresponding to the sequentially arranged color rings is a stepped shape, and the working length inside the body determined based on the color ring corresponding to the skin contact line is an estimate.

[0044] S130. Based on the virtual access point, in vivo working length, instrument model, and in vivo marker combination in the intraoperative video corresponding to each surgical instrument, determine the observation area image corresponding to the current observation area in the preoperative three-dimensional image, as well as the pose of each surgical instrument in the observation area image.

[0045] Since the virtual access point of each surgical instrument corresponds to the actual access point, once the combination of the virtual access point, working length in vivo, instrument model, and in vivo marker in the intraoperative video for each surgical instrument is determined, spatial constraints between the surgical instruments and the preoperative 3D image can be established. Since the surgical instruments are definitely located within the current observation area, the surgical instruments can be used as intraoperative spatial reference objects to determine the observation area image corresponding to the current observation area in the preoperative 3D image, as well as the pose of each surgical instrument in the observation area image.

[0046] Specifically, the current observation area is the area affected by the surgical instrument end effector, and the effects include at least one of physical contact, heat transfer, or tissue deformation. In other words, the area at least covers the working range of the end effector.

[0047] The observation area image is the portion of the preoperative 3D image that corresponds to the current observation area, i.e., the image area corresponding to the range of influence of the surgical instruments.

[0048] In one embodiment, all virtual approach points, all in vivo working lengths, all instrument models, and all in vivo markers, along with preoperative 3D images, are input into a pre-trained machine learning model to obtain an observation area image and the poses of each surgical instrument within that image. This embodiment can quickly and accurately determine the observation area image.

[0049] S140. From the observation direction, display the image of the observation area and the position of each surgical instrument in the image of the observation area.

[0050] This step aims to display the image of the observation area and the current position of each surgical instrument within the image, enabling the surgeon to intuitively determine the current anatomical space of the surgical instruments, thereby improving the accuracy of surgical judgment and the safety of the surgery.

[0051] To help users distinguish between the observation area image and the surgical instruments, the observation area image and the surgical instruments in the displayed results are identified by different colors.

[0052] In one embodiment, the observed area image and the two surgical instruments are identified by different colors in the results display.

[0053] In one embodiment, from the viewing direction, an image of the observation area and the poses of the axes and markings of each surgical instrument within the image are displayed. This embodiment does not display images of the surgical instruments, but only their axes and markings, to improve display speed.

[0054] In one embodiment, from the viewing direction, an image of the viewing area and the pose of each surgical instrument in the image of the viewing area are displayed, wherein the surgical instrument is drawn based on the instrument parameter data in the instrument model.

[0055] In one embodiment, the image of the observation area is superimposed onto the surgical instruments in the intraoperative video, and the relative positional relationship between the surgical instruments and the image of the observation area remains unchanged when the superimposed result is displayed.

[0056] The technical solution provided by this invention predetermines the instrument models of each surgical instrument and determines the in vivo working length of the surgical instrument based on the skin contact line and the instrument model. This achieves the technical effect of accurately determining the in vivo working length of the surgical instrument without relying on complex sensors or electromagnetic positioning systems. By using the virtual approach points corresponding to each surgical instrument in the preoperative 3D image, the intraoperative video, and the instrument models and in vivo working lengths of each surgical instrument, the current spatial pose information of each surgical instrument during the operation is uniformly mapped to the preoperative 3D image coordinate system, thereby determining the observation area image affected by the surgical instrument, i.e., the observation area image corresponding to the current observation area. Along the observation direction, the observation area image and the pose of each surgical instrument in the observation area image are displayed, realizing real-time or near real-time display of the observation area image and the pose of each surgical instrument in the observation area image. This provides a unified and quantifiable spatial reference benchmark for intraoperative video understanding, operation area analysis, and instrument behavior analysis, which is conducive to improving the intelligence level of surgical navigation, operational safety, and user surgical experience.

[0057] Based on the aforementioned embodiments, S140, displaying the image of the observation area and the poses of each surgical instrument in the image of the observation area from the observation direction, can be refined to displaying the predetermined anatomical structures in the image of the observation area and the poses of each surgical instrument along the observation direction.

[0058] The predetermined anatomical structures are key anatomical structures that are identified in advance, and their specific contents correspond to the target body parts in the preoperative 3D image. Taking a 3D image of the chest as an example, the predetermined anatomical structures include parts or all of the lung lobes, lung segments, target lesions, pulmonary arteries, pulmonary veins, bronchi, pleura, chest wall, and hilar region.

[0059] Specifically, after the observation area image is determined, the predetermined anatomical structure is identified in the observation area image. Then, the display device, such as the worktable, is controlled to display the predetermined anatomical structure identification results and the poses of each surgical instrument along the observation direction. In this way, the user can learn from the display content about the pose relationship between the surgical instruments and each target anatomical structure.

[0060] The predetermined anatomical structure identification results include one or more partial predetermined anatomical structures, and / or one or more complete predetermined anatomical structures.

[0061] This embodiment only displays the predetermined anatomical structures and the current positions of the surgical instruments that are of interest to the user, excluding non-predetermined anatomical structures. This reduces the interference of non-predetermined anatomical structures with the surgeon's visual judgment, allowing the surgeon to focus more clearly on the predetermined anatomical structures relevant to the current operation, thereby improving the accuracy of surgical judgment and the safety of the surgery. Non-predetermined anatomical structures include, but are not limited to, anatomical structures such as bone tissue, adipose tissue, and intestines that do not affect the judgment of the current surgical operation.

[0062] Figure 3 This is another schematic flowchart illustrating the observation area image provided in an embodiment of the present invention. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 3 As shown, the method in this embodiment may specifically include: S210, in response to a predetermined trigger operation, acquire the endoscope's observation direction, the preoperative three-dimensional image of the target object, and the virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image.

[0063] S220. For each surgical instrument, determine the working length inside the surgical instrument. The working length inside the surgical instrument is determined based on the skin contact line of the surgical instrument and the instrument model. The instrument model includes multiple marks and the length of each mark to the end of the surgical instrument.

[0064] S2301. For each surgical instrument, generate multiple target candidate poses based on the virtual access point and the working length inside the body.

[0065] During the operation, the surgical instruments swing and rotate around the actual access hole on the body surface as the fulcrum. That is, the end of the surgical instruments is located on a sphere with the actual access hole on the body surface as the center and the working length inside the body as the radius.

[0066] To this end, uniform sampling can be performed on the sphere to generate candidate end position combinations, where each candidate end position corresponds to a candidate pose.

[0067] In one embodiment, multiple candidate poses corresponding to each surgical instrument are directly used as multiple target candidate poses corresponding to it.

[0068] In one embodiment, for each surgical instrument, multiple candidate postures of each instrument are filtered based on the instrument's limiting angle to obtain multiple target candidate postures. This embodiment directly eliminates impossible candidate postures through candidate posture filtering, reducing the amount of data processing involved in subsequent projection processing and comparison of projection results, thereby improving the processing speed of this process.

[0069] S2302. Cross-combine the candidate postures of various surgical instruments to obtain multiple candidate posture combinations.

[0070] The pairings between candidate poses of each surgical instrument are enumerated to obtain multiple candidate pose combinations; each candidate pose combination includes one candidate pose of each surgical instrument.

[0071] For example, the first surgical instrument corresponds to multiple first target candidate postures, the second surgical instrument corresponds to multiple second target candidate postures, and a candidate posture combination includes a first target candidate posture and a second target candidate posture.

[0072] S2303. Project each candidate pose combination onto the target plane to obtain multiple projection results. The target plane is the plane where the intraoperative video is located.

[0073] Intraoperative video is captured by an endoscope. The target plane is the two-dimensional imaging plane where the intraoperative video is located, i.e., the image plane of the endoscope. The spatial position and orientation of this plane can be determined by the imaging parameters corresponding to the current pose of the endoscope, including parameters such as the current optical center position, optical axis direction, and field of view.

[0074] For each candidate pose combination, a projection transformation matrix is ​​determined based on the imaging parameters. Then, the candidate pose combination is mapped based on the projection transformation matrix to obtain the projection result.

[0075] In one embodiment, the projection result includes the axes of each surgical instrument and a plurality of marks distributed on the axes and located in the body, which is a combination of projection marks.

[0076] In one embodiment, the candidate pose combination includes only the marker coordinates on the internal segments of each surgical instrument, and correspondingly, the projection result includes only the marker projection result of each surgical instrument, that is, only the projection marker combination of each surgical instrument.

[0077] S2304. Based on the distance between markers, the direction of marker arrangement, the distance between instruments, the angle between instruments, and the positional relationship between each surgical instrument and the image center, determine the target projection result that matches the intraoperative video.

[0078] Intraoperative video was labeled and identified to obtain in vivo label combinations for each surgical instrument.

[0079] This embodiment utilizes the spatial geometric constraints between different surgical instruments and the distribution characteristics of the markings on each surgical instrument to select the target projection result that best matches the intraoperative video from multiple projection results.

[0080] Understandably, if a candidate pose combination represents the actual pose combination of all surgical instruments, then projecting this candidate pose combination onto the target plane should result in a projection that closely matches the instrument observation characteristics of the intraoperative video. Specifically, the projection result should be consistent with the intraoperative video in the following geometric features: the distance between markers on the same surgical instrument, the orientation of the markers, the shadow distance between surgical instruments, the angle between the axes of the surgical instruments, and the positional relationship between each surgical instrument and the image center. Based on these principles, constraints are constructed to evaluate the degree of matching between each projection result and the intraoperative video, and a target projection result matching the intraoperative video is selected from multiple projection results using these constraints.

[0081] S2305. Based on the candidate pose combination corresponding to the target projection result, determine the observation area image corresponding to the current observation area in the preoperative three-dimensional image, as well as the pose of each surgical instrument in the observation area image.

[0082] The candidate pose combination corresponding to the target projection result is taken as the target pose combination. Based on the pose of each surgical instrument in the target pose combination, the image part affected by the surgical instruments in the preoperative 3D image is determined, and this image part is taken as the observation area image corresponding to the current observation area, as well as the pose of each surgical instrument in the observation area image.

[0083] Understandably, the working length of each surgical instrument within the body is enclosed in the observation area, and the representation information of each surgical instrument in the image of the observation area is enclosed in the image of the observation area.

[0084] S240. From the observation direction, display the image of the observation area and the position of each surgical instrument in the image of the observation area.

[0085] The technical solution provided in this invention first generates multiple candidate poses for each surgical instrument; then, the candidate poses of each surgical instrument are cross-combined to obtain multiple candidate pose combinations; then, each candidate pose combination is projected onto a target plane to obtain multiple projection results, making the poses of the surgical instruments comparable to the intraoperative video; based on the distance between markers, the marker arrangement direction, the distance between instruments, the angle between instruments, and the positional relationship between each surgical instrument and the image center, the target projection result matching the intraoperative video is determined. This achieves improved accuracy in matching the projection result with the intraoperative video through multi-dimensional constraints, thereby improving the target projection result. The accuracy of the result determination is crucial; once the target projection result is determined, the poses of each surgical instrument in the corresponding candidate pose combination are also determined; once the poses of each surgical instrument are determined, the image portion affected by the surgical instruments in the preoperative 3D image is determined, and this image portion is the observation area image corresponding to the current observation area. In this way, the observation area image corresponding to the current observation area and the poses of each surgical instrument in the observation area image can be displayed; based on the observation area image and the poses of each surgical instrument in the observation area image, users can improve the accuracy of their interpretation of intraoperative videos, the accuracy of their analysis of surgical areas, and the accuracy of their judgment of surgical operations.

[0086] For example, the entry point of the first surgical instrument is denoted as P_A, and the entry point of the second surgical instrument is denoted as P_B. Both P_A and P_B represent the coordinate system of the body or the three-dimensional coordinate system of the preoperative CT scan. The working length of the first surgical instrument in vivo is denoted as S_A, and the working length of the second surgical instrument in vivo is denoted as S_B.

[0087] The first surgical instrument includes three markings, namely marking A_1, marking A_2, and marking A_3; the second surgical instrument includes three markings, namely marking B_1, marking B_2, and marking B_3.

[0088] Three markers for the first surgical instrument were identified from the intraoperative video: marker a_1, marker a_2, and marker a_3, where a_1 = (x_A1, y_A1), a_2 = (x_A2, y_A2), and a_3 = (x_A3, y_A3). Three more markers for the first surgical instrument were identified from the intraoperative video: marker b_1, marker b_2, and marker b_3, where b_1 = (x_B1, y_B1), b_2 = (x_B2, y_B2), and b_3 = (x_B3, y_B3).

[0089] Since different markers can be distinguished by color, width, arrangement order, or encoding method, it can be determined that markers a_1, a_2, and a_3 in the image correspond to markers A_1, A_2, and A_3 on the first surgical instrument, respectively, and markers b_1, b_2, and b_3 in the image correspond to markers B_1, B_2, and B_3 on the second surgical instrument, respectively. This correspondence is the basis for matching two-dimensional image information with three-dimensional spatial information.

[0090] In the body coordinate system, the virtual access points P_A and P_B of the first and second surgical instruments, as well as their in vivo working lengths S_A and S_B, are already determined. Therefore, the marking positions of the two surgical instruments within the body are spatially constrained. For the first surgical instrument, its markings A_1, A_2, and A_3 should be located within the instrument space originating from the virtual access point P_A and consistent with the in vivo working length S_A; for the second surgical instrument, its markings B_1, B_2, and B_3 should be located within the instrument space originating from the virtual access point P_B and consistent with the in vivo working length S_B.

[0091] Based on the aforementioned spatial constraints, candidate pose combinations of two surgical instruments in the body coordinate system are generated. Each candidate pose combination includes the candidate 3D positions of markers A_1, A_2, and A_3 of the first surgical instrument in the body coordinate system, and the candidate 3D positions of markers B_1, B_2, and B_3 of the enemy surgical instrument in the body coordinate system. Subsequently, a transformation matrix is ​​constructed using the imaging parameters of the thoracoscopic camera, and these candidate 3D markers are projected onto the target image plane to obtain the corresponding projection results.

[0092] The projection results are compared with the coordinates of the markers identified in the intraoperative video. Specifically, the positional differences between the projection results and markers a_1, a_2, a_3, b_1, b_2, and b_3 are compared. Furthermore, the alignment direction of the markers, the distance between instruments, the angle between instruments, and their positional relationship relative to the center of the image are compared to the intraoperative video observations. When a projection result has the highest matching degree with the intraoperative video, the candidate pose combination corresponding to that projection result is used as the matching result for the current moment.

[0093] After obtaining the matching result at the current moment, the position of the current observation area in the body coordinate system is determined. This current observation area can be represented as a local region in the preoperative 3D image, indicating approximately which spatial range within the patient's body the intraoperative video corresponds to.

[0094] This step allows us to obtain the spatial position of the current observation area in the preoperative 3D image coordinate system. This spatial position is taken as the observation area image and denoted as R_t, where t represents the current time point. R_t represents a local spatial range in the preoperative 3D image corresponding to the current thoracoscopic view. The processor can also simultaneously obtain the current observation direction V_t, as well as the relative positional relationship between the observation area image R_t and the two surgical instruments.

[0095] Figure 4 This is a schematic diagram of the structure of the observation area image determination device provided in an embodiment of the present invention. Figure 4 As shown, the image determination device for the observation area includes: The acquisition module 310 is used to acquire, in response to a predetermined trigger operation, the observation direction of the endoscope, the preoperative three-dimensional image of the target object, and the virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image. The length module 320 is used to determine the in vivo working length of each of the surgical instruments, the in vivo working length being determined based on the skin contact line of the surgical instrument and the instrument model, the instrument model including multiple markers and the length of each marker to the end of the surgical instrument; The determination module 330 is used to determine the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image, based on the virtual access point corresponding to each surgical instrument, the in vivo working length, the instrument model, and the combination of in vivo markers in the intraoperative video. The display module 340 is used to display the image of the observation area and the pose of each of the surgical instruments in the image of the observation area from the observation direction.

[0096] The technical solution provided by this invention predetermines the instrument models of each surgical instrument and determines the in vivo working length of the surgical instrument based on the skin contact line and the instrument model. This achieves the technical effect of accurately determining the in vivo working length of the surgical instrument without relying on complex sensors or electromagnetic positioning systems. By using the virtual approach points corresponding to each surgical instrument in the preoperative 3D image, the intraoperative video, and the instrument models and in vivo working lengths of each surgical instrument, the current spatial pose information of each surgical instrument during the operation is uniformly mapped to the preoperative 3D image coordinate system, thereby determining the observation area image affected by the surgical instrument, i.e., the observation area image corresponding to the current observation area. Along the observation direction, the observation area image and the pose of each surgical instrument in the observation area image are displayed, realizing real-time or near real-time display of the observation area image and the pose of each surgical instrument in the observation area image. This provides a unified and quantifiable spatial reference benchmark for intraoperative video understanding, operation area analysis, and instrument behavior analysis, which is conducive to improving the intelligence level of surgical navigation, operational safety, and user surgical experience.

[0097] In one embodiment, the display module 340 is used for: Along the observation direction, the predetermined anatomical structures in the image of the observation area and the poses of each of the surgical instruments are displayed.

[0098] In one embodiment, the length module 320 is used for: For each of the surgical instruments, a target marker corresponding to the surgical instrument is determined in the in vitro video, and the in vivo working length corresponding to the target marker is determined according to the instrument model; The target mark is the mark corresponding to the skin contact line.

[0099] In one embodiment, the tag combination includes at least two ring structures distributed along the axial direction, and the attribute data of different ring structures are different.

[0100] In one embodiment, the determining module 330 includes: A state generation unit is used to generate multiple target candidate poses for each of the surgical instruments based on the virtual access point and the in vivo working length. A combination unit is used to cross-combine the target candidate postures of each surgical instrument to obtain multiple candidate posture combinations; A projection unit is used to project each of the candidate pose combinations onto a target plane to obtain multiple projection results, wherein the target plane is the plane where the intraoperative video is located; The matching unit is used to determine the target projection result that matches the intraoperative video based on the distance between markers, the direction of marker arrangement, the distance between instruments, the angle between instruments, and the positional relationship between each surgical instrument and the image center. The determining unit is used to determine, based on the candidate pose combination corresponding to the target projection result, the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image.

[0101] In one embodiment, the state generation unit is used for: For each surgical instrument, multiple candidate postures are generated based on the virtual access point and the in vivo working length. Based on the limiting angle of the surgical instrument, the multiple candidate postures are filtered to obtain multiple target candidate postures.

[0102] In one embodiment, the body part corresponding to the preoperative three-dimensional image is the chest; The predetermined anatomical structures are part or all of the lung lobe, lung segment, target lesion, pulmonary artery, pulmonary vein, bronchus, pleura, chest wall, and hilar region.

[0103] The observation area image determination device provided in the embodiments of the present invention can execute the observation area image determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0104] It is worth noting that the various units and modules included in the above-mentioned observation area image determination device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0105] In some embodiments, the observation area image determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of the observation area image determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the observation area image determination method by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs for implementing the observation area image determination method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] This invention provides a computer-readable storage medium storing computer instructions for causing a processor to execute a method for determining an image of an observation region, including: In response to a predetermined trigger operation, the endoscope's observation direction, the preoperative three-dimensional image of the target object, and the virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image are acquired. For each of the surgical instruments, the in vivo working length of the surgical instrument is determined. The in vivo working length is determined based on the skin contact line of the surgical instrument and the instrument model, which includes multiple marks and the length of each mark to the end of the surgical instrument. Based on the virtual access point corresponding to each surgical instrument, the in vivo working length, the instrument model, and the combination of in vivo markers in the intraoperative video, the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image are determined. The image of the observation area and the pose of each surgical instrument in the image of the observation area are displayed from the observation direction.

[0109] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0112] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0113] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0114] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the observation area image determination method according to any embodiment of the invention.

[0115] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0117] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining an observation area image, characterized in that, A processor applied to a thoracoscopic device, the method comprising: In response to a predetermined trigger operation, the endoscope's observation direction, the preoperative three-dimensional image of the target object, and the virtual access points corresponding to each surgical instrument in the preoperative three-dimensional image are acquired. For each of the surgical instruments, the in vivo working length of the surgical instrument is determined. The in vivo working length is determined based on the skin contact line of the surgical instrument and the instrument model, which includes multiple marks and the length of each mark to the end of the surgical instrument. Based on the virtual access point corresponding to each surgical instrument, the in vivo working length, the instrument model, and the combination of in vivo markers in the intraoperative video, the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image are determined. The image of the observation area and the pose of each surgical instrument in the image of the observation area are displayed from the observation direction.

2. The method according to claim 1, characterized in that, The process of displaying the image of the observation area and the poses of each surgical instrument in the image of the observation area includes: Along the observation direction, the predetermined anatomical structures in the image of the observation area and the poses of each of the surgical instruments are displayed.

3. The method according to claim 1, wherein determining the in-body working length of each surgical instrument comprises: For each of the surgical instruments, a target marker corresponding to the surgical instrument is determined in the in vitro video, and the in vivo working length corresponding to the target marker is determined according to the instrument model; The target mark is the mark corresponding to the skin contact line.

4. The method according to claim 1 or 3, characterized in that, The marker combination includes at least two ring structures distributed along the axial direction, and the attribute data of different ring structures are different.

5. The method according to claim 1, wherein determining the observation area image corresponding to the current observation area in the preoperative three-dimensional image and the pose of each surgical instrument in the observation area image based on the target data combination of each surgical instrument comprises: For each of the surgical instruments, multiple target candidate poses are generated based on the virtual access point and the in vivo working length; The target candidate postures of each surgical instrument are cross-combined to obtain multiple candidate posture combinations; Each of the candidate pose combinations is projected onto a target plane to obtain multiple projection results, where the target plane is the plane containing the intraoperative video. Based on the distance between markers, the direction of marker arrangement, the distance between instruments, the angle between instruments, and the positional relationship between each surgical instrument and the image center, the target projection result matching the intraoperative video is determined; Based on the candidate pose combination corresponding to the target projection result, the observation area image corresponding to the current observation area in the preoperative three-dimensional image is determined, as well as the pose of each surgical instrument in the observation area image.

6. The method according to claim 5, wherein for each of the surgical instruments, generating multiple target candidate poses based on the virtual access point and the in vivo working length for each of the surgical instruments includes: For each of the surgical instruments, multiple candidate postures are generated based on the virtual access point and the working length inside the body. Based on the limiting angle of the surgical instrument, the multiple candidate postures are filtered to obtain multiple target candidate postures.

7. The method according to claim 2, The body part corresponding to the preoperative 3D image is the chest; The predetermined anatomical structures are part or all of the lung lobe, lung segment, target lesion, pulmonary artery, pulmonary vein, bronchus, pleura, chest wall, and hilar region.

8. A thoracoscopy device, characterized in that, include: A first surgical instrument and a second surgical instrument, wherein portions of the first surgical instrument and the second surgical instrument extend into the body of the target object to perform surgical operations, and include markings distributed along the axial direction, wherein the lengths of the different markings to the ends of the surgical instruments are different and known. Endoscopes are used to provide intraoperative video. A processor for executing the observation area image determination method according to any one of claims 1-7.

9. An image determination device for an observation area, characterized in that, A processor located in a thoracoscopic device, the device comprising: The acquisition module is used to acquire the observation direction of the endoscope, the preoperative three-dimensional image of the target object, and the virtual access point corresponding to each surgical instrument in the preoperative three-dimensional image in response to a predetermined trigger operation. A length module is used to determine the in vivo working length of each of the surgical instruments, the in vivo working length being determined based on the skin contact line of the surgical instrument and an instrument model, the instrument model including multiple markers and the length of each marker to the end of the surgical instrument; The determination module is used to determine the observation area image corresponding to the current observation area in the preoperative three-dimensional image, and the pose of each surgical instrument in the observation area image, based on the virtual access point corresponding to each surgical instrument, the in vivo working length, the instrument model, and the combination of in vivo markers in the intraoperative video. The display module is used to display the image of the observation area and the pose of each surgical instrument in the image of the observation area from the observation direction.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the observation area image determination method according to any one of claims 1-7.

Images