Data processing method, device, and system for surgical assistance

Abstract

The present invention provides a data processing method, apparatus, and system for surgical assistance. Among them, the data processing system for surgical assistance includes: a server, a bronchoscope, and a mixed reality display device; the server can communicate directly or indirectly with the mixed reality display device and the bronchoscope; the server is configured to: send intraoperative data and virtual data to the mixed reality display device; the mixed reality display device is configured to: acquire and display real-scene images of the surgical process; the displayed real-scene images are adapted to the real scenes observed by medical staff during the surgical process; based on the virtual data, display the appearance model of the virtual bronchial tree; display the real-time endoscopic images.

Description

Technical Field

[0001] The present invention relates to the medical field, and in particular to a data processing method, device, and system for surgical assistance. Background Art

[0002] With the development of medical device technology and the progress of imaging detection technology, especially the increasing popularity of high-resolution CT, the technology for detecting lung lesions has become increasingly mature.

[0003] Currently, whether during a surgical operation performed by a surgeon or during the process of an expert guiding a surgical operation locally or remotely, relevant information can only be obtained by viewing the information on a display screen. At this time, it is not convenient for doctors or experts to observe various information.

[0004] Moreover, whether on-site or remotely, during the process of guiding a surgical operation, the display screen of an expert usually only shows the video image of the surgical scene. In this way, the content displayed on the display screen is only the on-site video image, which cannot accurately reflect the specific operation status of the surgery and is not convenient for accurately guiding the operation of doctors. Summary of the Invention

[0005] The present invention provides a data processing method, device, and system for surgical assistance to solve problems such as inconvenience in observation.

[0006] According to a first aspect of the present invention, there is provided a data processing system for surgical assistance, including: a server, a bronchoscope, and a mixed reality display device; the server can communicate directly or indirectly with the mixed reality display device and the bronchoscope;

[0007] The server is used for:

[0008] Obtaining virtual data, where the virtual data includes data of a virtual bronchial tree of a human body;

[0009] Obtaining intraoperative data, where the intraoperative data includes: during the surgical process, real-time endoscopic images collected after the bronchoscope enters the human body;

[0010] Sending the intraoperative data and the virtual data to the mixed reality display device;

[0011] The mixed reality display device is used for:

[0012] Obtaining and displaying real-world scene images of the surgical process; the real-world scene images are obtained by an image acquisition device for image acquisition of the surgical process; the displayed real-world scene images are adapted to the real scene observed by the medical staff during the surgical process;

[0013] Based on the virtual data, displaying an appearance model of the virtual bronchial tree;

[0014] Display the real-time endoscopic image.

[0015] According to a second aspect of the present invention, there is provided a data processing method for surgical assistance, including:

[0016] Receiving virtual data and real-time endoscopic images from a server; the virtual data includes data of a virtual bronchial tree of a human body, and the real-time endoscopic image is collected after a bronchoscope enters the human body during a surgical procedure;

[0017] Obtaining and displaying, on a mixed reality display device, a real-scene image of a surgical procedure; the real-scene image is obtained by an image acquisition device collecting images of the surgical procedure; the displayed real-scene image is adapted to the real scene observed by medical staff during the surgical procedure;

[0018] Based on the virtual data, displaying, on the mixed reality display device, an appearance model of the virtual bronchial tree;

[0019] Displaying the real-time endoscopic image on the mixed reality display device.

[0020] According to a third aspect of the present invention, there is provided a data processing device for surgical assistance, including:

[0021] A receiving module for:

[0022] Receiving virtual data and real-time endoscopic images from a server; the virtual data includes data of a virtual bronchial tree of a human body, and the real-time endoscopic image is collected after a bronchoscope enters the human body during a surgical procedure;

[0023] A display module for:

[0024] Obtaining and displaying, on a mixed reality display device, a real-scene image of a surgical procedure; the real-scene image is obtained by an image acquisition device collecting images of the surgical procedure; the displayed real-scene image is adapted to the real scene observed by medical staff during the surgical procedure;

[0025] Based on the virtual data, displaying, on the mixed reality display device, an appearance model of the virtual bronchial tree;

[0026] Displaying the real-time endoscopic image on the mixed reality display device.

[0027] According to a fourth aspect of the present invention, there is provided a data processing method for surgical assistance, which is applied to a server, and the data processing method includes:

[0028] Obtain CT data of a human body, and based on the CT data, form virtual data, where the virtual data includes data of a virtual bronchial tree of the human body;

[0029] Obtain intraoperative data, where the intraoperative data includes: real-time endoscopic images collected after the bronchoscope enters the human body during the surgical procedure;

[0030] Obtain a real-scene image of the surgical procedure; the real-scene image is obtained by an image acquisition device collecting images of the surgical procedure;

[0031] Send the intraoperative data, the virtual data, and the real-scene image to a mixed reality display device, so that: the mixed reality display device displays the real-scene image of the surgical procedure, and the displayed real-scene image is adapted to the real scene observed by the medical staff during the surgical procedure; based on the virtual data, display an appearance model of the virtual bronchial tree; display the real-time endoscopic image.

[0032] According to a fifth aspect of the present invention, there is provided a data processing device for surgical assistance, which is applied to a server and includes:

[0033] An acquisition module, configured to:

[0034] Obtain virtual data, where the virtual data includes data of a virtual bronchial tree of a human body;

[0035] Obtain intraoperative data, where the intraoperative data includes: real-time endoscopic images collected after the bronchoscope enters the human body during the surgical procedure;

[0036] Obtain a real-scene image of the surgical procedure; the real-scene image is obtained by an image acquisition device collecting images of the surgical procedure;

[0037] A sending module, configured to:

[0038] Send the intraoperative data, the virtual data, and the real-scene image to a mixed reality display device, so that: the mixed reality display device displays the real-scene image of the surgical procedure, and the displayed real-scene image is adapted to the real scene observed by the medical staff during the surgical procedure; based on the virtual data, display an appearance model of the virtual bronchial tree, and: display the real-time endoscopic image.

[0039] According to a sixth aspect of the present invention, there is provided an electronic device, including a processor and a memory,

[0040] The memory is used to store code;

[0041] The processor is configured to execute the code in the memory to implement the method according to the second aspect or the fourth aspect.

[0042] According to the seventh aspect of the present invention, there is provided a storage medium having a computer program stored thereon, and when the program is executed by a processor, the method of the second aspect or the fourth aspect is implemented.

[0043] In the data processing method, device, and system for surgical assistance provided by the present invention, the mixed reality display device can not only display real-scene images, but also display the appearance model of the virtual bronchial tree and real-time endoscopic images, giving full play to the advantages of the mixed reality display device, realizing the combined display of real elements and virtual elements, enriching the diversity of the displayed data, and providing a more comprehensive and sufficient basis for surgical operations or surgical guidance. At the same time, the present invention realizes the display of information based on the mixed reality display device, facilitating doctors or experts to observe the information, and also facilitating the combined display of diverse information, enabling doctors or experts to observe various diverse information, and further improving the realism of the display result and the user experience.

[0044] In combination with the bronchoscope, the real-time endoscopic image can reflect the movement position, state, etc. of the bronchoscope, which is closely related to the operations during the surgical process, and thus can reflect the specific operating conditions of the surgery, providing an accurate and sufficient basis for surgical operations or surgical guidance, and facilitating the accurate implementation of surgical operations (for example, when the surgeon uses the mixed reality display device) or accurate guidance of the surgery (for example, when the expert guiding the surgery uses the mixed reality display device).

[0045] In addition, since the mixed reality display device can communicate with the server and the display content of the mixed reality display device is derived from the processing result of the server, in this solution, even if the expert guiding the surgery uses the mixed reality display device remotely, he can also observe information such as the appearance model of the virtual bronchial tree, real-time endoscopic images, and real-scene images, which helps to accurately guide the doctor's operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0047] Figure 1 It is a schematic structural diagram of an exemplary data processing system for surgical assistance in an embodiment of the present invention;

[0048] Figure 2 It is a schematic structural diagram of another exemplary data processing system for surgical assistance in an embodiment of the present invention;

[0049] Figure 3 It is a schematic structural diagram of another exemplary data processing system for surgical assistance in an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of the display effect of an exemplary mixed reality display device in an embodiment of the present invention;

[0051] Figure 5 It is a schematic diagram of the display effect of another exemplary mixed reality display device in an embodiment of the present invention;

[0052] Figure 6 It is a schematic diagram of the broadcasting principle of an exemplary broadcasting device in an embodiment of the present invention;

[0053] Figure 7 It is a schematic flowchart of an exemplary data processing method for surgical assistance in an embodiment of the present invention;

[0054] Figure 8 It is a schematic diagram of program modules of an exemplary data processing device for surgical assistance in an embodiment of the present invention;

[0055] Figure 9 It is a schematic flowchart of another exemplary data processing method for surgical assistance in an embodiment of the present invention;

[0056] Figure 10 It is a schematic diagram of program modules of another exemplary data processing device for surgical assistance in an embodiment of the present invention;

[0057] Figure 11 It is a schematic structural diagram of an exemplary electronic device in an embodiment of the present invention. Detailed implementation manners

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0059] In the description of the present invention, the claims, and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0060] The technical solution of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0061] Please refer to Figure 1 , in the embodiment of the present invention, a data processing system for surgical assistance includes: a server 12, a bronchoscope 13, and a mixed reality display device 11;

[0062] The server 12 can communicate directly or indirectly with the mixed reality display device 11 and the bronchoscope 13; for example, the server 12 can communicate with the mixed reality display device 11 and the bronchoscope 13 based on at least one of the Internet, local area network, Internet of Things, etc., or can also communicate directly with the mixed reality display device 11 and the bronchoscope 13.

[0063] The server 12 herein can refer to any device or combination of devices having data processing capabilities and external communication capabilities. In addition, in the data processing system for surgical assistance, the number of servers 12 can be one or more.

[0064] The bronchoscope 13 herein can include an image acquisition unit. The bronchoscope 13 can be understood as a device or combination of devices that can use the image acquisition unit to acquire real-time endoscopic images after entering the trachea of the human body. Among them, the bronchoscope 13 can also include a bending tube (such as an active bending tube and / or a passive bending tube), and the image acquisition unit can be disposed at one end of the bending tube. In addition, no matter what kind of bronchoscope is used, it does not deviate from the scope of the embodiment of the present invention.

[0065] The mixed reality display device 11 therein can be understood as a device that displays information in a mixed reality manner. The mixed reality therein, namely MR, specifically refers to Mixed Reality, which can be understood as a way of mixing and displaying real elements and virtual elements. In addition, the mixed reality display device 11 can also be described as an MR device. In a specific example, the mixed reality display device 11 can be a head-mounted mixed reality display. After the user wears it, various required information can be displayed within the user's visual field.

[0066] The user wearing the mixed reality display device 11 can be the surgeon performing the operation or the expert guiding the operation. The solution of the embodiment of the present invention can effectively take into account these two application scenarios.

[0067] Furthermore, regardless of which application scenario, because the display of information is realized based on the mixed reality display device, it is convenient for doctors or experts to observe the information, and it is also convenient to take into account the combined display of diverse information, enabling doctors or experts to observe various diverse information simultaneously. In addition, it can also improve the realism of the display result and enhance the user experience.

[0068] Figure 1 In the illustrated embodiment, the server 12 is used for:

[0069] Obtain virtual data;

[0070] Specifically, the CT data of the human body can be obtained first, and based on the CT data, virtual data is formed. It is also possible to directly receive the virtual data from other devices.

[0071] The human body therein can be understood as the human body currently being the surgical object;

[0072] The CT data can refer to the data obtained by performing a CT scan on the human body. Specifically, it can be the data obtained by performing a CT scan on the lungs. The CT data can mainly reflect the morphology of various tracheas in the lungs (which can also be understood as the morphology of the bronchial tree). Further, the CT data can also reflect the morphology of blood vessels, muscles, etc. At the same time, the device performing the CT scan can be any device.

[0073] The CT data can include CT images. For example, through CT scanning, a set of CT images can be formed for each position among multiple positions of the bronchial tree. The number of CT images in each group can be one or multiple.

[0074] The virtual data therein can include the data of the virtual bronchial tree and any other data existing relying on the virtual bronchial tree.

[0075] The virtual bronchial tree therein can be understood as a virtual model used to simulate the real bronchial tree in the human body. In one example, on the basis of the virtual bronchial tree, virtual physiological structures (such as virtual blood vessels, virtual muscles, etc.) can also be used to simulate physiological structures such as blood vessels and muscles near the bronchial tree.

[0076] Correspondingly, the data of the virtual bronchial tree can refer to any information describing the virtual bronchial tree. Specifically, when constructing the virtual bronchial tree, a point cloud of the virtual bronchial tree can be formed in the virtual coordinate system. In addition, any method of constructing a virtual bronchial tree based on CT data in the art can be used as an optional implementation manner of "forming virtual data based on the CT data".

[0077] In addition, virtual physiological structures such as virtual blood vessels and virtual muscles can also be constructed by referring to a similar method, and the virtual data can also include the data of the virtual physiological structures.

[0078] The virtual data can also include lesion location data, which characterizes the relative position between the virtual bronchial tree and the lesion area. Furthermore, the relative position between the real bronchial tree and the real lesion site can be characterized by the relative position between the virtual bronchial tree and the lesion area. For example, after constructing the virtual bronchial tree in the virtual coordinate system, the point cloud coordinates of the virtual bronchial tree in the virtual coordinate system can be obtained. On this basis, the position data of the lesion area in the virtual coordinate system can be used as the lesion location data. The lesion location data can be manually marked or automatically identified.

[0079] The virtual data can also include the data of the target path, which simulates the path of the bronchoscope reaching the lesion after entering the human body; the target path can be planned based on the lesion location data and the virtual bronchial tree. Any path planning method in the art can be used as an optional implementation manner of the embodiments of the present invention.

[0080] For example: The server can import the CT data of the patient's body into the lung navigation software, segment and reconstruct the three-dimensional models of the chest-related organs according to the CT images therein to obtain the virtual bronchial tree. Then, the lesion area can be selected manually or automatically for the virtual bronchial tree to form the lesion location data, and then the optimal path from the main carina of the virtual bronchial tree along the airway centerline to the lesion area can be planned to obtain the target path; on this basis, the server can output the data of the generated virtual bronchial tree, the lesion location data, and the data of the target path to the mixed reality display device.

[0081] Figure 1 In the shown embodiment, the server 12 is further configured to:

[0082] Obtain intraoperative data;

[0083] The intraoperative data therein can refer to any data collected or generated during the operation. Specifically, the intraoperative data may include: during the surgical process, real-time endoscopic images collected after the bronchoscope enters the human body. In addition to the real-time endoscopic images, the intraoperative data may also include, for example, information generated during the operation of the ventilator.

[0084] The real-time endoscopic images therein can be understood as the images collected by the image acquisition part of the bronchoscope.

[0085] Figure 1 In the illustrated embodiment, the server 12 is further configured to: send the intraoperative data and the virtual data to the mixed reality display device.

[0086] Figure 1 In the illustrated embodiment, the mixed reality display device 11 is configured to: display the real-time endoscopic images.

[0087] Specifically, within the field of view of a user wearing the mixed reality display device 11, the mixed reality display device 11 can display the real-time endoscopic images through an interface.

[0088] Figure 1 In the illustrated embodiment, the mixed reality display device 11 is further configured to: display the appearance model of the virtual bronchial tree based on the virtual data;

[0089] The appearance model therein can be understood as: the model formed by the outer surface of the virtual bronchial tree, or it can also be understood as: this appearance model satisfies that through the display of this appearance model, the content inside the virtual bronchial tree is not displayed (i.e., cannot be observed).

[0090] Figure 1 In the illustrated embodiment, the mixed reality display device 11 is further configured to: acquire and display the real-world scene images of the surgical process. The real-world scene images are obtained by an image acquisition device for image acquisition of the surgical process;

[0091] Among them, the real-world scene images are obtained by an image acquisition device for image acquisition of the surgical process; the displayed real-world scene images are adapted to the real scene observed by the medical staff (such as the surgeon, or also non-surgeon doctors or nurses) during the surgical process. Among them, some or all of the content, acquisition perspective, display size, etc. of the real-world scene images may be the same as or similar to the real scene observed by the surgeon. For example: the image acquisition device is integrated into the mixed reality display device 11 worn by the surgeon, and its position, angle, focal length, etc. can be adapted to the eyes of the surgeon.

[0092] The image acquisition device can be integrated into the mixed reality display device 11 worn by the surgeon. Furthermore, during the operation, the image actually seen by the surgeon can be acquired, which is the real - world scene image. In addition, the image acquisition device can also be independent of the mixed reality display device 11.

[0093] The real - world scene image can be forwarded by the server 12. For example, the server 12 is also used to: acquire the real - world scene image of the operation process (such as receiving the real - world scene image from the surgeon's mixed reality display device), and send the real - world scene image to each mixed reality display device. In addition, the real - world scene image can also be directly obtained by the mixed reality display device 11 from the image acquisition device. For example, if the image acquisition device is integrated into the mixed reality display device 11 worn by the surgeon, the mixed reality display device 11 worn by the surgeon can directly obtain the real - world scene image from it without receiving it from the server.

[0094] It can be seen that in Figure 1 the illustrated embodiment, the mixed reality display device can not only display the real - world scene image, but also display the appearance model of the virtual bronchial tree and the real - time endoscopic image, giving full play to the advantages of the mixed reality display device, realizing the combined display of real elements and virtual elements, enriching the diversity of the displayed data, and providing a more comprehensive and sufficient basis for surgical operations or surgical guidance. At the same time, the present invention realizes the display of information based on the mixed reality display device, which is convenient for doctors or experts to observe the information, and is also convenient for taking into account the combined display of diverse information, enabling doctors or experts to observe various diverse information, and can also improve the realism of the display result and enhance the user experience.

[0095] In combination with the bronchoscope, the real - time endoscopic image can reflect the movement position, state, etc. of the bronchoscope, which is closely related to the operations during the operation, and thus can reflect the specific operation conditions of the operation, providing an accurate and sufficient basis for surgical operations or surgical guidance, and facilitating the accurate implementation of surgical operations (such as when the surgeon uses the mixed reality display device) or accurate surgical guidance (such as when the expert guiding the operation uses the mixed reality display device).

[0096] In addition, since the mixed reality display device can communicate with the server, and the display content of the mixed reality display device is derived from the processing result of the server, in this solution, even if the expert guiding the operation uses the mixed reality display device remotely, he can also observe information such as the appearance model of the virtual bronchial tree, the real - time endoscopic image, and the real - world scene image, which helps to accurately guide the doctor's operation.

[0097] In one implementation, the server 12 or the mixed reality display device 11 is also used to: determine the real - time position of the bronchoscope in the human body during the operation of the human body;

[0098] Among them, the method for determining the real-time position can be arbitrarily changed according to requirements. For example, it can be determined based on the real-time endoscopic image and the sliced images of some or all positions in the virtual bronchial tree by matching the real-time endoscopic image and the sliced images. For another example, it can also be determined based on the detection of the pose of the bronchoscope by detection devices such as magnetic sensors, and then determined based on the detection results. For still another example, it can also be determined based on the target path and the actual movement path of the bronchoscope by matching the paths. No matter which method is adopted, it does not deviate from the scope of the embodiments of the present invention.

[0099] In one implementation manner, the mixed reality display device 11 is further configured to: based on the virtual data and the real-time position, display an internal navigation image of the target position inside the virtual bronchial tree;

[0100] Among them, the target position is adapted to the real-time position, and the internal navigation image refers to an image obtained by a virtual camera observing the inside of the virtual model at the target position. In addition, the viewing angle of the virtual camera can be designed to imitate the viewing angle of the bronchoscope.

[0101] In one implementation manner, the server 12 is further configured to: send at least part of the CT data of the human body to the mixed reality display device; the at least part of the CT data includes CT images;

[0102] The mixed reality display device 11 is further configured to:

[0103] Based on the real-time position, display the CT image of the real-time position.

[0104] Specifically, the CT image can be displayed by using a display window. A set of CT images (i.e., multiple CT images) of the real-time position can be displayed in one display window or can be respectively displayed in different display windows;

[0105] For example, when the bronchoscope reaches a certain position, the displayed CT image will also be switched to the CT image of that position accordingly. Furthermore, the displayed CT image can be adapted and changed as the position of the bronchoscope changes. Through the mixed reality display device 11, the user can simultaneously learn the CT image, the real-time endoscopic image, the appearance model, etc. of that position, so as to be able to display and enable the user to learn diverse data at the same time, and give full play to the advantages of the mixed reality display device.

[0106] Furthermore, through the above implementation manners, the CT image can be displayed in the display content of the mixed reality display device, so that the user can simultaneously learn the CT image, as well as information such as the real-time endoscopic image, the internal navigation image, and the appearance model of the virtual bronchial tree, providing sufficient information for the user as a basis for surgical guidance or surgical operation.

[0107] In one implementation, the mixed reality display device 11 is further configured to:

[0108] Display a three-dimensional perspective image of the virtual bronchial tree.

[0109] The three-dimensional perspective image can be understood as: for the virtual bronchial tree, a virtual camera outside the virtual bronchial tree observes the virtual bronchial tree (and can also observe virtual physiological structures such as virtual muscles and virtual blood vessels outside the virtual bronchial tree) to form an image. And in this image, some or all of the point clouds in the virtual bronchial tree (and other virtual physiological structures) can be displayed as transparent points. In addition, in the three-dimensional perspective image, the point clouds can also be colored, and some point clouds of the virtual bronchial tree and virtual physiological structures can be filtered out.

[0110] Specifically, a display window can be used to display the three-dimensional perspective image.

[0111] Through the above implementation, a three-dimensional perspective image can be displayed in the display content of the mixed reality display device, providing sufficient information for the user as a basis for surgical guidance or surgical operation.

[0112] In one implementation, if the virtual data includes lesion location data, then:

[0113] The mixed reality display device 11 is further configured to:

[0114] When displaying the appearance model and / or three-dimensional perspective image of the virtual bronchial tree, based on the lesion location data, display the lesion area.

[0115] For example, the lesion area can be marked in the appearance model of the virtual bronchial tree with a display element of a specified color.

[0116] Through the above solution, the location of the lesion area can be displayed in the display content of the mixed reality display device, enabling the user to simultaneously obtain information such as the location of the lesion area, real-time endoscopic images, internal navigation images, and the appearance model of the virtual bronchial tree, providing sufficient information for the user as a basis for guidance or operation.

[0117] In one implementation, if the virtual data includes data of a target path, then:

[0118] The mixed reality display device 11 is further configured to:

[0119] When displaying the appearance model and / or three-dimensional perspective image of the virtual bronchial tree, display the target path therein.

[0120] For example, in the appearance model of the virtual bronchial tree, the target path can be displayed by using display lines of a specified color.

[0121] Among them, by displaying the target path, the user can learn the planned path, and then operate or specify the operation based on this, which can ensure that the operation result can accurately meet the surgical requirements.

[0122] In addition, in one example, while displaying the target path, the real-time position (i.e., the target position) can also be synchronously displayed. For example, the real-time position of the bronchoscope can be displayed by using display elements of a specific color and shape.

[0123] In one implementation, the server 12 can also be used to check and confirm that the acquired data is normal and valid.

[0124] In one implementation, the mixed reality display device 11 is further configured to:

[0125] Adjust the displayed content in response to the interaction information of the current user;

[0126] The interaction information is detected by an interaction device, and the interaction device is integrated into the mixed reality display device or is configured to be able to communicate with the mixed reality display device.

[0127] In a specific example, the adjustment may include at least one of the following:

[0128] Shrink the appearance model;

[0129] Enlarge the appearance model;

[0130] Rotate the appearance model;

[0131] Change the display position of the appearance model;

[0132] Close the display of the appearance model;

[0133] Open the display of the appearance model;

[0134] For example, for the appearance model of the virtual bronchial tree, the user (such as the surgeon or expert) can perform operations such as dragging, rotating, scaling, etc. on it through gestures. For example, the display position of the appearance model can be changed through a dragging gesture, the appearance model can be rotated through a rotating gesture, and the appearance model can be enlarged or shrunk through a scaling gesture.

[0135] In a specific example, the adjustment may include at least one of the following:

[0136] Shrink the display window;

[0137] Enlarge the display window;

[0138] Changing the position of the display window;

[0139] Opening of the display window;

[0140] Closing of the display window;

[0141] Wherein, the display window is a window for displaying a corresponding image, and the corresponding image is any one of the following: the real-time endoscopic image, the internal navigation image, the CT image in the CT data, and the three-dimensional perspective image of the virtual bronchial tree.

[0142] It can be seen that in the display content of the mixed reality display device 11, it may only display some or all of the real-time endoscopic image, the internal navigation image, the CT image in the CT data, the three-dimensional perspective image of the virtual bronchial tree, and the real-world scene image of the surgical process, and specifically what is displayed can be determined and changed based on the adjustment.

[0143] The interaction device may, for example, include at least one of the following:

[0144] A device for detecting user gestures. Correspondingly, the interaction information may include gesture information;

[0145] A device for detecting changes in the user's eyes; correspondingly, the interaction information may include eye change information; the eye changes may, for example, include changes in the position and orientation of the eyeballs, whether the eyes blink, etc.;

[0146] A device for detecting user voice commands. The corresponding interaction information may include voice commands.

[0147] Please refer to Figure 2 wherein the mixed reality display device 21 shown can be understood with reference to the mixed reality display device 11 in the embodiments shown and their various alternative embodiments, and the server 22 shown can be understood with reference to Figure 1 the server 11 in the embodiments shown and their various alternative embodiments, and the bronchoscope 23 shown can be understood with reference to Figure 1 the bronchoscope 13 in the embodiments shown and their various alternative embodiments. For repeated content, it will not be elaborated. Figure 1 In the embodiments shown, the number of the mixed reality display devices 21 is at least two; the mixed reality display devices 21 can be used by the surgeon performing the surgical operation and also by the expert guiding the surgery. Based on the functions of the mixed reality display devices 21, it can effectively balance the needs of both possibilities.

[0148] Figure 2

[0149] ​In one application scenario, the mixed reality display device can be used only by the operating surgeon. In another application scenario, the mixed reality display device can be used only by the expert guiding the operation. In yet another application scenario, the operating surgeon and the expert guiding the operation can use different mixed reality display devices respectively.

[0150] In one implementation, the content displayed by multiple said mixed reality display devices is synchronized. Among them, when the operating surgeon and the expert guiding the operation use different mixed reality display devices respectively, it can be ensured that the information viewed by the expert can be adapted to the information viewed by the operating surgeon, guaranteeing the accuracy of the guidance.

[0151] On this basis, if combined with the adjustment of the displayed content, then: the adjustment of the displayed content by one user can be synchronized to the mixed reality display devices of other users in a timely manner. Furthermore, it can be further facilitated to realize the sharing, communication and guidance of information among users.

[0152] For example: The expert can adjust and magnify the display window or the appearance model through the mixed reality display device. At this time, the operating surgeon's mixed reality display device can also observe the same adjustment result. Furthermore, the content of certain perspectives and positions can be shown to the operating surgeon, thus assisting the expert's guidance (for example). Conversely, the operating surgeon can also adjust and show the corresponding content to the expert, and different experts can also adjust and show the corresponding content to each other.

[0153] For another example, the user wearing the mixed reality display device can operate independently or collaboratively through interaction methods such as eye gaze, gestures, and voice, and share the adjustment results and the displayed content in real time.

[0154] In addition, each mixed reality display device can support the collaborative operation of multiple mixed reality display devices. For example, after a user rotates the appearance model through the mixed reality display device, another user can magnify the rotated appearance model through the mixed reality display device. The server can synchronously update the data of all mixed reality display devices, and doctors wearing other mixed reality display devices can all share various data in real time. During remote guidance, doctors can give instant feedback or guidance, improving the surgical efficiency.

[0155] In one implementation, the operation permissions of different users for the content in the adjustment are different. The operation permissions can be, for example, the operation permissions allowing the implementation of certain adjustment or adjustments, or can be, for example, the operation permissions prohibiting the implementation of certain adjustment or adjustments; for example, some users only have the operation permissions for the two adjustments of magnifying and shrinking the appearance model. For another example, some users do not have the operation permissions for magnifying and shrinking a certain display window.

[0156] In a further example, the operation authority of the expert and that of the operating surgeon can be different. In another example, the operation authorities of different doctors can be different. The different doctors can be, for example: doctors with different responsibilities, doctors with different qualifications, doctors with different identifiers (such as names, certificate codes, etc.), doctors from different departments, and so on.

[0157] When the operating surgeon uses the mixed reality display device, no additional personnel are required. The mixed reality display device will fuse, render, and display all the images in the field of view of the operating surgeon, and the operating surgeon does not need to turn his head to view the images of other devices.

[0158] Figure 2 In the illustrated embodiment, the data processing system for surgical assistance further includes: a CT scanning device 24. The CT data can be collected by the CT scanning device 24, and the server 22 can communicate directly or indirectly with the CT scanning device 24.

[0159] Figure 2 In the illustrated embodiment, the processing system for surgical assistance further includes an C-arm 28. The C-arm 28 can communicate directly or indirectly with the server. Furthermore, the C-arm 28 can scan the human body to obtain an X-ray scan result, and then, the X-ray scan result can be fed back to the server.

[0160] In one example, after the bronchoscope enters the human body, the server can also achieve the positioning of the real-time position based on the X-ray scan result. For example, after the bronchoscope enters the human body, the actual movement trajectory of the bronchoscope can be calculated, and then the actual movement trajectory can be registered with the X-ray scan result to locate the real-time position of the bronchoscope. In another example, the server can also use the X-ray scan result when constructing a virtual bronchial tree. In yet another example, the server can also display the X-ray scan result in the mixed reality display device.

[0161] Figure 2 In the illustrated embodiment, the data processing system for surgical assistance further includes: a relay device 25 and / or a recording device 26.

[0162] The recording device 26 is used to record the content displayed by the mixed reality display device; for example, it can be recorded as a video or other forms of data.

[0163] The relay device 25 is used to relay the content displayed by the mixed reality display device. For example, the displayed content can be relayed to other display devices (such as display screens, computers, etc.).

[0164] The relay device 25 and the recording device 26 can be configured to be able to communicate directly or indirectly with the server 22, and can also be configured to be able to communicate directly or indirectly with at least one of the mixed reality display devices 21.

[0165] In the above solutions, through live broadcast and recording, it is convenient to achieve postoperative observation and learning, review, and intraoperative observation and learning.

[0166] In one implementation, through the lung navigation software installed on the server, the data generated by the server (such as virtual data) can be imported into the mixed reality display device. After modifying the server IP address to the address of the current server, one of the mixed reality display devices can be connected to the computer. Then, the screen tracked by the mixed reality display device can be synchronously displayed on the computer. At this time, the computer can be regarded as a live broadcast device.

[0167] Figure 2 In the illustrated embodiment, the surgical assistance data processing system may further include: medical devices such as a ventilator 27. The server 22 may also be used to: obtain the working data generated when these medical devices work, and send the working data to the mixed reality display device 21;

[0168] Correspondingly, the mixed reality display device 21 can display the working data. Furthermore, based on these working data, more sufficient basis can be provided for the user's surgical operation or surgical guidance.

[0169] Please refer to Figure 3 , in which the mixed reality display device 31 shown can be understood with reference to the mixed reality display device 11 in the illustrated embodiment and its various alternative embodiments, and can also be understood with reference to Figure 1 the mixed reality display device 21 in the illustrated embodiment and its alternative embodiments. The server 32 shown can be understood with reference to the server 12 in the illustrated embodiment and its various alternative embodiments, and can also be understood with reference to Figure 2 the server 22 in the illustrated embodiment and its alternative embodiments. The live broadcast device 33 and the recording device 34 shown can be understood with reference to Figure 1 the live broadcast device 25 and the recording device 26 in the illustrated embodiment and its alternative embodiments. For repeated content, it will not be elaborated again. Figure 2 the live broadcast device 25 and the recording device 26 in the illustrated embodiment and its alternative embodiments. For repeated content, it will not be elaborated again. Figure 2 the live broadcast device 25 and the recording device 26 in the illustrated embodiment and its alternative embodiments. For repeated content, it will not be elaborated again.

[0170] Figure 3 In the illustrated embodiment, the server 32, the mixed reality display device 31, the live broadcast device 33, and the recording device 34 can communicate directly, or can communicate based on the router 35, or can access the network based on the router 35, so as to communicate with other devices.

[0171] In addition, the communication between the server 32 and the mixed reality display device 31 can be implemented based on the network communication unit. The network communication unit can be the router 35 or other network devices. Furthermore, through the network communication unit, data transmission can be achieved between the server and the mixed reality display device.

[0172] For example, data such as the virtual bronchial tree data, lesion location data, target path data generated by the server side, and images (real-time endoscopic images) and signal data of various devices collected can be sent to each mixed reality display device 31 through the network communication module. The adjustments generated by the interaction can also be returned to the server in real time, and then the data changes brought about by the adjustments can be synchronized to other mixed reality display devices 31, thereby realizing the real-time sharing of each mixed reality display device 31.

[0173] In a specific example, the server 32 can be a desktop computer with a relatively high hardware configuration and good system performance, which can process various data more quickly and efficiently and perform a large number of complex calculations. It can also be connected to other devices (such as a bronchoscope), collect information from other devices (such as a bronchoscope), and synchronously transmit it to the mixed reality display device; the mixed reality display device is a mixed reality head-mounted display, which can not only simply process the data received from the server, render and update the displayed content in real time, but also perform interactions and synchronize the adjustments brought about by the interactions to the server.

[0174] Figure 4 In the shown embodiment, a display result of a mixed reality display device 31 is provided. Among them, a three-dimensional perspective image of the virtual bronchial tree can be displayed using the display window 41, a real-time endoscopic image can be displayed using the display window 42, an internal navigation image can be displayed using the display window 43, a CT image can be displayed using the display window 44. At the same time, the appearance model of the virtual bronchial tree can be displayed within the specified area 45. In the background 46 behind each display window, appearance model, and button, the mixed reality display device 31 can provide a real surgical scene (i.e., a real scene image). The image acquisition device for collecting the real scene image can be integrated into the mixed reality display device 31 (such as the mixed reality display device 31 worn by the surgeon in charge), and its viewing angle can be adapted to the viewing angles of the eyes of the user wearing the mixed reality display device 31 (such as the surgeon in charge).

[0175] Figure 5 The shown embodiment can be understood as Figure 4 a concrete display of the shown display result, Figure 5 The shown perspective image can be understood as Figure 4 the shown three-dimensional perspective image, Figure 5 The shown bronchoscope screen can be understood as Figure 4 the shown real-time endoscopic image, Figure 5 The shown three-dimensional model internal navigation screen can be understood as Figure 4 the shown internal navigation image, Figure 5 The shown current position CT image can be understood as Figure 4 the shown CT image, Figure 5The bronchial tree shown can be understood as Figure 4 the appearance model shown, Figure 5 the content of the background part in can be understood as Figure 4 the background 46 shown.

[0176] During the operation, if the leading doctor wears a mixed reality display device, he can not only synchronously operate the bronchoscope and other surgical instruments, but also synchronously operate and adjust the display content of the mixed reality display device without additional personnel configuration, which can achieve the effect of facilitating operation. At the same time, the mixed reality display device will fuse and render all images and display them in the doctor's field of vision (for example, in the manner shown in Figure 4 ), and the doctor does not need to turn his head to watch the pictures of other devices. In addition, when the expert wears a mixed reality display device, if he displays the real scene image, he can observe the image observed by the leading doctor from a perspective similar to that of the leading doctor. When the expert wears a mixed reality display device, the real scene image may not be displayed either.

[0177] For the relay device, in some solutions, the content shown can be directly transmitted and then displayed after relay without sampling, compressing, etc.

[0178] In other solutions, since the content to be transmitted is large, it can be processed to a certain extent.

[0179] When the relay device relays the content shown by the mixed reality display device, it is specifically used for:

[0180] Sampling the display object (such as CT image, appearance model, three-dimensional perspective image, internal navigation image, real-time endoscope image, etc.) to obtain the sampled data;

[0181] Transmitting the sampled data;

[0182] After transmission, reconstructing the sampled data to obtain the reconstruction result.

[0183] The above three steps can be implemented by different device units of the relay device, for example, it can include:

[0184] A receiving and processing unit for sampling the display object to obtain the sampled data;

[0185] A transmission unit for transmitting the sampled data; for example, it can be a transmission line in a wired transmission mode, or a communication unit in a wireless transmission mode, or both a transmission line and a communication unit can be used;

[0186] A reconstruction processing unit for reconstructing the sampled data after transmission to obtain the reconstruction result.

[0187] When the relay device samples the display object to obtain the sampled data, it specifically is used for:

[0188] Input the display object into the corresponding first convolutional network, and use the convolutional processing of the corresponding first convolutional network to perform compressive sensing sampling on the display object to obtain the sampled data.

[0189] Since the contents and dimensions of multiple display objects are different, different types of convolutional processing can be respectively adopted. For example:

[0190] The convolutional processing implemented by the first convolutional network corresponding to the internal navigation image and real-time endoscopy image among multiple display objects is 2D convolution;

[0191] The convolutional processing implemented by the first convolutional network corresponding to the appearance model of the virtual bronchial tree among the multiple display objects is 1D convolution;

[0192] The convolutional processing implemented by the first convolutional network corresponding to the three-dimensional perspective image of the virtual bronchial tree among the multiple display objects is 1D convolution or 2D convolution;

[0193] The convolutional processing implemented by the first convolutional network corresponding to the CT image among the multiple display objects is 1D convolution.

[0194] For specific examples, please refer to Figure 6 , since different convolutional processing is implemented for different display objects, the sampling of the first convolutional network 61 can be understood as multi-modal compressive sensing sampling, thus focusing on multi-modal data: three-dimensional perspective images, CT images (such as 3D CT images formed by a group of CT images), simulated endoscopy images, and appearance models rendered by point clouds. They use convolutional sampling of different dimensions. Furthermore, in this example, it is not necessary to perform block sampling on the data, so there will be no block effect in reconstruction.

[0195] For example, please refer to the following formula:

[0196] Y = W φ X (1)

[0197] Formula 1 is a formula for compressive sensing sampling common in the prior art. Y represents the sampling result, and X represents the original data (i.e., the data of multiple display objects). Due to the size of the sampling matrix, it is necessary to cut X into fixed blocks for sampling and then merge them after reconstruction. However, when using the first convolutional network for convolution, after introducing convolutional sampling of different dimensions, the formula is as follows:

[0198]

[0199] N represents the dimension of X. Furthermore, the convolutional kernel can be selected according to the dimension of X. Because of the characteristics of convolution, it can process data with variable sizes, so there is no need for chunking and no block effect will occur.

[0200] In one implementation, when the relay device reconstructs the sampled data to obtain a reconstruction result, it is specifically used for:

[0201] Performing convolutional processing on the sampled data by using a second convolutional network to obtain the data after the first recovery;

[0202] Performing convolutional processing on the data after the first recovery by using a shallow convolutional network to obtain the data after the second recovery; the shallow convolutional network is subjected to knowledge distillation by a deep convolutional network during training;

[0203] Performing data enhancement on the data after the second recovery by using an attention network to obtain the reconstruction result of each display object.

[0204] Among them, the first convolutional network, the second convolutional network, the shallow convolutional network, the deep convolutional network, and the attention network can be trained together. Furthermore, the first convolutional network, the second convolutional network, the shallow convolutional network, the deep convolutional network, and the attention network can have the ability to accurately sample and reconstruct, ensuring that the relay result can accurately and clearly restore the actual content of each display object.

[0205] In specific examples, please refer to Figure 6 , the second convolutional network 62 can be a convolutional network with depthwise separable convolution, and its output result can be characterized as:

[0206] Among them, considering the huge amount of data required to be transmitted by the mixed reality display device, using an ordinary convolutional network will result in a huge amount of computation. Therefore, a convolutional network with depthwise separable convolution is used to solve this problem. Specifically, taking the 2D convolution in the network as an example, 32 2D convolutional kernels of 3×3×32 can be used, and its parameters are 3×3×32×32 = 9216.

[0207] By using depthwise separable convolution, 32 2D convolutional kernels of 3×3×32 can be separated into 1 2D convolutional kernel of 3×3×32 and 32 2D convolutional kernels of 1×1×32, and the total number of parameters is 1×3×3×32 + 32×1×1×32 = 1312 parameters. The number of parameters is reduced by 85%.

[0208] In one implementation, during training, the output data of the second convolutional network will be transmitted to the deep convolutional network and the shallow convolutional network respectively. When reconstructing the sampled data, the output data of the second convolutional network is only transmitted to the shallow convolutional network.

[0209] In a specific example, please refer to Figure 6 , where Deep CNN represents the deep convolutional network 64, Shallow CNN represents the shallow convolutional network 63, and a knowledge distillation mechanism can be used during network training.

[0210] Specifically, the deep convolutional network 34 can be composed of K (for example, 12) residual modules. Each residual module consists of a 3×3×32 convolution, batch normalization (BN), a rectified linear unit (ReLU), a 3×3×32 convolution, and a skip connection. The shallow convolutional network 63 is composed of K / 2 (for example, 6) residual modules. The loss function for knowledge distillation is:

[0211] L1 = KLdiv(F Deep (X0) - F Shallow (X0)) + ||X label - F Shallow (X0)||2 (3)

[0212] KLdiv represents the KL divergence, X0 is the initialization reconstruction of compressive sensing, X label is the label of the reconstructed data, F Deep (X0) is the output result of the deep convolutional network, F Shallow (X0) is the output result of the shallow convolutional network.

[0213] Among them, through knowledge distillation, the knowledge of the deep convolutional network 64 can be transferred to the shallow convolutional network 63 during training.

[0214] In one implementation, please refer to Figure 6 , the number of attention networks is two. The two attention networks are the first attention network 65 that uses the window method attention mechanism without displacement, and the second attention network 66 that uses the window method attention mechanism with displacement.

[0215] After the data after secondary recovery is processed by the first attention network 65 and the second attention network 66 in sequence, the reconstruction result of each object can be formed.

[0216] In a specific example, please refer to Figure 6, LN represents inter-layer regularization, using variance and expectation regularization, MLP represents a fully connected network, W-MSA (Window Attention) represents the attention mechanism of the window method, and SW-MSA (Shifted Window Attention) represents the attention mechanism of the window method with cyclic shift. The cyclic shift performs data augmentation on the data. Furthermore, when the first attention network 65 and the second attention network 66 are adopted, the attention network can provide better robustness.

[0217] In addition, an attention mechanism module is also used to capture long-range dependencies. The window method reduces the time required for subsequent attention mechanism calculations by dividing and combining the input, which is beneficial for real-time high-definition compressive sensing reconstruction and obtaining a high-definition restored reconstruction result.

[0218] The algorithm therein can be expressed by the formula:

[0219]

[0220] F attention represents Figure 6 the two attention networks with dashed lines in

[0221] X l+1 represents the reconstruction result;

[0222] The loss function of the reconstruction result can be:

[0223] L2 = ||X label - X l+1 ||2 (5)

[0224] X label represents the label used during training;

[0225] ||X label - X l+1 ||2 represents the calculation of the second norm.

[0226] Please refer to Figure 7 , an embodiment of the present invention provides a data processing method for surgical assistance, including:

[0227] S71: Receive virtual data from the server;

[0228] S72: Receive real-time endoscopic images from the server;

[0229] S73: Based on the virtual data, display the appearance model of the virtual bronchial tree on the mixed reality display device;

[0230] S74: Display the real-time endoscopic images on the mixed reality display device;

[0231] S75: Obtain and display the real - scene image of the surgical process on the mixed - reality display device;

[0232] Figure 7 The data - processing method shown can be applied to the mixed - reality display device. Furthermore, Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 Some or all of the functions implemented by the mixed - reality display device in the illustrated embodiments and their alternative embodiments can be understood as Figure 7 the steps of the data - processing method for surgical assistance shown. Figure 7 For the technical terms, alternative embodiments, and technical effects in the illustrated embodiments, reference can be made to Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 the illustrated embodiments and their alternative embodiments. For repeated content, it will not be elaborated again.

[0233] Figure 7 In one implementation of the illustrated embodiment, the data - processing method for surgical assistance further includes:

[0234] Based on the virtual data and the real - time position, display the internal navigation image of the target position inside the virtual bronchial tree on the mixed - reality display device, where the real - time position is determined by the server or the mixed - reality display device.

[0235] Figure 7 In one implementation of the illustrated embodiment, the data - processing method for surgical assistance further includes:

[0236] Receive at least part of the CT data of the human body sent by the server;

[0237] Based on the real - time position, display the CT image of the real - time position on the mixed - reality display device.

[0238] Figure 7 In one implementation of the illustrated embodiment, the data - processing method for surgical assistance further includes:

[0239] Display the three - dimensional perspective image of the virtual bronchial tree on the mixed - reality display device.

[0240] Figure 7 In one implementation of the illustrated embodiment, the data - processing method for surgical assistance further includes:

[0241] Receive the lesion - position data sent by the server;

[0242] When the mixed reality display device displays the appearance model and / or three-dimensional perspective image of the virtual bronchial tree, the lesion area is displayed based on the lesion position data.

[0243] Figure 7 In one implementation of the illustrated embodiment, the data processing method for surgical assistance further includes:

[0244] Receiving a target path;

[0245] When the mixed reality display device displays the appearance model and / or three-dimensional perspective image of the virtual bronchial tree, the target path is displayed therein.

[0246] Figure 7 In one implementation of the illustrated embodiment, the data processing method for surgical assistance further includes:

[0247] Obtaining a real-scene image of the surgical procedure;

[0248] Displaying the real-scene image on the mixed reality display device.

[0249] Figure 7 In one implementation of the illustrated embodiment, the data processing method for surgical assistance further includes:

[0250] Obtaining interaction information of the current user;

[0251] In response to the interaction information of the current user, adjusting the content displayed on the mixed reality display device.

[0252] Corresponding to Figure 7 For the data processing method shown, please refer to Figure 8 , an embodiment of the present invention provides a data processing device 80 for surgical assistance, including:

[0253] A receiving module 81, configured to:

[0254] Receive virtual data and real-time endoscopic images from a server; the virtual data includes data of the virtual bronchial tree of the human body, the virtual data is formed by the server based on the CT data of the human body, and the real-time endoscopic image is collected after the bronchoscope enters the human body during the surgical procedure;

[0255] A display module 82, configured to:

[0256] Based on the virtual data, display the appearance model of the virtual bronchial tree on a mixed reality display device;

[0257] Display the real-time endoscopic image on the mixed reality display device;

[0258] Obtain and display the real - scene images of the surgical procedure; the real - scene images are obtained by the image acquisition device during the image acquisition of the surgical procedure; the displayed real - scene images are adapted to the real scene observed by the medical staff during the surgical procedure.

[0259] In one of the embodiments, the display module 82 is further configured to:

[0260] Based on the virtual data and the real - time position of the bronchoscope in the human body, display the internal navigation image of the target position inside the virtual bronchial tree on the mixed - reality display device.

[0261] In one of the embodiments, the receiving module 81 is further configured to: receive at least part of the CT data of the human body sent by the server;

[0262] The display module 82 is further configured to display the CT image of the real - time position on the mixed - reality display device based on the real - time position.

[0263] In one of the embodiments, the display module 82 is further configured to: display the three - dimensional perspective image of the virtual bronchial tree on the mixed - reality display device.

[0264] In one of the embodiments, the receiving module 81 is further configured to: receive the lesion position data sent by the server;

[0265] The display module 82 is further configured to: when displaying the appearance model and / or three - dimensional perspective image of the virtual bronchial tree on the mixed - reality display device, display the lesion area based on the lesion position data.

[0266] In one of the embodiments, the receiving module 81 is further configured to: receive the target path;

[0267] The display module 82 is further configured to: when displaying the appearance model and / or three - dimensional perspective image of the virtual bronchial tree on the mixed - reality display device, display the target path therein.

[0268] In one of the embodiments, the surgical - assistance data - processing device further includes:

[0269] An interactive acquisition module, configured to acquire the interactive information of the current user;

[0270] An adjustment module, configured to adjust the content displayed on the mixed - reality display device in response to the interactive information of the current user.

[0271] Please refer to Figure 9, an embodiment of the present invention further provides a data processing method for surgical assistance, which is applied to a server. The data processing method includes:

[0272] S91: Obtain virtual data;

[0273] S92: Obtain intraoperative data;

[0274] S93: Obtain a real - scene image of the surgical process;

[0275] S94: Send the intraoperative data, the virtual data, and the real - scene image to a mixed - reality display device.

[0276] This data processing method can be applied to a server. Furthermore, Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 Some or all of the functions implemented by the server in the embodiments and their optional implementation manners shown can be understood as Figure 9 The steps of the data processing method for surgical assistance shown. Figure 9 For the technical terms, optional implementation manners, and technical effects in the embodiments shown, reference can be made to Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 The embodiments and their optional implementation manners shown.

[0277] Figure 9 In an implementation manner of the embodiment shown, the data processing method further includes: During the surgical process of the human body, determine the real - time position of the bronchoscope in the human body and feed back the real - time position to the mixed - reality display device. Figure 7 In an implementation manner of the embodiment shown, the data processing method further includes:

[0278] Send at least part of the CT data of the human body to the mixed - reality display device.

[0279] Corresponding to Figure 9 For the embodiment shown, please refer to Figure 10 , an embodiment of the present invention further provides a data processing device 100 for surgical assistance, which is applied to a server and includes:

[0280] An acquisition module 101, configured to:

[0281] Obtain virtual data;

[0282] Obtain intraoperative data;

[0283] Obtain a real - scene image of the surgical process; The real - scene image is obtained by an image acquisition device for image acquisition of the surgical process;

[0284] The sending module 102 is configured to:

[0285] Send the intraoperative data, the virtual data, and the real - world scene image to the mixed - reality display device.

[0286] Figure 10 In one implementation of the illustrated embodiment, the sending module 102 is further configured to:

[0287] During the surgical procedure on the human body, determine the real - time position of the bronchoscope within the human body and feedback the real - time position to the mixed - reality display device.

[0288] In one implementation, the sending module 102 is further configured to:

[0289] Send at least part of the CT data of the human body to the mixed - reality display device.

[0290] Please refer to Figure 11 , which provides an electronic device 110, including:

[0291] A processor 111; and,

[0292] A memory 112 for storing executable instructions of the processor;

[0293] Wherein, the processor 111 is configured to execute the methods involved above by executing the executable instructions.

[0294] The processor 111 can communicate with the memory 112 via a bus 113.

[0295] The embodiment of the present invention also provides a computer - readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the methods involved above are implemented.

[0296] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above - mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer - readable storage medium. When the program is executed, it executes the steps including the above - mentioned method embodiments; and the aforementioned storage medium includes: various media such as ROM, RAM, magnetic disks, or optical disks that can store program codes.

[0297] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing system for surgical assistance, characterized in that, Comprising: A server, a bronchoscope, and a mixed reality display device; the server can communicate directly or indirectly with the mixed reality display device and the bronchoscope; The server is used for: Obtaining virtual data, where the virtual data includes data of the virtual bronchial tree of the human body; Obtaining intraoperative data, where the intraoperative data includes: during the surgical process, real-time endoscopic images collected after the bronchoscope enters the human body; Sending the intraoperative data and the virtual data to the mixed reality display device; The mixed reality display device is used for: Obtaining and displaying real-world scene images of the surgical process; the real-world scene images are obtained by an image acquisition device collecting images of the surgical process; the displayed real-world scene images are adapted to the real scene observed by the medical staff during the surgical process; Based on the virtual data, displaying the external model of the virtual bronchial tree; Displaying the real-time endoscopic images; Wherein: The content displayed by multiple mixed reality display devices is synchronized; multiple mixed reality display devices include a mixed reality display device used by an expert and a mixed reality display device of the surgeon-in-charge. The image actually seen by the surgeon-in-charge is the real-world scene image. The image acquisition device is integrated into the mixed reality display device worn by the surgeon-in-charge, and its position and angle are adapted to the eyes of the surgeon-in-charge; The server or the mixed reality display device is used for: during the surgical process of the human body, determining the real-time position of the bronchoscope in the human body; The mixed reality display device is further used for: Based on the virtual data and the real-time position, displaying the internal navigation image of the target position inside the virtual bronchial tree; Wherein, the target position is adapted to the real-time position, and the internal navigation image refers to an image obtained by a virtual camera observing the inside of the virtual bronchial tree at the target position.

2. The surgical assistance data processing system according to claim 1, wherein The server is further used for: Obtaining the CT data of the human body, and the virtual data is formed based on the CT data; Sending at least part of the CT data of the human body to the mixed reality display device; the at least part of the CT data includes CT images; The mixed reality display device is further used for: Based on the real-time position, displaying the CT image of the real-time position.

3. The surgical assistance data processing system according to claim 1, wherein The mixed reality display device is further used for: Displaying the three-dimensional perspective image of the virtual bronchial tree.

4. The surgical assistance data processing system according to claim 1, wherein The virtual data further includes lesion position data, and the lesion position data characterizes the relative position between the virtual bronchial tree and the lesion area; The mixed reality display device is further used for: When displaying the external model and / or three-dimensional perspective image of the virtual bronchial tree, based on the lesion position data, displaying the lesion area.

5. The surgical assistance data processing system according to claim 1, wherein The virtual data includes data of a target path, and the target path simulates the path of the bronchoscope reaching the lesion after entering the human body; The mixed reality display device is further configured to: When presenting the appearance model and / or three-dimensional perspective image of the virtual bronchial tree, present the target path therein.

6. The data processing system for surgical assistance according to any one of claims 1 to 5, characterized in that, The mixed reality display device is further configured to: In response to the interaction information of the current user, adjust the presented content, where the interaction information is detected by an interaction device, and the interaction device is integrated with the mixed reality display device or is configured to be able to communicate with the mixed reality display device.

7. The surgical assistance data processing system according to claim 6, wherein The adjustment includes at least one of the following: Shrink the appearance model; Enlarge the appearance model; Rotate the appearance model; Change the display position of the appearance model; Close the display of the appearance model; Open the display of the appearance model; Shrink the display window; Enlarge the display window; Change the position of the display window; Open the display window; Close the display window; Wherein, the display window is a window for displaying a corresponding image, and the corresponding image is any one of the following: the real-time endoscopic image, the internal navigation image of the bronchoscope, the CT image of the human body, the three-dimensional perspective image of the virtual bronchial tree.

8. The data processing system for surgical assistance according to any one of claims 1 to 5, characterized in that, It further includes a CT scanning device, the virtual data is formed based on CT data, the CT data is collected by the CT scanning device, and the server can communicate directly or indirectly with the CT scanning device.

9. The data processing system for surgical assistance according to any one of claims 1 to 5, characterized in that It further includes a recording device and / or a relaying device; The recording device is used to record the content presented by the mixed reality display device; The relaying device is used to relay the content presented by the mixed reality display device.

10. The data processing system for surgical assistance according to any one of claims 1 to 5, characterized in that The mixed reality display device is a head-mounted mixed reality display.

11. A data processing device for surgical assistance, characterized in that, It includes: A receiving module, configured to: Receive virtual data and real-time endoscopic images from the server; the virtual data includes data of the virtual bronchial tree of the human body, and the real-time endoscopic image is collected after the bronchoscope enters the human body during the operation; A presenting module, configured to: Obtain and present, on the mixed reality display device, the real scene image of the operation process; the real scene image is obtained by an image acquisition device collecting images of the operation process; The presented real scene image is adapted to the real scene observed by the medical staff during the operation; Based on the virtual data, present the appearance model of the virtual bronchial tree on the mixed reality display device; Present the real-time endoscopic image on the mixed reality display device; Wherein: The content presented by multiple mixed reality display devices is synchronized; the multiple mixed reality display devices include the mixed reality display device used by experts and the mixed reality display device of the surgeon. What the surgeon actually sees is the real scene image. The image acquisition device is integrated with the mixed reality display device worn by the surgeon, and its position and angle are adapted to the eyes of the surgeon; The presenting module is further configured to: Based on the virtual data and the real-time position of the bronchoscope in the human body, an internal navigation image of the target position inside the virtual bronchial tree is displayed on the mixed reality display device; during the surgical procedure on the human body, the real-time position of the bronchoscope in the human body is determined by the server or the mixed reality display device; wherein, the target position is adapted to the real-time position, and the internal navigation image refers to an image obtained by a virtual camera observing the inside of the virtual bronchial tree at the target position.

12. A data processing device for surgical assistance, applied to a server, characterized in that, It includes: An acquisition module, configured to: Acquire virtual data, where the virtual data includes data of the virtual bronchial tree of the human body; Acquire intraoperative data, where the intraoperative data includes: real-time endoscopic images acquired after the bronchoscope enters the human body during the surgical procedure; Acquire real-world scene images of the surgical procedure; the real-world scene images are obtained by an image acquisition device acquiring images of the surgical procedure; A sending module, configured to: Send the intraoperative data, the virtual data, and the real-world scene images to the mixed reality display device, so that: the mixed reality display device displays the real-world scene images of the surgical procedure, and the displayed real-world scene images are adapted to the real scene observed by the medical staff during the surgical procedure; based on the virtual data, display the external model of the virtual bronchial tree, and: display the real-time endoscopic images; based on the virtual data and the real-time position of the bronchoscope in the human body, display the internal navigation image of the target position inside the virtual bronchial tree on the mixed reality display device; during the surgical procedure on the human body, the real-time position of the bronchoscope in the human body is determined by the server or the mixed reality display device; Wherein: The target position is adapted to the real-time position, and the internal navigation image refers to an image obtained by a virtual camera observing the inside of the virtual bronchial tree at the target position; The content displayed by multiple mixed reality display devices is synchronized; the multiple mixed reality display devices include a mixed reality display device remotely used by an expert and a mixed reality display device of the surgeon, and the image actually seen by the surgeon is the real-world scene image, the image acquisition device is integrated in the mixed reality display device worn by the surgeon, and its position, angle, and focal length are adapted to the eyes of the surgeon.

13. An electronic device, characterized in that, It includes a processor and a memory, The memory is used to store code; The processor is configured to execute the code in the memory to implement the functions of the device described in claim 11 or 12.

14. A storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the functions of the device described in claim 11 or 12.

Images