Data processing method and device based on data visual interaction technology

Abstract

The invention provides a data processing method and device based on a data visual interaction technology, and the method and device are applied to auxiliary mixed reality equipment for minimally invasive surgery. Displaying a matched virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time image and eyeball tracking information of the user; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; displaying a virtual endoscope image in a second target area based on the intraoperative real-time image, eyeball tracking information of the user and real-time position tracking information of an endoscope; based on the eyeball tracking information of the user, the display state of the virtual three-dimensional anatomical structure in the first target area and the display state of the virtual endoscope image in the second target area are adjusted till the operation is completed, and the precision and efficiency of the minimally invasive operation can be improved on the basis of reducing the risk of the minimally invasive operation.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and device based on data visual interaction technology. Background Art

[0002] With the rapid development of modern medicine, minimally invasive surgical techniques represented by laparoscopic techniques have gradually been widely used due to their advantages of less trauma and faster recovery. However, during minimally invasive surgery, doctors can only observe the local conditions of the patient's body through an endoscope, which makes the surgery more difficult, inefficient and has high safety risks.

[0003] In order to solve the above problems, many researchers have begun to try to apply Mixed Reality (MR) technology to minimally invasive surgery to help doctors obtain clear anatomical structures in the patient's body, so as to reduce the difficulty and safety risks of surgery. However, existing mixed reality devices cannot dynamically monitor the patient's organ tissues and surrounding tissues in real time, and the virtual anatomical structures displayed by mixed reality devices are easy to affect the doctor's vision, resulting in an increase in the difficulty and safety risks of surgery. Summary of the invention

[0004] The present invention provides a data processing method and device based on data visual interaction technology, which is used to solve the problem that the mixed reality equipment in the prior art cannot dynamically monitor the patient's organ tissue and peripheral tissue in real time, and the virtual anatomical structure displayed by the mixed reality equipment easily affects the doctor's line of sight, resulting in an increase in surgical difficulty and safety risks, so as to improve the accuracy and efficiency of minimally invasive surgery while reducing the risks of minimally invasive surgery.

[0005] In one aspect, the present invention provides a data processing method based on data visual interaction technology, the method being applied to an auxiliary mixed reality device for minimally invasive surgery, comprising: Receiving an intraoperative real-time image acquired by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time image and eye tracking information of a user; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; Based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in the second target area; Based on the eye tracking information of the user, the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area are adjusted until the operation is completed.

[0006] In an optional embodiment of the present application, displaying a matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information specifically includes: Based on the intraoperative real-time image, determining the user's current field of view; Determine the user's area of ​​interest based on the user's eye tracking information; the area of ​​interest is an area within the current field of view; A first target region is determined based on the region of interest of the user, and a matching virtual three-dimensional anatomical structure is displayed in the first target region.

[0007] In an optional embodiment of the present application, determining the first target area based on the user's area of ​​interest and displaying the matching virtual three-dimensional anatomical structure in the first target area specifically includes: Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the surgical area and the area of ​​interest in the user's current field of view is taken as the first target area; A target virtual three-dimensional anatomical structure matching the first target region in the virtual three-dimensional anatomical structure is determined, and the target virtual three-dimensional anatomical structure is displayed in the first target region.

[0008] In an optional embodiment of the present application, displaying a virtual endoscopic image in the second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes: Based on the intraoperative real-time image, determining the user's current field of view; Determine the user's area of ​​interest based on the user's eye tracking information; the area of ​​interest is an area within the current field of view; A second target area is determined based on the real-time position tracking information of the endoscope, and a virtual endoscope image is displayed in the second target area.

[0009] In an optional embodiment of the present application, determining the second target area based on the real-time position tracking information of the endoscope and displaying the virtual endoscope image in the second target area specifically includes: Determine the area where the endoscope is located in the user's current field of view based on the real-time position tracking information of the endoscope; Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the operation area in the user's current field of view and the area where the endoscope is located is taken as the second target area, and the virtual endoscope image is displayed in the second target area based on the posture of the endoscope.

[0010] In an optional embodiment of the present application, adjusting the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the eye tracking information of the user specifically includes: Based on the eye tracking information of the user, updating the first target area and the second target area; Based on the updated first target area and the second target area, an updated virtual three-dimensional anatomical structure and a virtual endoscopic image are displayed.

[0011] In an optional embodiment of the present application, the method further includes: The current surgical stage is determined based on the eye tracking information of the user, and the update frequency of the first target area and the second target area is adjusted based on the current surgical stage.

[0012] In a second aspect, the present invention further provides a data processing device based on data visual interaction technology, wherein the device is applied to an auxiliary mixed reality device for minimally invasive surgery, comprising: The anatomical structure display module is used to receive the intraoperative real-time image acquired by the image acquisition unit, and display the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image; A virtual endoscopic image display module, used for displaying a virtual endoscopic image in a second target area based on the intraoperative real-time image, the user's eye tracking information and the real-time position tracking information of the endoscope; A display state adjustment module is used to adjust the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the user's eye tracking information until the operation is completed.

[0013] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method based on data visual interaction technology as described above is implemented.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data processing method based on the data visual interaction technology as described above is implemented.

[0015] The present invention provides a data processing method and device based on data visual interaction technology, which are applied to auxiliary mixed reality equipment for minimally invasive surgery. By receiving the real-time intraoperative image acquired by an image acquisition unit, a matching virtual three-dimensional anatomical structure is displayed in a first target area based on the real-time intraoperative image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope, a virtual endoscopic image is displayed in a second target area; based on the eye tracking information of the user, the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area are adjusted until the surgery is completed, so that the patient's organ tissue and surrounding tissue can be monitored dynamically in real time, and the virtual anatomical structure displayed by the mixed reality device is not easy to affect the doctor's line of sight, thereby improving the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a data processing method based on data visual interaction technology provided by the present invention; Figure 2 A schematic diagram of the display process of the virtual three-dimensional anatomical structure provided by the present invention; Figure 3 A schematic diagram of the display process of the virtual endoscopic image provided by the present invention; Figure 4 A schematic diagram of the adjustment process of the display status provided by the present invention; Figure 5 A structural block diagram of a data processing device based on data visual interaction technology provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The specific implementation process of the data processing method and device based on the data visual interaction technology in the embodiment of the present invention will be described below.

[0020] Figure 1 A flow chart of a data processing method based on data visual interaction technology provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method is applied to an auxiliary mixed reality device for minimally invasive surgery, and the method may include: Step 101, receiving a real-time intraoperative image acquired by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the real-time intraoperative image and user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image.

[0021] Specifically, based on the above content, it can be known that the existing mixed reality devices cannot monitor the patient's organ tissue and surrounding tissue in real time and dynamically, and the virtual anatomical structure displayed by the mixed reality device is easy to affect the doctor's line of sight, resulting in an increase in the difficulty of surgery and safety risks. The fundamental reason is that the existing mixed reality device only mechanically displays the pre-established three-dimensional anatomical structure model in the user's field of view, and does not follow the user's line of sight to adjust, resulting in the user's inability to correspond the three-dimensional anatomical structure model with the real patient lesion, and thus unable to understand the patient's organ tissue and surrounding tissue in real time. And it is precisely because of the mechanical display method that the virtual image blocks the user's line of sight, resulting in the user being unable to directly observe the real surgical situation, further increasing the difficulty of surgery and safety risks. In response to this problem, the present invention proposes a data processing method based on data visual interaction technology, which can adjust the display method of virtual information based on the user's line of sight, thereby assisting the user to perform real-time dynamic monitoring of the patient's organ tissue and surrounding tissue, and the virtual anatomical structure displayed by the mixed reality device is not easy to affect the user's line of sight. It can be understood that the user of the embodiment of the present invention specifically refers to a doctor performing minimally invasive surgery. The auxiliary mixed reality device for minimally invasive surgery includes: A head mounted display (HMD) is used to overlay virtual images onto the user's field of view, allowing the user to see a combination of real-world and virtual elements.

[0022] Motion sensors are used to track the user's head and body movements to ensure that the virtual image remains consistent relative to the real world.

[0023] The eye tracking sensor is used to obtain the user's eye tracking information, wherein the user's eye tracking information includes the user's eye movement information and pupil position information.

[0024] The image acquisition unit is used to capture the user's visual field image, i.e., the aforementioned real-time image during the operation. Preferably, the image acquisition unit is integrated with the head-mounted display, and is specifically arranged at the center of the front part of the head-mounted display (i.e., the part corresponding to the user's face), and the height is close to the user's eye height. Based on this, it is possible to maximize the matching of the viewing angle of the image acquisition unit with the user's viewing angle, thereby ensuring that the acquired real-time image during the operation is an accurate image of the user's visual field.

[0025] A computing unit is used to process images and data and perform real-time calculations. The computing unit can be an independent computer or a processor integrated in the head-mounted display.

[0026] A software platform is used to support MR applications, including development tools and applications. The software platform is responsible for managing the creation, display and interaction of virtual content.

[0027] The network connection unit is used to realize data transmission between the device and the cloud or other devices, supporting remote collaboration and real-time data update.

[0028] The power supply is used to provide the necessary power to the device to ensure that the device continues to operate during use.

[0029] The above components work together to enable mixed reality devices to provide accurate support information when users perform minimally invasive surgeries, improving the efficiency and success rate of surgery while reducing safety risks.

[0030] More specifically, after the minimally invasive surgery begins, the auxiliary mixed reality device of the embodiment of the present invention will receive the intraoperative real-time image captured by the image acquisition unit, and display the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; it can be understood that the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image. The surgical area includes a lesion area and a surgical route area. The lesion area is the area where the lesion is located, that is, the target area where surgery is required, and the surgical route area is the area corresponding to the route of the endoscope entering the lesion area. Based on this, the embodiment of the present invention can accurately display the virtual three-dimensional anatomical structure during the process of the endoscope entering the lesion area and during the operation, thereby helping the user to observe the accurate three-dimensional anatomical structure of the patient during the process of implanting the endoscope and during the operation, thereby reducing the difficulty and safety risks of the operation. The surgical route area is preferably obtained by simulating the surgical process based on the virtual three-dimensional anatomical structure, based on which the accuracy and safety of the surgical route area can be ensured.

[0031] Figure 2 A schematic diagram of the display process of the virtual three-dimensional anatomical structure provided by the present invention, such as Figure 2As shown, the display of a matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information specifically includes: Step 1011, determining the user's current visual field based on the intraoperative real-time image; Step 1012, determining the user's region of interest based on the user's eye tracking information; the region of interest is the region within the current field of view; Step 1013: determine a first target region based on the region of interest of the user, and display a matching virtual three-dimensional anatomical structure in the first target region.

[0032] Specifically, it is understandable that the aforementioned image acquisition unit will continue to acquire real-time images during the operation. Based on this, the auxiliary mixed reality device of the embodiment of the present invention can continuously monitor the user's field of view during the operation. It is worth noting that the user's line of sight will be focused on the surgical area most of the time after the operation begins, including during the implantation of the endoscope, the user's line of sight will be focused on the aforementioned surgical route area, and during the operation, the user's line of sight will be focused on the aforementioned lesion area. Based on this, after determining the user's current field of view, the embodiment of the present invention further determines the user's area of ​​interest (i.e., the area where the user's line of sight is focused) based on the user's eye tracking information. It is understandable that the area of ​​interest is the area in the current field of view.

[0033] It should also be noted that although the user's line of sight will be focused on the surgical area most of the time after the operation begins, sometimes the user's attention will be diverted for various reasons, such as wiping sweat, taking surgical tools, or observing endoscopic images. At this time, it is not necessary to display the aforementioned virtual three-dimensional anatomical structure and virtual endoscopic image to avoid affecting the user's normal field of vision. Based on this, the embodiment of the present invention further determines the first target area based on the user's area of ​​interest, and displays the matching virtual three-dimensional anatomical structure in the first target area. More specifically, the determination of the first target area based on the user's area of ​​interest and the display of the matching virtual three-dimensional anatomical structure in the first target area specifically include: Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the surgical area and the area of ​​interest in the user's current field of view is taken as the first target area; A target virtual three-dimensional anatomical structure matching the first target region in the virtual three-dimensional anatomical structure is determined, and the target virtual three-dimensional anatomical structure is displayed in the first target region.

[0034] It is worth noting that in the embodiment of the present invention, before the operation begins, the user will set a surgical area mark on the skin of the corresponding part of the patient's body based on the aforementioned predetermined surgical area. Specifically, the surgical area mark can be generated by spraying colored pigments or any other method that can be recorded and identified by images, and the embodiment of the present invention does not specifically limit this. Based on this, the embodiment of the present invention can quickly determine the surgical area in the user's current field of view based on the real-time intraoperative image through an image recognition algorithm, and then use the overlapping area of ​​the surgical area in the user's current field of view and the area of ​​interest as the first target area. On this basis, the target virtual three-dimensional anatomical structure matching the first target area in the virtual three-dimensional anatomical structure can be quickly and accurately determined, and the target virtual three-dimensional anatomical structure can be displayed in the first target area. Based on this, it is possible to provide the user with an accurate virtual three-dimensional anatomical structure display on the basis of avoiding affecting the user's normal line of sight, so that the user can perform the operation accurately and efficiently and avoid safety risks.

[0035] Step 102, based on the intraoperative real-time image, the user's eye tracking information and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in the second target area; Specifically, the embodiments of the present invention have found through research that in order to maximize the accuracy, efficiency and safety of the operation, in addition to the accurate display of the virtual three-dimensional anatomical structure during the operation, it is also necessary to accurately grasp the position and posture of the endoscope. Based on this, the embodiments of the present invention further display the virtual endoscope image in the second target area based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope to help the user understand the endoscope status at each time period after the operation begins. More specifically, Figure 3 A schematic diagram of the display process of the virtual endoscopic image provided by the present invention is shown in FIG. Figure 3 As shown, the virtual endoscope image is displayed in the second target area based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope, specifically including: Step 1021, determining the user's current visual field based on the intraoperative real-time image; Step 1022, based on the user's eye tracking information, determine the user's area of ​​interest; the area of ​​interest is the area in the current field of view; Step 1023: determine a second target area based on the real-time position tracking information of the endoscope, and display a virtual endoscope image in the second target area.

[0036] Specifically, for the display of virtual endoscopic images, similar to the display process of virtual three-dimensional anatomical structures, it is also necessary to first determine the user's current field of view based on the real-time intraoperative image, and then determine the user's area of ​​interest based on the user's eye tracking information, the area of ​​interest being the area within the current field of view, and finally determine the second target area based on the real-time position tracking information of the endoscope, and display the virtual endoscopic image in the second target area. It is understandable that the real-time position tracking information of the endoscope can be obtained through a positioning device disposed in the endoscope, and the real-time position tracking information of the endoscope includes coordinate information and posture information, based on which, the embodiment of the present invention can quickly and accurately determine the display area and display posture of the virtual endoscopic image. It is worth noting that the virtual endoscopic image is a pre-constructed three-dimensional endoscopic image, based on which, it can help the user accurately determine the intraoperative state of the endoscope.

[0037] More specifically, determining the second target area based on the real-time position tracking information of the endoscope and displaying the virtual endoscope image in the second target area specifically includes: Determine the area where the endoscope is located in the user's current field of view based on the real-time position tracking information of the endoscope; Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the operation area in the user's current field of view and the area where the endoscope is located is taken as the second target area, and the virtual endoscope image is displayed in the second target area based on the posture of the endoscope.

[0038] It can be understood that, similar to the display of virtual three-dimensional anatomical structures, the display process of the virtual endoscopic image in the embodiment of the present invention is: first, the area where the endoscope is located in the user's current field of view is determined based on the real-time position tracking information of the endoscope, and then the surgical area in the user's current field of view is determined based on the real-time intraoperative image; finally, the overlapping area between the surgical area in the user's current field of view and the area where the endoscope is located is used as the second target area, and the virtual endoscopic image is displayed in the second target area based on the posture of the endoscope. Based on this, the accuracy and efficiency of minimally invasive surgery can be maximized on the basis of accurately displaying the position and posture of the virtual endoscopic image, and the safety risks can be reduced. At the same time, the workload of virtual endoscopic image display can also be reduced.

[0039] Furthermore, on the basis of the above, the embodiments of the present invention can also display the patient's blood pressure, heart rate and other medical signs through auxiliary mixed reality devices. Based on this, it can avoid the user frequently checking the corresponding vital signs monitoring instruments to determine the patient's status, further improving the efficiency of the operation.

[0040] Step 103: Based on the eye tracking information of the user, the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area are adjusted until the operation is completed.

[0041] Specifically, Figure 4 A schematic diagram of the adjustment process of the display status provided by the present invention is shown in FIG. Figure 4 As shown, adjusting the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the eye tracking information of the user specifically includes: Step 1031, updating the first target area and the second target area based on the eye tracking information of the user; Step 1032 , based on the updated first target area and the second target area, display the updated virtual three-dimensional anatomical structure and the virtual endoscopic image.

[0042] Based on the foregoing, it can be understood that as the surgical process continues to progress, the user's area of ​​interest will also change, and accordingly, the first target area and the second target area will also change, and the target virtual three-dimensional anatomical structure and virtual endoscopic image that need to be displayed will also change. Based on this, the embodiment of the present invention will continue to track the user's eye state, and update the first target area and the second target area based on the user's eye tracking information, and then display the updated virtual three-dimensional anatomical structure and virtual endoscopic image based on the updated first target area and the second target area. Through the above-mentioned update method, it can be ensured that the user can accurately and dynamically monitor the status of the patient's organ tissue, peripheral tissue and endoscope after the operation begins.

[0043] Furthermore, the method further comprises: The current surgical stage is determined based on the eye tracking information of the user, and the update frequency of the first target area and the second target area is adjusted based on the current surgical stage.

[0044] Based on the above embodiments, it can be known that the surgical stages of the embodiments of the present invention can be roughly divided into the endoscope implantation stage and the lesion treatment stage. Through research, the present invention has found that since the endoscope implantation stage does not involve the lesion treatment process, the requirements for operation accuracy are relatively low. If the first target area and the second target area are frequently updated at a conventional update frequency, it will bring a huge workload to the auxiliary mixed reality device and will not generate positive benefits; on the contrary, for the lesion treatment stage, since it involves the lesion treatment process, the safety risk is high. Only updating the first target area and the second target area based on the conventional update frequency will not meet the accuracy and safety requirements of the surgical process. In response to this problem, the embodiment of the present invention first determines the current surgical stage based on the user's eye tracking information, and then adjusts the update frequency of the first target area and the second target area based on the current surgical stage.

[0045] More specifically, the present invention has found through research that there are obvious differences in the user's area of ​​interest for the endoscope implantation stage and the lesion treatment stage. In the endoscope implantation stage, the user will roughly estimate the position of the endoscope in the patient's body based on the implantation depth of the endoscope due to previous operating habits, and the user's line of sight will observe the patient's body changes to determine whether the position of the endoscope in the patient's body is consistent with the estimate. It can be seen that in the endoscope implantation stage, the user's line of sight will focus on the aforementioned surgical route area for a long time. Based on this, the embodiment of the present invention can determine the duration of the user's line of sight focusing on the surgical route area based on the user's eye tracking information, and then determine whether the current surgical stage is the endoscope implantation stage. For example, when it is determined that the duration of the user's line of sight focusing on the surgical route area is greater than the first preset duration threshold, the current surgical stage is judged to be the endoscope implantation stage. Similarly, in the lesion treatment stage, the present invention has found through research that the user's line of sight will focus on the aforementioned lesion area for a long time. Based on this, the embodiment of the present invention can determine the duration of the user's line of sight focusing on the lesion area based on the user's eye tracking information, and then determine whether the current surgical stage is the lesion treatment stage. For example, when it is determined that the time the user's line of sight is focused on the lesion area is greater than the second preset time threshold, the current surgical stage is determined to be the lesion treatment stage. Based on this, the steps of determining the current surgical stage in the embodiment of the present invention are: Based on the user's eye tracking information and the intraoperative real-time image, the user's area of ​​interest and the corresponding sight focusing time are determined. If the user's area of ​​interest is the surgical route area, and the time when the user's sight is focused on the surgical route area is greater than the first preset time threshold, then the current surgical stage is judged to be the endoscopic implantation stage; if the user's area of ​​interest is the lesion area, and the time when the user's sight is focused on the lesion area is greater than the second preset time threshold, then the current surgical stage is judged to be the lesion treatment stage. It is understandable that since the auxiliary mixed reality device will record the generation time of the intraoperative real-time image, the user's sight focusing time in any area of ​​interest can be determined based on continuous intraoperative real-time image analysis.

[0046] After determining the current surgical stage, the embodiment of the present invention can adjust the update frequency of the first target area and the second target area. Specifically, if the current surgical stage is the endoscope implantation stage, the update frequency is adjusted to the first frequency, and if the current surgical stage is the lesion treatment stage, the update frequency is adjusted to the second frequency. The first frequency is less than the second frequency.

[0047] It can be understood that the first preset time threshold, the second preset time threshold, the first frequency and the second frequency can be adaptively adjusted according to different types of surgeries, and the embodiment of the present invention does not specifically limit this.

[0048] Furthermore, the auxiliary mixed reality device of the embodiment of the present invention can also obtain and display the real-time images of the patient's body taken by the endoscope. It can be understood that the information displayed by the auxiliary mixed reality device at this time includes the real-time images of the patient's body, the virtual three-dimensional anatomical structure and the virtual endoscopic image. Based on this, the user can accurately grasp the three-dimensional state of the surgical area, the state of the endoscope and the state of the surgical instruments in the endoscope (in the lesion treatment stage, the real-time images of the patient's body taken by the endoscope include surgical instruments). At the same time, the user does not need to observe the patient's internal conditions through other display devices, thereby maximizing the accuracy and efficiency of minimally invasive surgery and minimizing safety risks. On this basis, the embodiment of the present invention can also judge the posture of the endoscope through the real-time image of the patient's body, and correct the virtual endoscopic image to further improve the accuracy of the virtual endoscopic image.

[0049] The solution provided by the present invention receives the real-time intraoperative image acquired by the image acquisition unit, and displays a matching virtual three-dimensional anatomical structure in a first target area based on the real-time intraoperative image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope, a virtual endoscopic image is displayed in a second target area; based on the user's eye tracking information, the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area are adjusted until the operation is completed, so that the patient's organ tissue and surrounding tissue can be monitored dynamically in real time, and the virtual anatomical structure displayed by the mixed reality device is not easy to affect the doctor's line of sight, thereby improving the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery.

[0050] Figure 5 A structural block diagram of a data processing device based on data visual interaction technology is provided for an embodiment of the present invention, such as Figure 5 As shown, the device is applied to a mixed reality auxiliary device for minimally invasive surgery, and the device comprises: The anatomical structure display module 201 is used to receive the intraoperative real-time image acquired by the image acquisition unit, and display a matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image; A virtual endoscope image display module 202, for displaying a virtual endoscope image in a second target area based on the intraoperative real-time image, the user's eye tracking information and the real-time position tracking information of the endoscope; The display state adjustment module 203 is used to adjust the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the eye tracking information of the user until the operation is completed.

[0051] The solution provided by the present invention receives the intraoperative real-time image acquired by the image acquisition unit through the anatomical structure display module 201, and displays the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image; the virtual endoscope image display module 202 displays the virtual endoscope image in the second target area based on the intraoperative real-time image, the user's eye tracking information and the real-time position tracking information of the endoscope; the display state adjustment module 203 adjusts the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information until the operation is completed, so that the patient's organ tissue and peripheral tissue can be monitored dynamically in real time, and the virtual anatomical structure displayed by the mixed reality device is not easy to affect the doctor's line of sight, thereby improving the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery.

[0052] In an optional embodiment of the present application, displaying a matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information specifically includes: Based on the intraoperative real-time image, determining the user's current field of view; Determine the user's area of ​​interest based on the user's eye tracking information; the area of ​​interest is an area within the current field of view; A first target region is determined based on the region of interest of the user, and a matching virtual three-dimensional anatomical structure is displayed in the first target region.

[0053] In an optional embodiment of the present application, determining the first target area based on the user's area of ​​interest and displaying the matching virtual three-dimensional anatomical structure in the first target area specifically includes: Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the surgical area and the area of ​​interest in the user's current field of view is taken as the first target area; A target virtual three-dimensional anatomical structure matching the first target region in the virtual three-dimensional anatomical structure is determined, and the target virtual three-dimensional anatomical structure is displayed in the first target region.

[0054] In an optional embodiment of the present application, displaying a virtual endoscopic image in the second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes: Based on the intraoperative real-time image, determining the user's current field of view; Determine the user's area of ​​interest based on the user's eye tracking information; the area of ​​interest is an area within the current field of view; A second target area is determined based on the real-time position tracking information of the endoscope, and a virtual endoscope image is displayed in the second target area.

[0055] In an optional embodiment of the present application, determining the second target area based on the real-time position tracking information of the endoscope and displaying the virtual endoscope image in the second target area specifically includes: Determine the area where the endoscope is located in the user's current field of view based on the real-time position tracking information of the endoscope; Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the operation area in the user's current field of view and the area where the endoscope is located is taken as the second target area, and the virtual endoscope image is displayed in the second target area based on the posture of the endoscope.

[0056] In an optional embodiment of the present application, adjusting the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the eye tracking information of the user specifically includes: Based on the eye tracking information of the user, updating the first target area and the second target area; Based on the updated first target area and the second target area, an updated virtual three-dimensional anatomical structure and a virtual endoscopic image are displayed.

[0057] In an optional embodiment of the present application, the display state adjustment module 203 is further configured to perform the following operations: The current surgical stage is determined based on the eye tracking information of the user, and the update frequency of the first target area and the second target area is adjusted based on the current surgical stage.

[0058] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302 and the memory 303 communicate with each other through the communication bus 304. The processor 301 may call the logic instructions in the memory 303 to execute a data processing method based on data visual interaction technology, the method comprising: receiving a real-time intraoperative image acquired by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the real-time intraoperative image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; displaying a virtual endoscopic image in a second target area based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope; and adjusting the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the user's eye tracking information until the surgery is completed.

[0059] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0060] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data processing method based on data visual interaction technology provided by the above-mentioned methods, the method including: receiving a real-time intraoperative image acquired by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the real-time intraoperative image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope, displaying a virtual endoscopic image in a second target area; based on the user's eye tracking information, adjusting the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area until the operation is completed.

[0061] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data processing method based on data visual interaction technology provided by the above-mentioned methods, the method comprising: receiving a real-time intraoperative image acquired by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the real-time intraoperative image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; based on the real-time intraoperative image, the user's eye tracking information and the real-time position tracking information of the endoscope, displaying a virtual endoscopic image in a second target area; based on the user's eye tracking information, adjusting the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area until the operation is completed.

[0062] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0063] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the data processing method based on the data visual interaction technology described in each embodiment or some parts of the embodiment.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on data visual interaction technology, the method is applied to an auxiliary mixed reality device for minimally invasive surgery, characterized in that: include: receiving an intraoperative real-time image acquired by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time image and eye tracking information of a user; The virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; Based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in the second target area; Based on the eye tracking information of the user, the display status of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area are adjusted until the operation is completed.

2. The data processing method based on data visual interaction technology according to claim 1 is characterized in that: The displaying of a matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information specifically includes: Based on the intraoperative real-time image, determining the user's current field of view; Determine the user's area of ​​interest based on the user's eye tracking information; the area of ​​interest is an area within the current field of view; A first target region is determined based on the region of interest of the user, and a matching virtual three-dimensional anatomical structure is displayed in the first target region.

3. The data processing method based on data visual interaction technology according to claim 2 is characterized in that: The method of determining the first target area based on the user's area of ​​interest and displaying the matching virtual three-dimensional anatomical structure in the first target area specifically includes: Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the surgical area and the area of ​​interest in the user's current field of view is taken as the first target area; A target virtual three-dimensional anatomical structure matching the first target region in the virtual three-dimensional anatomical structure is determined, and the target virtual three-dimensional anatomical structure is displayed in the first target region.

4. The data processing method based on data visual interaction technology according to claim 1 is characterized in that: The displaying of a virtual endoscope image in the second target area based on the intraoperative real-time image, the user's eye tracking information and the real-time position tracking information of the endoscope specifically includes: Based on the intraoperative real-time image, determining the user's current field of view; Determine the user's area of ​​interest based on the user's eye tracking information; the area of ​​interest is an area within the current field of view; A second target area is determined based on the real-time position tracking information of the endoscope, and a virtual endoscope image is displayed in the second target area.

5. The data processing method based on data visual interaction technology according to claim 4 is characterized in that: The method of determining the second target area based on the real-time position tracking information of the endoscope and displaying the virtual endoscope image in the second target area specifically includes: Determine the area where the endoscope is located in the user's current field of view based on the real-time position tracking information of the endoscope; Determine the surgical area within the user's current field of view based on the intraoperative real-time image; The overlapping area between the operation area in the user's current field of view and the area where the endoscope is located is taken as the second target area, and the virtual endoscope image is displayed in the second target area based on the posture of the endoscope.

6. The data processing method based on data visual interaction technology according to claim 1 is characterized in that: The adjusting, based on the eye tracking information of the user, the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area specifically includes: Based on the eye tracking information of the user, updating the first target area and the second target area; Based on the updated first target area and the second target area, an updated virtual three-dimensional anatomical structure and a virtual endoscopic image are displayed.

7. The data processing method based on data visual interaction technology according to claim 6 is characterized in that: The method further comprises: The current surgical stage is determined based on the eye tracking information of the user, and the update frequency of the first target area and the second target area is adjusted based on the current surgical stage.

8. A data processing device based on data visual interaction technology, the device is applied to auxiliary mixed reality equipment for minimally invasive surgery, characterized in that: include: The anatomical structure display module is used to receive the intraoperative real-time image acquired by the image acquisition unit, and display the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; The virtual three-dimensional anatomical structure is generated based on a pre-acquired surgical area image; A virtual endoscopic image display module, used for displaying a virtual endoscopic image in a second target area based on the intraoperative real-time image, the user's eye tracking information and the real-time position tracking information of the endoscope; A display state adjustment module is used to adjust the display state of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscopic image in the second target area based on the user's eye tracking information until the operation is completed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data processing method based on data visual interaction technology as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method based on data visual interaction technology as described in any one of claims 1 to 7 is implemented.

Images