A data processing method and apparatus based on data visual interaction technology

Abstract

The present invention provides a data processing method and apparatus based on data visual interaction technology, which are applied to an assisted mixed reality device for minimally invasive surgery, and include: receiving real-time intraoperative images collected by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the real-time intraoperative images and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; displaying a virtual endoscope image in a second target area based on the real-time intraoperative images, the user's eye tracking information, and the real-time position tracking information of the endoscope; adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information until the surgery is completed, which can improve the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data processing method and apparatus based on data visual interaction technology. Background Art

[0002] With the rapid development of modern medicine, minimally invasive surgical techniques represented by endoscopic techniques have gradually been widely applied due to their advantages such as small trauma and quick recovery. However, during minimally invasive surgery, doctors can only observe the local conditions inside the patient's body through an endoscope, resulting in high surgical difficulty, low surgical efficiency, and high safety risks.

[0003] To solve the above problems, many researchers have begun to try to apply mixed reality (MR) technology to the minimally invasive surgery process to help doctors obtain clear anatomical structures inside the patient's body, so as to reduce the surgical difficulty and safety risks. However, existing mixed reality devices cannot real-time dynamically monitor the patient's organ tissues and surrounding tissues, and the virtual anatomical structures displayed by the mixed reality devices are likely to affect the doctor's line of sight, resulting in an increase rather than a decrease in surgical difficulty and safety risks. Summary of the Invention

[0004] The present invention provides a data processing method and apparatus based on data visual interaction technology to solve the problems in the prior art that existing mixed reality devices cannot real-time dynamically monitor the patient's organ tissues and surrounding tissues, and the virtual anatomical structures displayed by the mixed reality devices are likely to affect the doctor's line of sight, resulting in an increase rather than a decrease in surgical difficulty and safety risks, so as to improve the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risks of minimally invasive surgery.

[0005] On the one hand, the present invention provides a data processing method based on data visual interaction technology, and the method is applied to an assisted mixed reality device for minimally invasive surgery, including:

[0006] Receiving intraoperative real-time images collected by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images;

[0007] Based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in a second target area;

[0008] Based on the user's eye tracking information, adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area until the surgery is completed.

[0009] In an alternative embodiment of the present application, the step of presenting a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time image and the user's eye tracking information specifically includes:

[0010] Based on the intraoperative real-time image, determine the user's current field of view;

[0011] Based on the user's eye tracking information, determine the area of interest; the area of interest is an area within the current field of view;

[0012] Based on the area of interest of the user, determine the first target area, and present a matching virtual three-dimensional anatomical structure in the first target area.

[0013] In an alternative embodiment of the present application, the step of determining the first target area based on the area of interest of the user and presenting a matching virtual three-dimensional anatomical structure in the first target area specifically includes:

[0014] Based on the intraoperative real-time image, determine the surgical area within the user's current field of view;

[0015] Take the overlapping area of the surgical area within the user's current field of view and the area of interest as the first target area;

[0016] Determine the target virtual three-dimensional anatomical structure in the virtual three-dimensional anatomical structure that matches the first target area, and present the target virtual three-dimensional anatomical structure in the first target area.

[0017] In an alternative embodiment of the present application, the step of presenting a virtual endoscope image in a second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes:

[0018] Based on the intraoperative real-time image, determine the user's current field of view;

[0019] Based on the user's eye tracking information, determine the area of interest; the area of interest is an area within the current field of view;

[0020] Based on the real-time position tracking information of the endoscope, determine the second target area, and present a virtual endoscope image in the second target area.

[0021] In an alternative embodiment of the present application, the step of determining the second target area based on the real-time position tracking information of the endoscope and presenting a virtual endoscope image in the second target area specifically includes:

[0022] Based on the real-time position tracking information of the endoscope, determine the area where the endoscope is located within the user's current field of view;

[0023] Determine the surgical area in the user's current field of view based on the intraoperative real-time image;

[0024] Take the overlapping area between the surgical area in the user's current field of view and the area where the endoscope is located as the second target area, and display the virtual endoscope image in the second target area based on the pose of the endoscope.

[0025] In an alternative embodiment of the present application, the adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information specifically includes:

[0026] Update the first target area and the second target area based on the user's eye tracking information;

[0027] Based on the updated first target area and the second target area, display the updated virtual three-dimensional anatomical structure and the virtual endoscope image.

[0028] In an alternative embodiment of the present application, the method further includes:

[0029] Determine the current surgical stage based on the user's eye tracking information, and adjust the update frequencies of the first target area and the second target area based on the current surgical stage.

[0030] In a second aspect, the present invention further provides a data processing device based on data visual interaction technology, which is applied to an assisted mixed reality device for minimally invasive surgery, and includes:

[0031] An anatomical structure display module, configured to receive the intraoperative real-time image collected by the image acquisition unit, and display the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image;

[0032] A virtual endoscope image display module, configured to display the virtual endoscope image in the second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope;

[0033] A display state adjustment module, configured to adjust the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information until the surgery is completed.

[0034] In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the data processing method based on data visual interaction technology as described above.

[0035] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the data processing method based on the data visual interaction technology as described above.

[0036] A data processing method and device based on the data visual interaction technology provided by the present invention are applied to an assisted mixed reality device for minimally invasive surgery. By receiving real-time intraoperative images collected by an image acquisition unit, a matching virtual three-dimensional anatomical structure is displayed in a first target area based on the real-time intraoperative images and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; based on the real-time intraoperative images, the user's eye tracking information, and the real-time position tracking information of the endoscope, a virtual endoscope image is displayed in a second target area; based on the user's eye tracking information, the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area are adjusted until the surgery is completed, which can monitor the patient's organ tissues and surrounding tissues in real time and dynamically, and the virtual anatomical structure displayed by the mixed reality device is not likely to affect the doctor's line of sight, thereby improving the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0038] Figure 1 It is a schematic flowchart of a data processing method based on the data visual interaction technology provided by the present invention;

[0039] Figure 2 It is a schematic flowchart of the display process of the virtual three-dimensional anatomical structure provided by the present invention;

[0040] Figure 3 It is a schematic flowchart of the display process of the virtual endoscope image provided by the present invention;

[0041] Figure 4 It is a schematic flowchart of the adjustment process of the display state provided by the present invention;

[0042] Figure 5 It is a structural block diagram of a data processing device based on the data visual interaction technology provided by the present invention;

[0043] Figure 6 It is a schematic structural diagram of an electronic device provided by the present invention. Detailed implementation manners

[0044] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without any creative efforts shall fall within the protection scope of the present invention.

[0045] The following will illustrate the specific implementation process of the data processing method and device based on the data visual interaction technology in the embodiments of the present invention.

[0046] Figure 1 The flowchart of a data processing method based on the data visual interaction technology provided for the embodiments of the present invention is as Figure 1 shown. The method is applied to an assisted mixed reality device for minimally invasive surgery, and the method may include:

[0047] Step 101: Receive the intraoperative real-time image collected by the image acquisition unit, and display a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image.

[0048] Specifically, based on the foregoing content, it can be known that the existing mixed reality devices cannot monitor the patient's organ tissues and surrounding tissues in real time and dynamically, and the virtual anatomical structures displayed by the mixed reality devices are likely to affect the doctor's line of sight, resulting in an increase rather than a decrease in the surgical difficulty and safety risks. The root cause is that the existing mixed reality devices only mechanically display the pre-established three-dimensional anatomical structure model in the user's field of view without adjusting it according to the user's line of sight, resulting in the user being unable to correspond the three-dimensional anatomical structure model to the real patient's lesion, and thus unable to understand the situation of the patient's organ tissues and surrounding tissues in real time. Moreover, due to this mechanical display method, the virtual image blocks the user's line of sight, resulting in the user being unable to directly observe the real surgical situation, further increasing the surgical difficulty and safety risks. To solve this problem, the present invention proposes a data processing method based on the data visual interaction technology, which can adjust the display mode of virtual information based on the user's line of sight, thereby assisting the user to monitor the patient's organ tissues and surrounding tissues in real time and dynamically, and the virtual anatomical structures displayed by the mixed reality device are not likely to affect the user's line of sight. It can be understood that the user in the embodiments of the present invention specifically refers to a doctor performing minimally invasive surgery. The assisted mixed reality device for minimally invasive surgery includes:

[0049] A Head Mounted Display (HMD) is used to superimpose virtual images onto the user's field of view. Based on this, the user can see the combination of the real world and virtual elements.

[0050] Motion sensors are used to track the user's head and body movements to ensure that the relative position of the virtual image and the real world remains consistent.

[0051] An eye tracking sensor is used to obtain the user's eye tracking information, which includes the movement information of the user's eyes and the position information of the pupils.

[0052] An image acquisition unit is used to capture the user's field of view image, that is, the aforementioned intraoperative real-time image. Preferably, the image acquisition unit is integrated with the head mounted display and is specifically set at the central position of the front part of the head mounted display (i.e., the part corresponding to the user's face), and the height is close to the height of the user's eyes. Based on this, it can ensure that the viewing angle of the image acquisition unit matches the user's viewing angle to the greatest extent, and further ensure that the captured intraoperative real-time image is an accurate user's field of view image.

[0053] A computing unit is used to process images and data and perform real-time calculations. The computing unit can be an independent computer or a processor integrated in the head mounted display.

[0054] A software platform is used to provide support for MR applications, including development tools and applications. The software platform is responsible for managing the creation, display, and interaction of virtual content.

[0055] A network connection unit is used to achieve data transmission between the device and the cloud or other devices, supporting remote collaboration and real-time data update.

[0056] A power supply is used to provide the required power for the device to ensure that the device operates continuously during use.

[0057] The above components work together to enable the mixed reality device to provide accurate support information during the user's performance of minimally invasive surgery, improve the efficiency and success rate of the surgery, and reduce safety risks at the same time.

[0058] More specifically, after the minimally invasive surgery starts, the auxiliary mixed reality device of the embodiment of the present invention will receive the intraoperative real-time images collected by the image acquisition unit, and display the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time images and the user's eye tracking information; it can be understood that the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area images. The surgical area includes the lesion area and the surgical route area. The lesion area is the area where the lesion is located, that is, the target area that needs to be operated on. The surgical route area is the area corresponding to the route of the endoscope entering the lesion area. Based on this, the embodiment of the present invention can accurately display the virtual three-dimensional anatomical structure during the process of the endoscope entering the lesion area and during the surgical process, thereby helping the user to observe the accurate three-dimensional anatomical structure of the patient during the process of implanting the endoscope and during the surgical process, and further reducing the surgical difficulty and safety risk. The surgical route area is preferably obtained by simulating the surgical process based on the virtual three-dimensional anatomical structure. Based on this, the accuracy and safety of the surgical route area can be ensured.

[0059] Figure 2 Schematic diagram of the display process of the virtual three-dimensional anatomical structure provided by the present invention, as Figure 2 shown, the displaying of the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time images and the user's eye tracking information specifically includes:

[0060] Step 1011, determine the user's current visual field range based on the intraoperative real-time images;

[0061] Step 1012, determine the user's region of interest based on the user's eye tracking information; the region of interest is the area within the current visual field range;

[0062] Step 1013, determine the first target area based on the user's region of interest, and display the matching virtual three-dimensional anatomical structure in the first target area.

[0063] Specifically, it can be understood that the aforementioned image acquisition unit will continuously collect intraoperative real-time images. Based on this, the auxiliary mixed reality device of the embodiment of the present invention can continuously monitor the user's visual field range during the operation. It should be noted that most of the time after the operation starts, the user's line of sight will focus on the surgical area. During the process of implanting the endoscope, the user's line of sight will focus on the aforementioned surgical route area. During the surgical process, the user's line of sight will focus on the aforementioned lesion area. Based on this, after the embodiment of the present invention determines the user's current visual field range, it further determines the user's region of interest (i.e., the area where the user's line of sight focuses) based on the user's eye tracking information. It can be understood that the region of interest is the area within the current visual field range.

[0064] It should also be noted that although the user's line of sight mostly focuses on the surgical area after the operation starts, sometimes the attention will be distracted for various reasons, such as wiping sweat, retrieving surgical tools, or observing the endoscopic image. At this time, it is not necessary to display the aforementioned virtual three-dimensional anatomical structure and virtual endoscopic image so as not to affect the user's normal vision. Based on this, the embodiment of the present invention further determines a first target area based on the user's area of interest and displays a matching virtual three-dimensional anatomical structure in the first target area. More specifically, the determining the first target area based on the user's area of interest and displaying a matching virtual three-dimensional anatomical structure in the first target area specifically includes:

[0065] Determine the surgical area in the user's current field of view based on the intraoperative real-time image;

[0066] Use the overlapping area of the surgical area in the user's current field of view and the area of interest as the first target area;

[0067] Determine the target virtual three-dimensional anatomical structure in the virtual three-dimensional anatomical structure that matches the first target area, and display the target virtual three-dimensional anatomical structure in the first target area.

[0068] It should be noted that in the embodiment of the present invention, before the operation starts, the user will set a surgical area mark on the skin of the corresponding part of the patient's body based on the aforementioned pre-determined surgical area. Specifically, the surgical area mark can be generated by spraying with colored paint or any other method that can be recorded and recognized by an image. The embodiment of the present invention does not make specific limitations on this. Based on this, the embodiment of the present invention can quickly determine the surgical area in the user's current field of view based on the intraoperative real-time image through an image recognition algorithm, and then use the overlapping area of the surgical area in the user's current field of view and the area of interest as the first target area. On this basis, it is possible to quickly and accurately determine the target virtual three-dimensional anatomical structure in the virtual three-dimensional anatomical structure that matches the first target area and display the target virtual three-dimensional anatomical structure in the first target area. Based on this, it is possible to provide an accurate display of the virtual three-dimensional anatomical structure for the user on the basis of avoiding affecting the user's normal line of sight, so that the user can perform the operation accurately and efficiently and avoid safety risks.

[0069] Step 102, display a virtual endoscopic image in a second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope;

[0070] Specifically, through research, the embodiments of the present invention have found that to maximize the accuracy, efficiency, and safety of surgery, in addition to accurately displaying the virtual three-dimensional anatomical structure during the surgery, it is also necessary to accurately grasp the position and posture of the endoscope. Based on this, the embodiments of the present invention further display a virtual endoscope image in the second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope, so as to help the user understand the state of the endoscope at each stage after the surgery starts. More specifically, Figure 3 is a schematic diagram of the display process of the virtual endoscope image provided by the present invention, as Figure 3 shown, the displaying of the virtual endoscope image based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes:

[0071] Step 1021, determining the current visual field range of the user based on the intraoperative real-time image;

[0072] Step 1022, determining the area of interest of the user based on the user's eye tracking information; the area of interest is the area within the current visual field range;

[0073] Step 1023, determining the second target area based on the real-time position tracking information of the endoscope and displaying the virtual endoscope image in the second target area.

[0074] Specifically, for the display of the virtual endoscope image, similar to the display process of the virtual three-dimensional anatomical structure, it is also necessary to first determine the current visual field range of the user based on the intraoperative real-time image, and then determine the area of interest of the user based on the user's eye tracking information. The area of interest is the area within the current visual field range. Finally, determine the second target area based on the real-time position tracking information of the endoscope and display the virtual endoscope image in the second target area. It can be understood that the real-time position tracking information of the endoscope can be obtained through a positioning device arranged in the endoscope. The real-time position tracking information of the endoscope includes coordinate information and attitude information. Based on this, the embodiments of the present invention can quickly and accurately determine the display area and display attitude of the virtual endoscope image. It should be noted that the virtual endoscope image is a pre-constructed three-dimensional endoscope image. Based on this, it can help the user accurately determine the intraoperative state of the endoscope.

[0075] More specifically, the determining of the second target area based on the real-time position tracking information of the endoscope and the displaying of the virtual endoscope image in the second target area specifically includes:

[0076] Determining the area where the endoscope is located within the current visual field range of the user based on the real-time position tracking information of the endoscope;

[0077] Determine the surgical area in the user's current field of view based on the intraoperative real-time image;

[0078] Take the overlapping area of the surgical area in the user's current field of view and the area where the endoscope is located as the second target area, and display the virtual endoscope image in the second target area based on the pose of the endoscope.

[0079] It can be understood that, similar to the display of the virtual three-dimensional anatomical structure, the process of displaying the virtual endoscope image in the embodiments of the present invention is as follows: First, determine the area where the endoscope is located in the user's current field of view based on the real-time position tracking information of the endoscope, and then determine the surgical area in the user's current field of view based on the intraoperative real-time image; Finally, take the overlapping area of the surgical area in the user's current field of view and the area where the endoscope is located as the second target area, and display the virtual endoscope image in the second target area based on the pose of the endoscope. Based on this, on the basis of accurately displaying the position and pose of the virtual endoscope image, the accuracy and efficiency of minimally invasive surgery can be maximally improved, the safety risk can be reduced, and at the same time, the workload of displaying the virtual endoscope image can also be reduced.

[0080] Furthermore, on the basis of the above, the embodiments of the present invention can also display the patient's vital sign information such as blood pressure and heart rate through an assisted mixed reality device. Based on this, it is possible to avoid the user from frequently checking the corresponding vital sign monitoring instruments to determine the patient's condition, and further improve the surgical efficiency.

[0081] Step 103, based on the user's eye tracking information, adjust the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area until the operation is completed.

[0082] Specifically, Figure 4 is a schematic diagram of the adjustment process of the display state provided by the present invention. As Figure 4 shown, the adjustment of the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information specifically includes:

[0083] Step 1031, update the first target area and the second target area based on the user's eye tracking information;

[0084] Step 1032, display the updated virtual three-dimensional anatomical structure and virtual endoscope image based on the updated first target area and second target area.

[0085] As can be understood from the foregoing, as the surgical process progresses, the user's region of interest also changes. Correspondingly, the first target region and the second target region also change, and the target virtual three-dimensional anatomical structure and virtual endoscope image to be displayed also change. Based on this, the embodiments of the present invention continuously track the user's eye state and update the first target region and the second target region based on the user's eye tracking information. Furthermore, based on the updated first target region and second target region, the updated virtual three-dimensional anatomical structure and virtual endoscope image are displayed. Through the above update method, it can be ensured that the user can accurately monitor the state of the patient's organ tissues, surrounding tissues, and endoscope in real time dynamically after the surgery starts.

[0086] Further, the method further includes:

[0087] Determine the current surgical stage based on the user's eye tracking information, and adjust the update frequency of the first target region and the second target region based on the current surgical stage.

[0088] Based on the foregoing embodiments, the surgical stages of the embodiments of the present invention can be generally divided into an endoscope implantation stage and a lesion treatment stage. The present invention finds through research that since the endoscope implantation stage does not involve the lesion treatment process, the requirement for operation accuracy is relatively low. If the first target region and the second target region are frequently updated with a conventional update frequency, it will bring a huge workload to the assisted mixed reality device without generating positive benefits. On the contrary, for the lesion treatment stage, since it involves the lesion treatment process and the safety risk is high, only updating the first target region and the second target region based on the conventional update frequency will not meet the accuracy and safety requirements of the surgical process. To solve this problem, the embodiments of the present invention first determine the current surgical stage based on the user's eye tracking information, and then adjust the update frequency of the first target region and the second target region based on the current surgical stage.

[0089] More specifically, through research, the present invention discovers that there are obvious differences in the regions of interest of users during the endoscopic implantation stage and the lesion treatment stage. During the endoscopic implantation stage, due to previous operating habits, users will roughly estimate the position of the endoscope in the patient's body based on the implantation depth of the endoscope. At the same time, the user's line of sight will observe the physical changes of the patient to determine whether the position of the endoscope in the patient's body conforms to the estimate. Thus, it can be known that during the endoscopic implantation stage, the user's line of sight will focus on the aforementioned surgical route area for a long time. Based on this, the embodiments of the present invention can determine the duration for which the user's line of sight focuses on the surgical route area based on the user's eye tracking information, and then determine whether the current surgical stage is the endoscopic implantation stage. For example, when it is determined that the duration for which the user's line of sight focuses on the surgical route area is greater than the first preset duration threshold, it is determined that the current surgical stage is the endoscopic implantation stage. Similarly, during the lesion treatment stage, the present invention discovers through research that the user's line of sight will focus on the aforementioned lesion area for a long time. Based on this, the embodiments of the present invention can determine the duration for which the user's line of sight focuses on the lesion area based on the user's eye tracking information, and then determine whether the current surgical stage is the lesion treatment stage. For example, when it is determined that the duration for which the user's line of sight focuses on the lesion area is greater than the second preset duration threshold, it is determined that the current surgical stage is the lesion treatment stage. Based on this, the steps for the embodiments of the present invention to determine the current surgical stage are as follows:

[0090] Based on the user's eye tracking information and the intraoperative real-time image, determine the user's region of interest and the corresponding line of sight focusing duration. If the user's region of interest is the surgical route area and the duration for which the user's line of sight focuses on the surgical route area is greater than the first preset duration threshold, it is determined that the current surgical stage is the endoscopic implantation stage; if the user's region of interest is the lesion area and the duration for which the user's line of sight focuses on the lesion area is greater than the second preset duration threshold, it is determined that the current surgical stage is the lesion treatment stage. It can be understood that since the assisted mixed reality device records the generation time of the intraoperative real-time image, based on the analysis of consecutive intraoperative real-time images, the line of sight focusing duration of the user in any region of interest can be determined.

[0091] After determining the current surgical stage, the embodiments of the present invention can adjust the update frequencies of the first target area and the second target area. Specifically, if the current surgical stage is the endoscopic implantation stage, the update frequency is adjusted to the first frequency; if the current surgical stage is the lesion treatment stage, the update frequency is adjusted to the second frequency. Among them, the first frequency is less than the second frequency.

[0092] It can be understood that the first preset duration threshold, the second preset duration threshold, the first frequency, and the second frequency can all be adaptively adjusted according to different surgical types, and the embodiments of the present invention do not make specific limitations on this.

[0093] Furthermore, the auxiliary mixed reality device according to the embodiment of the present invention can also acquire and display the real-time image of the patient's body taken by the endoscope. It can be understood that the information displayed by the auxiliary mixed reality device at this time includes the real-time image of the patient's body, the virtual three-dimensional anatomical structure, and the virtual endoscope image. Based on this, the user can accurately grasp the three-dimensional state of the surgical area, the state of the endoscope, and the state of the surgical instruments in the endoscope (during the lesion treatment stage, the real-time image of the patient's body taken by the endoscope includes the surgical instruments). At the same time, the user does not need to observe the patient's internal situation through other display devices, thereby maximizing the accuracy and efficiency of minimally invasive surgery and minimizing the safety risk. On this basis, the embodiment of the present invention can also judge the posture of the endoscope through the real-time image of the patient's body and correct the virtual endoscope image to further improve the accuracy of the virtual endoscope image.

[0094] The solution provided by the present invention receives the intraoperative real-time image collected by the image acquisition unit, and displays the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image; based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope, the virtual endoscope image is displayed in the second target area; based on the user's eye tracking information, the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area are adjusted until the operation is completed, which can monitor the patient's organ tissues and surrounding tissues in real time and dynamically, and the virtual anatomical structure displayed by the mixed reality device is not likely to affect the doctor's line of sight, thereby improving the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery.

[0095] Figure 5 The following is a structural block diagram of a data processing device based on data visual interaction technology provided by an embodiment of the present invention, as Figure 5 shown. The device is applied to an auxiliary mixed reality device for minimally invasive surgery, and the device includes:

[0096] An anatomical structure display module 201, configured to receive the intraoperative real-time image collected by the image acquisition unit, and display the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time image and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on the pre-acquired surgical area image;

[0097] A virtual endoscope image display module 202, configured to display the virtual endoscope image in the second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope;

[0098] A display state adjustment module 203, configured to adjust the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the eye tracking information of the user until the operation is completed.

[0099] The solution provided by the present invention receives intraoperative real-time images collected by an image acquisition unit through an anatomical structure display module 201, and displays a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the eye tracking information of the user; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; a virtual endoscope image display module 202 displays a virtual endoscope image in a second target area based on the intraoperative real-time images, the eye tracking information of the user, and the real-time position tracking information of the endoscope; a display state adjustment module 203 adjusts the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the eye tracking information of the user until the operation is completed, can monitor the patient's organ tissues and surrounding tissues in real time and dynamically, and the virtual anatomical structure displayed by the mixed reality device is not likely to affect the doctor's line of sight, thereby improving the accuracy and efficiency of minimally invasive surgery on the basis of reducing the risk of minimally invasive surgery.

[0100] In an alternative embodiment of the present application, the displaying the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time images and the eye tracking information of the user specifically includes:

[0101] Based on the intraoperative real-time images, determine the current visual field range of the user;

[0102] Based on the eye tracking information of the user, determine the area of interest of the user; the area of interest is an area within the current visual field range;

[0103] Based on the area of interest of the user, determine the first target area, and display the matching virtual three-dimensional anatomical structure in the first target area.

[0104] In an alternative embodiment of the present application, the determining the first target area based on the area of interest of the user and displaying the matching virtual three-dimensional anatomical structure in the first target area specifically includes:

[0105] Based on the intraoperative real-time images, determine the surgical area within the current visual field range of the user;

[0106] Take the overlapping area of the surgical area within the current visual field range of the user and the area of interest as the first target area;

[0107] Determine the target virtual three-dimensional anatomical structure in the virtual three-dimensional anatomical structure that matches the first target area, and display the target virtual three-dimensional anatomical structure in the first target area.

[0108] In an alternative embodiment of the present application, presenting the virtual endoscope image in the second target area based on the intraoperative real-time image, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes:

[0109] Determining the user's current visual field range based on the intraoperative real-time image;

[0110] Determining the user's region of interest based on the user's eye tracking information; the region of interest is a region within the current visual field range;

[0111] Determining the second target area based on the real-time position tracking information of the endoscope and presenting the virtual endoscope image in the second target area.

[0112] In an alternative embodiment of the present application, determining the second target area based on the real-time position tracking information of the endoscope and presenting the virtual endoscope image in the second target area specifically includes:

[0113] Determining the area where the endoscope is located within the user's current visual field range based on the real-time position tracking information of the endoscope;

[0114] Determining the surgical area within the user's current visual field range based on the intraoperative real-time image;

[0115] Taking the overlapping area between the surgical area within the user's current visual field range and the area where the endoscope is located as the second target area, and presenting the virtual endoscope image in the second target area based on the pose of the endoscope.

[0116] In an alternative embodiment of the present application, adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information specifically includes:

[0117] Updating the first target area and the second target area based on the user's eye tracking information;

[0118] Presenting the updated virtual three-dimensional anatomical structure and virtual endoscope image based on the updated first target area and second target area.

[0119] In an alternative embodiment of the present application, the display state adjustment module 203 is further configured to perform the following operations:

[0120] Determining the current surgical stage based on the user's eye tracking information and adjusting the update frequencies of the first target area and the second target area based on the current surgical stage.

[0121] Figure 6 Illustrates a schematic physical structure diagram of an electronic device, such as Figure 6As shown in the figure, the electronic device may include: a processor 301, a communications interface 302, a memory 303, and a communication bus 304. Among them, the processor 301, the communications interface 302, and the memory 303 complete communication with each other through the communication bus 304. The processor 301 may call the logical instructions in the memory 303 to execute a data processing method based on data visual interaction technology. The method includes: receiving intraoperative real-time images collected by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in a second target area; based on the user's eye tracking information, adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area until the operation is completed.

[0122] In addition, when the logical instructions in the above-mentioned memory 303 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data processing method based on the data visual interaction technology provided by the above-mentioned various methods. The method includes: receiving intraoperative real-time images collected by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in a second target area; based on the user's eye tracking information, adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area until the operation is completed.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the data processing method based on the data visual interaction technology provided by the above-mentioned various methods. The method includes: receiving intraoperative real-time images collected by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the user's eye tracking information; the virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in a second target area; based on the user's eye tracking information, adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area until the operation is completed.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the data processing method based on data visual interaction technology described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on data visual interaction technology, the method is applied to an assisted mixed reality device for minimally invasive surgery, and is characterized in that, Including: Receiving intraoperative real-time images collected by an image acquisition unit, and displaying a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the user's eye tracking information; The virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; Based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope, displaying a virtual endoscope image in a second target area; Based on the user's eye tracking information, adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area until the operation is completed; The step of displaying a matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time images and the user's eye tracking information specifically includes: Based on the intraoperative real-time images, determining the user's current field of view; Based on the user's eye tracking information, determining the user's region of interest; the region of interest is an area within the current field of view; Determining a first target area based on the user's region of interest, and displaying a matching virtual three-dimensional anatomical structure in the first target area; The step of determining a first target area based on the user's region of interest and displaying a matching virtual three-dimensional anatomical structure in the first target area specifically includes: Based on the intraoperative real-time images, determining the surgical area within the user's current field of view; Taking the overlapping area between the surgical area within the user's current field of view and the region of interest as the first target area; Determining a target virtual three-dimensional anatomical structure in the virtual three-dimensional anatomical structure that matches the first target area, and displaying the target virtual three-dimensional anatomical structure in the first target area; The step of displaying a virtual endoscope image in the second target area based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes: Based on the intraoperative real-time images, determining the user's current field of view; Based on the user's eye tracking information, determining the user's region of interest; the region of interest is an area within the current field of view; Determining a second target area based on the real-time position tracking information of the endoscope, and displaying a virtual endoscope image in the second target area; The step of determining a second target area based on the real-time position tracking information of the endoscope and displaying a virtual endoscope image in the second target area specifically includes: Based on the real-time position tracking information of the endoscope, determining the area where the endoscope is located within the user's current field of view; Based on the intraoperative real-time images, determining the surgical area within the user's current field of view; Taking the overlapping area between the surgical area within the user's current field of view and the area where the endoscope is located as the second target area, and displaying a virtual endoscope image in the second target area based on the posture of the endoscope.

2. The data processing method based on data visual interaction technology according to claim 1, characterized in that The step of adjusting the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information specifically includes: Based on the user's eye tracking information, updating the first target area and the second target area; Based on the updated first target area and the second target area, displaying the updated virtual three-dimensional anatomical structure and the virtual endoscope image.

3. The data processing method based on data visual interaction technology according to claim 2, characterized in that The method further includes: Determining a current surgical stage based on the user's eye tracking information, and adjusting the update frequencies of the first target area and the second target area based on the current surgical stage.

4. A data processing device based on data visual interaction technology, the device being applied to an assisted mixed reality device for minimally invasive surgery, characterized in that, It includes: An anatomical structure display module, configured to receive intraoperative real-time images collected by an image acquisition unit, and display a matching virtual three-dimensional anatomical structure in a first target area based on the intraoperative real-time images and the user's eye tracking information; The virtual three-dimensional anatomical structure is generated based on pre-acquired surgical area images; A virtual endoscope image display module, configured to display a virtual endoscope image in a second target area based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope; A display state adjustment module, configured to adjust the display states of the virtual three-dimensional anatomical structure in the first target area and the virtual endoscope image in the second target area based on the user's eye tracking information until the surgery is completed; The displaying of the matching virtual three-dimensional anatomical structure in the first target area based on the intraoperative real-time images and the user's eye tracking information specifically includes: Determining the user's current visual field range based on the intraoperative real-time images; Determining the user's region of interest based on the user's eye tracking information; the region of interest is an area within the current visual field range; Determining a first target area based on the user's region of interest, and displaying a matching virtual three-dimensional anatomical structure in the first target area; The determining of the first target area based on the user's region of interest and displaying a matching virtual three-dimensional anatomical structure in the first target area specifically includes: Determining the surgical area within the user's current visual field range based on the intraoperative real-time images; Taking the overlapping area of the surgical area within the user's current visual field range and the region of interest as the first target area; Determining a target virtual three-dimensional anatomical structure in the virtual three-dimensional anatomical structure that matches the first target area, and displaying the target virtual three-dimensional anatomical structure in the first target area; The displaying of the virtual endoscope image in the second target area based on the intraoperative real-time images, the user's eye tracking information, and the real-time position tracking information of the endoscope specifically includes: Determining the user's current visual field range based on the intraoperative real-time images; Determining the user's region of interest based on the user's eye tracking information; the region of interest is an area within the current visual field range; Determining a second target area based on the real-time position tracking information of the endoscope, and displaying a virtual endoscope image in the second target area; The determining of the second target area based on the real-time position tracking information of the endoscope and displaying a virtual endoscope image in the second target area specifically includes: Determining the area where the endoscope is located within the user's current visual field range based on the real-time position tracking information of the endoscope; Determining the surgical area within the user's current visual field range based on the intraoperative real-time images; Taking the overlapping area of the surgical area within the user's current visual field range and the area where the endoscope is located as the second target area, and displaying a virtual endoscope image in the second target area based on the posture of the endoscope.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data processing method based on data visual interaction technology as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method based on data visual interaction technology as described in any one of claims 1 to 3.

Images