Information display method and device in operation process, storage medium and electronic equipment

Abstract

The invention discloses an information display method and device in an operation process, a storage medium and electronic equipment. The method comprises the following steps: acquiring operation auxiliary data, wherein the operation auxiliary data comprises two-dimensional data and / or three-dimensional data; receiving three-dimensional video data collected by the surgical robot in the surgical process; the operation auxiliary data and the three-dimensional video data are displayed on a display part of the VR glasses, a display page displayed by the display part comprises a first display area and a second display area, the first display area is used for displaying the three-dimensional video data, and the second display area is used for displaying the three-dimensional video data. The second display area is used for displaying the operation auxiliary data. The real-time three-dimensional video data and the operation auxiliary data collected by the operation robot are synchronously displayed in the display page displayed by the display equipment of the VR glasses, the watching mode of the three-dimensional video data and the operation auxiliary data is simplified, and the definition of data display is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a method, device, storage medium and electronic equipment for displaying information during a surgery. Background Art

[0002] In current surgical procedures, surgical robots can be used to perform surgery, improving surgical automation and surgical results. The surgical robot is equipped with an imaging system that can collect three-dimensional video data during the operation and display it on a display device (such as a monitor). Doctors can use 3D glasses to view the surgical images displayed on the display device.

[0003] In the process of implementing the present invention, it was found that there are at least the following technical problems in the prior art: multiple display devices are required during the operation, and other relevant information required for the operation can be displayed on other display devices, such as two-dimensional data or three-dimensional data. Doctors wearing 3D glasses will have a sense of visual excess when watching a 2D monitor, which affects the doctor's viewing clarity of data of different dimensions. Summary of the invention

[0004] The present invention provides a method, device, storage medium and electronic device for displaying information during a surgery, so as to realize synchronous display of three-dimensional video and surgical auxiliary data of the surgery process, and improve the clarity and viewing experience of displaying different data.

[0005] According to one aspect of the present invention, a method for displaying information during surgery is provided, which is applied to VR glasses, and the method comprises:

[0006] Acquiring surgical auxiliary data, wherein the surgical auxiliary data includes two-dimensional data and / or three-dimensional data;

[0007] Receiving three-dimensional video data collected by the surgical robot during the operation;

[0008] The surgical auxiliary data and the three-dimensional video data are displayed on a display component of the VR glasses, and the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

[0009] Optionally, the surgical auxiliary data is pre-configured; or, the surgical auxiliary data is transmitted by a surgical console during the operation, wherein the VR glasses are connected to the surgical console;

[0010] The surgical auxiliary data is data matching the surgical type, and includes one or more of the following: an organizational data model and surgical process-related data, wherein the surgical process-related data includes one or more of video, image, and text.

[0011] Optionally, the method further includes one or more of the following:

[0012] receiving a data control instruction transmitted by a surgical console, and executing a control operation on the surgical auxiliary data in response to the surgical control instruction;

[0013] In response to a voice control instruction of an operator, performing a control operation on the surgical assistance data;

[0014] identifying eye movement information of the operator, generating an eye movement control instruction based on the eye movement information, and performing a control operation on the surgical auxiliary data based on the eye movement control instruction;

[0015] The surgical stage of the surgical procedure is identified through the three-dimensional video data, and a control operation is performed on the surgical auxiliary data based on the identified surgical stage.

[0016] Optionally, the display page also includes a third display area;

[0017] The method also includes: receiving information transmitted by an associated device, and displaying the content of the information in the third display area, wherein the associated device is a terminal device that has established a binding relationship with the VR glasses in advance.

[0018] Optionally, the display component includes a rendering area and a perspective area, wherein the rendering area is the corresponding area in the display component of each display area in the display page; the perspective area is the area on the display component except the rendering area, and the perspective area sees through the surgical environment.

[0019] Optionally, the method further includes: in response to an adjustment operation on each display area in the display page, updating a display effect of each display area in the display page.

[0020] Optionally, the surgical robot is located locally and is equipped with a video transmitter;

[0021] Alternatively, the VR glasses are located at a remote end, and the three-dimensional video data collected by the surgical robot is transmitted to the VR glasses via a remote communication control module.

[0022] Optionally, the surgical robot is an endoscopic robot, and the three-dimensional video data is endoscopic video data.

[0023] According to another aspect of the present invention, there is provided an information display device during surgery, which is configured in VR glasses and includes:

[0024] A first data acquisition module, used for acquiring surgical auxiliary data, wherein the surgical auxiliary data includes two-dimensional data and / or three-dimensional data;

[0025] A second data acquisition module is used to receive three-dimensional video data collected by the surgical robot during the operation;

[0026] The display module is used to display the surgical auxiliary data and the three-dimensional video data on the display component of the VR glasses, and the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

[0027] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0028] at least one processor; and

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the information display method during a surgical procedure described in any embodiment of the present invention.

[0031] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the information display method during a surgical procedure described in any embodiment of the present invention when executed.

[0032] The technical solution of the embodiment of the present invention, by setting up VR glasses, synchronously displays the real-time three-dimensional video data and surgical auxiliary data collected by the surgical robot in the display page displayed by the display device of the VR glasses, so that the operator can watch the above-mentioned real-time three-dimensional video data and surgical auxiliary data on the same display page, simplifies the viewing method of the three-dimensional video data and surgical auxiliary data, and improves the clarity of data display.

[0033] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 is a flow chart of a method for displaying information during a surgery provided by Embodiment 1 of the present invention;

[0036] Figure 2 is a schematic diagram of a surgical system provided by an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of a surgical system provided by an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of a display page displayed by a display component of VR glasses provided by an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a display page displayed by a display component of VR glasses provided by an embodiment of the present invention;

[0040] Figure 6 It is a structural schematic diagram of an information display device during a surgical procedure provided by Embodiment 1 of the present invention;

[0041] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] Embodiment 1

[0045] Figure 1 This is a flowchart of a method for displaying information during surgery provided by Embodiment 1 of the present invention. This embodiment is applicable to the case where three-dimensional video data and surgical auxiliary data collected by a surgical robot are simultaneously displayed through VR glasses during surgery. The method can be executed by an information display device during surgery. The information display device during surgery can be implemented in the form of hardware and / or software, and the information display device during surgery can be configured in VR glasses. In this embodiment, the surgical system is configured with VR glasses for displaying the data content required to be displayed during surgery. The surgical system may include VR glasses, a surgical robot and a surgical console, wherein the surgical robot is a new type of medical device that integrates many disciplines such as clinical medicine, biomechanics, mechanics, computer science, and microelectronics. The surgical robot assists doctors in performing complex surgical operations through a clear imaging system and a flexible robotic arm, and completes operations such as intraoperative positioning, cutting, puncture, hemostasis, and suturing. Among them, the robotic arm of the surgical robot is configured with different surgical instruments, such as a scalpel. The imaging system configured by the surgical robot can be used to collect three-dimensional video data during surgery and provide data reference for the doctor's surgical process. It is understandable that different surgical robots can be used for different types of surgeries; or different surgical instruments can be configured on the surgical robot for different types of surgeries. The type of surgical robot is not limited here and can be determined according to the type of surgery. The above-mentioned surgical robot is equipped with a video acquisition component, such as a camera, for collecting three-dimensional video data of the surgical process. The configuration position of the video acquisition component in the surgical robot is not limited here and can be set according to the surgical requirements.

[0046] In some embodiments, the surgical robot may be an endoscopic robot, and accordingly, the three-dimensional video data collected by the endoscopic robot during the surgical process is endoscopic video data.

[0047] The surgical console can be connected to the surgical robot by wire or wirelessly, and is equipped with control components, including but not limited to buttons, armrests (or operating hands), touch pads, and foot control panels, etc. The doctor sends control instructions to the surgical robot by operating the control components on the surgical console to perform the surgical procedure.

[0048] In the above embodiment, the surgical system is also equipped with an imaging trolley, which can be used to display different types of medical images, such as but not limited to CT images and MRI images, etc. The imaging trolley can be integrated with a variety of medical image data processing equipment and display equipment.

[0049] In some embodiments, the surgical robot is a local device, that is, the surgical robot, the surgical console and the VR glasses are located in the same surgical space (for example, the same operating room), and the surgical robot is equipped with a video transmitter. The surgical robot transmits the collected three-dimensional video data to the VR glasses in real time through the video generator, so as to display it in real time in the VR glasses. Figure 2 , Figure 2 Schematic diagram of a surgical system provided by an embodiment of the present invention.

[0050] In some embodiments, the surgical console and VR glasses are located at the remote end, the surgical robot is located at the local end, and the remote end and the local end transmit data via a remote communication control module. Figure 3 , Figure 3 is a schematic diagram of a surgical system provided by an embodiment of the present invention, Figure 3 The surgical robot in the example is an endoscopic robot. The doctor is located at the remote end, and the target object for the operation is located at the local end. The doctor performs the operation through the surgical console at the remote end, and transmits the control instructions to the surgical robot at the local end through the remote communication control module. The surgical robot responds to gesture instructions and performs the operation; and collects three-dimensional video data of the operation in real time, and transmits the three-dimensional video data to the VR glasses at the remote end through the remote communication control module to show it to the doctor. Both the remote end and the local end are equipped with a remote communication control module, and the remote communication control modules respectively configured at the remote end and the local end are connected through a network.

[0051] In this embodiment, the data displayed during the operation also includes surgical auxiliary data, which can be understood as data displayed to the doctor in order to assist in the execution of the operation. The surgical auxiliary data can be two-dimensional data and / or three-dimensional data. In view of the synchronous display of three-dimensional video data and surgical auxiliary data collected by the surgical robot, this embodiment provides a method for displaying information during the operation, which is applied to VR glasses, and the VR glasses display the three-dimensional video data and surgical auxiliary data collected by the surgical robot.

[0052] like Figure 1 As shown, the method includes:

[0053] S110 , acquiring surgical auxiliary data, where the surgical auxiliary data includes two-dimensional data and / or three-dimensional data.

[0054] S120: Receive three-dimensional video data collected by the surgical robot during the operation.

[0055] S130. Display the surgical auxiliary data and the three-dimensional video data on a display component of the VR glasses, wherein the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

[0056] The surgical auxiliary data is data matching the surgical type, including one or more of the following: an organizational data model and surgical process-related data, wherein the surgical process-related data includes one or more of video, image, and text. The data form of the surgical auxiliary data can be one or more of a three-dimensional video form, a two-dimensional video form, an image form, a text form, and an organizational data model.

[0057] The surgical auxiliary data in the form of three-dimensional video or two-dimensional video can be the demonstration video data corresponding to the surgical type of the current surgical procedure, for example, it can be the historical video data of the historical surgical procedure corresponding to the surgical type. Exemplarily, the demonstration video data can be screened out from the historical video data of the historical surgical procedure corresponding to the surgical type. Specifically, the medical record information of the target object is matched with the medical record information of the historical object of the surgical type, and the medical record information of the historical object with the highest similarity is matched, and the historical video data corresponding to the medical record information of the matched historical object is used as the demonstration video data. Exemplarily, the demonstration video data can be screened out by an expert from the historical video data of the historical surgical procedure corresponding to the surgical type. Exemplarily, the historical video data of the historical surgical procedure corresponding to the surgical type can be evaluated based on the surgical evaluation model to obtain the evaluation result of the historical video data, and the evaluation result can be in the form of a score, and the historical video data corresponding to the optimal evaluation result is used as the demonstration evaluation data. A database is maintained in advance, and the database can include the demonstration video data corresponding to each surgical type, and the corresponding demonstration video data can be called from the database according to the surgical type of the current surgical procedure.

[0058] In some embodiments, the surgical process may include multiple surgical stages. The above database may include demonstration video data of multiple surgical stages corresponding to each surgical type. The demonstration video data may be determined in the above database according to the surgical type and surgical stage.

[0059] The surgical auxiliary data in the form of images may include but are not limited to medical images of the target object, demonstration images of surgical organs, etc. The medical images of the target object may include but are not limited to CT images and MRI images, etc.

[0060] The text-based surgical auxiliary data may be text prompt information corresponding to the surgical type, and the text prompt information may include but is not limited to surgical step prompt information and precaution prompt information, etc. The tissue data model may be a three-dimensional data model of the surgical organ, such as a three-dimensional data model of the heart, a three-dimensional data model of the lungs, etc.

[0061] The surgical auxiliary data may include at least one of the above-mentioned different forms of data. The surgical auxiliary data is pre-configured, for example, pre-configured before the start of the surgical procedure. Optionally, the surgical auxiliary data is called from a database through VR glasses; optionally, the surgical auxiliary data is called from a database through a surgical console and the surgical auxiliary data is transmitted to the VR glasses; optionally, the surgical auxiliary data may be externally imported into the VR glasses.

[0062] Optionally, surgical assistance data is also transmitted via a surgical console during surgery.

[0063] The VR glasses receive the three-dimensional video data collected by the surgical robot in real time, and synchronously display the three-dimensional video data and surgical auxiliary data on the display component of the VR glasses. The display component of the VR glasses can be the lenses of the VR glasses. A display page is displayed on the display component of the VR glasses, and the display page is used to display the three-dimensional video data and the surgical auxiliary data. The display page includes a first display area and a second display area. The first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data, thereby realizing the synchronous display of the three-dimensional video data and the surgical auxiliary data on the same display page. Exemplarily, Figure 4 It is a schematic diagram of a display page displayed by the display component of the VR glasses provided in an embodiment of the present invention.

[0064] The rendering method of the first display area is a three-dimensional data rendering method, which can display the three-dimensional video data transmitted in real time by the surgical robot. The rendering method of the second display area is a three-dimensional data rendering method or a two-dimensional data rendering method. Accordingly, the surgical auxiliary data displayed in the second display area can be two-dimensional data or three-dimensional data. By using different data rendering methods in different display areas on the same display page, it is possible to display two-dimensional data and / or three-dimensional data in the same display area, thereby improving data clarity.

[0065] Optionally, the second display area includes a plurality of sub-areas, each of which can be used to display one item of the surgical auxiliary data, and the number of sub-areas in the second display area can be determined according to the number of data items of the surgical auxiliary data displayed. Any sub-area can display two-dimensional data or three-dimensional data, and different sub-areas can correspond to different rendering methods, for example, a three-dimensional data rendering method and a two-dimensional data rendering method.

[0066] The rendering mode corresponding to the second display area or any sub-area in the second display area can be determined according to the data type of the surgical auxiliary data displayed in the second display area or the sub-area in the second display area, and the data type includes a two-dimensional data type and a three-dimensional data type. A corresponding relationship between the display area or the sub-area and the displayed data is established, a rendering mode of the display area or the sub-area is determined based on the data type of the displayed data, and data is displayed based on the determined rendering mode.

[0067] In the above embodiment, before data display is performed (for example, before the operation begins), in response to the adjustment operation of each display area in the display page, the display effect of each display area in the display page is updated. The above adjustment operation may include one or more of a position adjustment operation, a size adjustment operation and a screen attribute adjustment operation, wherein the position adjustment operation can be used to adjust the layout of the display area (or sub-area) in the display page, and update the display position of one or more display areas (or sub-areas). The size adjustment operation is used to set the size information of one or more display areas (or sub-areas) in the display page, and the screen attribute adjustment operation is used to set the screen attribute information of one or more display areas (or sub-areas) in the display page, wherein the screen attribute information includes but is not limited to attribute information such as brightness, hue and saturation.

[0068] For example, the first display area can be set at the center of the display screen through the position adjustment operation, or the distribution of the second display area on the display page can be adjusted. For example, the area size of the first display area can be increased through the size adjustment operation to facilitate accurate capture of the surgical screen. For example, the screen brightness of the first display area or the second display area can be increased through the screen attribute adjustment operation to improve the clarity of the screen.

[0069] The above adjustment operation can be achieved through the adjustment buttons on the VR glasses, or through voice adjustment, or through touch or external control, which is not limited here. The display effect of each display area in the display page is set through the adjustment operation to meet the doctor's data display needs and meet the personalized needs of different doctors.

[0070] In the above embodiment, the surgical auxiliary data can also be controlled during the operation to meet the control requirements of the surgical auxiliary data during the operation, wherein the control of the surgical auxiliary data includes but is not limited to deleting the surgical auxiliary data from the display page, playing the surgical auxiliary data in video form, stopping the playing of the surgical auxiliary data, and switching the surgical auxiliary data in the second display area.

[0071] Optionally, the control of surgical auxiliary data may be implemented by: receiving a data control instruction transmitted by a surgical console, and performing a control operation on the surgical auxiliary data in response to the surgical control instruction. The surgical console is provided with a control component for VR glasses. The operator generates a data control instruction by operating the control component, and transmits the data control instruction to the VR glasses. The VR glasses receive and respond to the data control instruction, and perform a control operation on the surgical auxiliary data. The operator here may be a doctor or a doctor's assistant.

[0072] For example, the surgical auxiliary data displayed by the VR glasses can be synchronously displayed on the surgical console, or the surgical auxiliary data information displayed by the VR glasses can be displayed on the surgical console, and the operator can operate on the surgical console according to the control requirements of the surgical auxiliary data. Among them, the control requirements for surgical auxiliary data can be conveyed by the doctor through voice. The operator is informed of the control requirements for surgical auxiliary data, and operates according to the surgical auxiliary data or the information of surgical auxiliary data displayed on the surgical console to generate data control instructions.

[0073] Optionally, the implementation method of controlling the surgical auxiliary data may be: in response to the voice control instruction of the operator, the control operation is performed on the surgical auxiliary data. In this embodiment, the operator may be a wearer of VR glasses, for example, a doctor. The demonstration voice information of the operator is collected in advance in the VR glasses, and the voice features of the operator are extracted and stored through the demonstration voice information. When the voice information is detected, the collected voice information is matched with the pre-extracted voice features to determine whether the speaker corresponding to the voice information is an operator with control authority. If the speaker corresponding to the voice information is an operator with control authority, the semantics in the voice information is analyzed, and a voice control instruction is generated, and the above-mentioned voice control instruction is executed to realize the control operation on the surgical auxiliary data. Exemplarily, the voice control instruction may be "delete xx data", or "play xx video data", or "delete data 1", etc. Here, "data 1", "xx data" and "xx video data" are the identifiers of the surgical auxiliary data, which may be the name of the surgical auxiliary data or a pre-set ID. The identifier of the surgical auxiliary data is not limited here, and can be used to uniquely mark the surgical auxiliary data.

[0074] Exemplarily, the surgical auxiliary data may include video data of a certain surgical stage of a surgical type that matches the surgical procedure. The above video data may be played when the surgical procedure enters the above surgical stage or before entering the above surgical stage, and the video playback may be stopped after the above surgical stage ends. Therefore, based on the operator's demand for playing the above video data, the operator's voice control instructions are detected and control operations are performed on the surgical auxiliary data.

[0075] Optionally, the implementation method of controlling the surgical auxiliary data may be: identifying the eye movement information of the operator, generating an eye movement control instruction based on the eye movement information, and performing a control operation on the surgical auxiliary data based on the eye movement control instruction. The VR glasses here are head-mounted devices with the function of recognizing eye movement information. The operator's intention can be determined according to the operator's eye movement information, and an eye movement control instruction can be generated. Specifically, the surgical auxiliary data selected by the operator is determined according to the direction information in the eye movement information, and the eye movement control instruction for the selected surgical auxiliary data is generated according to the gaze duration or blinking action in the eye movement information. Exemplarily, when it is determined that surgical auxiliary data A (for example, video data) is selected according to the direction information in the eye movement information, the gaze duration in the eye movement information exceeds the preset duration, and a play instruction for surgical auxiliary data A is generated. There may be multiple eye movement control instructions here, and correspondingly, different eye movement control instructions correspond to different eye movement information, which is not limited here.

[0076] Optionally, the implementation method of controlling the surgical auxiliary data may be: identifying the surgical stage of the surgical process through the three-dimensional video data, and performing a control operation on the surgical auxiliary data based on the identified surgical stage. Here, the control operation performed on the surgical auxiliary data may be to display the surgical auxiliary data matching the surgical stage. Exemplarily, the surgical auxiliary data may include data corresponding to multiple surgical stages, and the surgical stage corresponding to the three-dimensional video data is determined by analyzing the three-dimensional video data collected by the surgical robot, and the surgical auxiliary data corresponding to the surgical stage is controlled to be displayed. Exemplarily, the machine learning model obtained by pre-training can be used to analyze the surgical auxiliary data through the machine learning model to obtain the surgical stage output by the machine learning model. Here, the machine learning model can output a surgical stage identifier, and the surgical auxiliary data corresponding to the surgical stage identifier is determined through the surgical stage identifier. Exemplarily, the surgical auxiliary data may be demonstration video data, which includes video data corresponding to multiple surgical stages, and the video data corresponding to the surgical stage identifier can be played through the identified surgical stage identifier to achieve accurate display of the surgical auxiliary data and provide data reference for the operator.

[0077] The technical solution of this embodiment controls the surgical auxiliary data through one or more of the above-mentioned methods, thereby realizing intelligent control of the surgical auxiliary data during the operation and meeting the operator's demand for display of the surgical auxiliary data.

[0078] Based on the above embodiment, the display page further includes a third display area, which is used to display information transmitted by the associated device; for example, see Figure 5 , Figure 5 It is a schematic diagram of a display page displayed by the display component of the VR glasses provided in an embodiment of the present invention.

[0079] During the operation, the VR glasses can receive information transmitted by the associated device and display the content of the information in the third display area, wherein the associated device is a terminal device that has established a binding relationship with the VR glasses in advance. The terminal device that has established a binding relationship with the VR glasses can be a doctor's terminal device. Optionally, the sender of the emergency message is pre-set in the terminal device. When the terminal device receives a message sent by the sender of the emergency message, the message sent by the sender of the emergency message is transmitted to the VR glasses and displayed in the third display area, so as to facilitate the display of the emergency information to the operator.

[0080] In the above embodiment, the display component includes a rendering area and a perspective area, wherein the rendering area is the corresponding area of ​​each display area in the display page in the display component; the perspective area is the area on the display component except the rendering area, and the perspective area sees through the surgical environment.

[0081] The rendering area may be a corresponding area of ​​the first display area, the second display area, and the third display area in the display component. It is understandable that the first display area, the second display area, and the third display area may be updated in position and size according to the adjustment operation, and accordingly, the rendering area is variable and is determined according to the position of each of the above display areas.

[0082] The area outside the rendering area in the display component is the perspective area, which is in a perspective state. The operator can view the surgical environment through the perspective area. Accordingly, the operator can view the surgical equipment and other operators in the surgical environment, operate other surgical equipment, and interact with other operators.

[0083] In the above embodiment, the three-dimensional video data is also stored through VR glasses, which can be used for subsequent queries or as a teaching video, etc.

[0084] The technical solution of this embodiment, by setting up VR glasses, synchronously displays the real-time three-dimensional video data and surgical auxiliary data collected by the surgical robot in the display page displayed by the display device of the VR glasses, so that the operator can watch the above-mentioned real-time three-dimensional video data and surgical auxiliary data on the same display page, which simplifies the viewing method of three-dimensional video data and surgical auxiliary data and improves the clarity of data display.

[0085] Figure 6 FIG. 1 is a schematic diagram of a device for displaying information during a surgical procedure provided by an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0086] A first data acquisition module 210, used to acquire surgical auxiliary data, where the surgical auxiliary data includes two-dimensional data and / or three-dimensional data;

[0087] A second data acquisition module 220 is used to receive three-dimensional video data collected by the surgical robot during the operation;

[0088] The display module 230 is used to display the surgical auxiliary data and the three-dimensional video data on the display component of the VR glasses, and the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

[0089] The technical solution of this embodiment, by setting up VR glasses, synchronously displays the real-time three-dimensional video data and surgical auxiliary data collected by the surgical robot in the display page displayed by the display device of the VR glasses, so that the operator can watch the above-mentioned real-time three-dimensional video data and surgical auxiliary data on the same display page, which simplifies the viewing method of three-dimensional video data and surgical auxiliary data and improves the clarity of data display.

[0090] Based on the above embodiment, optionally, the surgical auxiliary data is pre-configured; or, the surgical auxiliary data is transmitted by a surgical console during the operation, wherein the VR glasses are connected to the surgical console;

[0091] The surgical auxiliary data is data matching the surgical type, and includes one or more of the following: an organizational data model and surgical process-related data, wherein the surgical process-related data includes one or more of video, image, and text.

[0092] Based on the above embodiment, optionally, the device further includes a control module, which is used to perform one or more of the following: receiving a data control instruction transmitted by a surgical console, and performing a control operation on the surgical auxiliary data in response to the surgical control instruction;

[0093] In response to a voice control instruction of an operator, performing a control operation on the surgical assistance data;

[0094] identifying eye movement information of the operator, generating an eye movement control instruction based on the eye movement information, and performing a control operation on the surgical auxiliary data based on the eye movement control instruction;

[0095] The surgical stage of the surgical procedure is identified through the three-dimensional video data, and a control operation is performed on the surgical auxiliary data based on the identified surgical stage.

[0096] Based on the above embodiment, optionally, the display page further includes a third display area;

[0097] The display module 230 is further used to receive information transmitted by an associated device and display the content of the information in the third display area, wherein the associated device is a terminal device that has established a binding relationship with the VR glasses in advance.

[0098] Based on the above embodiment, optionally, the display component includes a rendering area and a perspective area, wherein the rendering area is the corresponding area of ​​each display area in the display page in the display component; the perspective area is the area on the display component except the rendering area, and the perspective area sees through the surgical environment.

[0099] On the basis of the above embodiment, optionally, the display module 230 is further configured to update the display effect of each display area in the display page in response to an adjustment operation on each display area in the display page.

[0100] Based on the above embodiments, optionally, the surgical robot is located locally and is equipped with a video transmitter; or, the VR glasses are located remotely and the three-dimensional video data collected by the surgical robot is transmitted to the VR glasses via a remote communication control module; the surgical robot is an endoscopic robot and the three-dimensional video data is endoscopic video data.

[0101] The information display device during a surgery provided by an embodiment of the present invention can execute the information display method during a surgery provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0102] Figure 71 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0103] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0104] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a method for displaying information during surgery.

[0106] In some embodiments, the information display method during surgery can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the information display method during surgery described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the information display method during surgery in any other appropriate manner (for example, by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The computer programs for implementing the information display method during the surgical procedure of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a method for displaying information during a surgical procedure, the method comprising:

[0110] Acquire surgical auxiliary data, wherein the surgical auxiliary data includes two-dimensional data and / or three-dimensional data; receive three-dimensional video data collected by the surgical robot during the operation; display the surgical auxiliary data and the three-dimensional video data on the display component of the VR glasses, and the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for displaying information during surgery, characterized in that: Applied to VR glasses, the method comprises: Acquiring surgical auxiliary data, wherein the surgical auxiliary data includes two-dimensional data and / or three-dimensional data; Receiving three-dimensional video data collected by the surgical robot during the operation; The surgical auxiliary data and the three-dimensional video data are displayed on a display component of the VR glasses, and the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

2. The method according to claim 1, characterized in that The surgical auxiliary data is pre-configured; or, the surgical auxiliary data is transmitted by a surgical console during the operation, wherein the VR glasses are connected to the surgical console; The surgical auxiliary data is data matching the surgical type, and includes one or more of the following: an organizational data model and surgical process-related data, wherein the surgical process-related data includes one or more of video, image, and text.

3. The method according to claim 1, characterized in that The method may further include one or more of the following: receiving a data control instruction transmitted by a surgical console, and executing a control operation on the surgical auxiliary data in response to the surgical control instruction; In response to a voice control instruction of an operator, performing a control operation on the surgical assistance data; identifying eye movement information of the operator, generating an eye movement control instruction based on the eye movement information, and performing a control operation on the surgical auxiliary data based on the eye movement control instruction; The surgical stage of the surgical procedure is identified through the three-dimensional video data, and a control operation is performed on the surgical auxiliary data based on the identified surgical stage.

4. The method according to claim 1, characterized in that: The display page also includes a third display area; The method also includes: receiving information transmitted by an associated device, and displaying the content of the information in the third display area, wherein the associated device is a terminal device that has established a binding relationship with the VR glasses in advance.

5. The method according to any one of claims 1 to 4, characterized in that: The display component includes a rendering area and a perspective area, wherein the rendering area is the corresponding area of ​​each display area in the display page in the display component; the perspective area is the area on the display component except the rendering area, and the perspective area sees through the surgical environment.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to the adjustment operation on each display area in the display page, the display effect of each display area in the display page is updated.

7. The method according to claim 1, characterized in that The surgical robot is located locally and is equipped with a video transmitter; or the VR glasses are located remotely and the three-dimensional video data collected by the surgical robot is transmitted to the VR glasses via a remote communication control module; The surgical robot is an endoscopic robot, and the three-dimensional video data is endoscopic video data.

8. An information display device during surgery, characterized in that: Configured in VR glasses, including: A first data acquisition module, used for acquiring surgical auxiliary data, wherein the surgical auxiliary data includes two-dimensional data and / or three-dimensional data; A second data acquisition module is used to receive three-dimensional video data collected by the surgical robot during the operation; The display module is used to display the surgical auxiliary data and the three-dimensional video data on the display component of the VR glasses, and the display page displayed by the display component includes a first display area and a second display area, wherein the first display area is used to display the three-dimensional video data, and the second display area is used to display the surgical auxiliary data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for displaying information during a surgical procedure according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the information display method during a surgical procedure according to any one of claims 1 to 7 when executed.

Images