A device and method for integrated display of surgical robot data

Abstract

The present invention relates to a device and method for integrating and displaying data of a surgical robot, comprising: a data acquisition unit, a data processing unit, a display unit, an external storage unit, and a communication unit; the data acquisition unit comprises a video acquisition unit and an information acquisition unit; the data processing unit comprises a signal conversion unit, an internal storage unit, a video synthesis unit, and a backup unit; the signal conversion unit is used for converting data collected by the data acquisition unit and data output by the data processing unit; the video synthesis unit is used for synthesizing the data collected by the data acquisition unit; the display unit is used for displaying data processed by the data processing unit; the external storage unit is used for storing data processed by the data processing unit; the communication unit is used for data communication among the data acquisition unit, the data processing unit, the display unit, and the external storage unit, and the communication includes wireless communication and wired communication.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and in particular to a device and method for integrating and displaying surgical robot data. Background Art

[0002] With the rapid development of robot technology, especially the development of surgical robot technology, the application scope and demand of surgical robots are also expanding. Surgical robots are stable and precise. Doctors can use surgical robots to complete high-precision surgical movements and reduce the chance of bacterial infection, thereby improving the quality of surgery. Existing surgical robot equipment is large and has many displays, which is not conducive to timely exploration during surgery by doctors; and it is inconvenient for medical staff to communicate in real time when they are at a long distance. Since the existing endoscopic video is directly connected to the 3D display, and the surgical status is connected to another separate display, the disadvantage is that there are too many screens, and the medical staff in the operating room cannot obtain all the information in time; in addition, the existing technology uses commercial video capture cards to send the endoscopic video acquisition to the industrial computer for synthesis processing with the surgical status, and then transmit it to the video capture card for display. The disadvantage is that the system is large in size and the delay is uncontrollable. The delay during surgery will cause medical accidents, so it is very important to control the time. Summary of the invention

[0003] The purpose of the present invention is to propose a device and method for integrated display of surgical robot data in view of the prior art.

[0004] Specifically, a device for integrating and displaying surgical robot data includes: a data acquisition unit, a data processing unit, a display unit, an external storage unit, and a communication unit;

[0005] The data acquisition unit includes a video acquisition unit and an information acquisition unit;

[0006] The data processing unit includes FPGA and DSP; the FPGA includes a first signal conversion unit, an internal storage unit, a video synthesis unit, and a backup unit, and the DSP includes a second signal conversion unit and an image conversion unit; the first signal conversion unit is used to convert the data sent by the video acquisition unit and the data output by the FPGA received by the FPGA; the second signal conversion unit is used to convert the data sent by the information acquisition unit and the data output by the DSP received by the DSP; the image conversion unit is used to convert the data received by the DSP into image data; the video synthesis unit is used to synthesize the data received by the FPGA; the display unit is used to display the data processed by the received data processing unit, and the display unit includes a local display and / or a remote display;

[0007] The external storage unit is used to store the data processed by the data processing unit;

[0008] The communication unit is used for data communication among the data acquisition unit, the data processing unit, the display unit and the external storage unit, and the communication includes wireless communication and wired communication.

[0009] Furthermore, the video acquisition unit is installed on the endoscope at the operation end, and is used to collect the video inside the operation object in real time; the information acquisition unit is installed at the control end, and is used to collect gesture instructions and / or typed instructions in real time. The endoscope includes a gastroscope, a colonoscope, a cystoscope, a bronchoscope, a thoracoscope, a laparoscope, a laryngoscope, a nasopharyngeal mirror, a colposcope, a hysteroscope, and other devices that can directly observe the state of the internal tissues and organs of the human body.

[0010] Furthermore, the first signal conversion unit is used to convert the received signal of the video acquisition unit into an RGB signal; the second signal conversion unit is used to decode the signal received by the DSP and encode the signal sent by the DSP; the internal storage unit realizes temporary storage of data and high-speed access; the video synthesis unit synthesizes the received data collected by the video acquisition unit and the data sent by the DSP; the local display displays the data synthesized by the video synthesis unit; the remote display displays the synthesized video sent by the DSP; and the external storage unit stores the synthesized video sent by the DSP.

[0011] Furthermore, the data collected by the video acquisition unit communicates with the FPGA; the data collected by the information acquisition unit communicates with the DSP; the first signal conversion unit converts the signal of the data collected by the video acquisition unit into an RGB signal; the DSP processes the received instruction data into image data and sends it to the PFGA; the image data sent by the DSP is stored in the internal storage unit; the video synthesis unit synthesizes the video data stored in the internal storage unit and the image data sent by the DSP; the first signal conversion unit converts the synthesized data into video data in RGB format; the backup unit backs up the video data, transmits the video data to the local display unit for display, and transmits the video data to the DSP; the DSP transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display.

[0012] Furthermore, it also includes a voice output unit, which is installed at the surgical end and the control end or at the surgical end and the remote control end, communicates with the DSP, and is used to output voice; the information acquisition unit also includes an audio acquisition unit, and the audio acquisition unit includes a first audio acquisition unit and a second audio acquisition unit; the communication unit also includes data communication between the data acquisition unit, the data processing unit, the display unit, the external storage unit and the voice output unit, and the communication includes wireless communication and wired communication.

[0013] Specifically, a method for integrating and displaying surgical robot data includes: a data acquisition unit acquires data, and communicates the acquired data with a data processing unit through a communication unit; the data processing unit converts the data acquired by the data acquisition unit and synthesizes and backs up the data; the data processing unit converts the synthesized data and communicates with an external storage unit and / or a display unit through a communication unit.

[0014] Furthermore, the data acquisition unit collects data and communicates the collected data with the data processing unit through the communication unit. The specific steps are: the video acquisition unit collects video data and sends the collected video data to the FPGA through the communication unit; the information acquisition unit collects gesture instructions and / or typed instructions and sends the collected data to the DSP through the communication unit.

[0015] Furthermore, the data processing unit converts the data collected by the data collection unit and synthesizes and backs up the data. The specific steps are: the data processing unit receives the video data and data sent to the data processing unit through the communication unit, the signal conversion unit converts the video data and data into RGB signal data, and stores them in the internal storage unit, so as to realize the movement of a large amount of data between the internal storage unit and the bus-based video processing IP; the data in the internal storage unit is extracted and the video is synthesized by the video synthesis unit, and the synthesized video is backed up.

[0016] Furthermore, the video acquisition unit acquires video data and sends the acquired video data to the FPGA through the communication unit; the information acquisition unit acquires gesture instructions and / or input instructions and sends the acquired instructions to the DSP; the FPGA converts the received video data signal into an RGB signal through the first signal conversion unit; the RGB data is sent to the bus through the Video to AXI4 IP module; and the VDMA (Video Direct Memory The Access video register direct access and access) IP module stores RGB data in the internal storage unit; realizes the movement of large amounts of data between the system internal storage unit and the bus-based video processing IP; the DSP receives the instruction data collected by the information acquisition unit, processes the instruction data received by the DSP into image data, and sends it to the PFGA through the GPMC bus; the PFGA receives the image data sent by the DSP and stores it in the internal storage unit; the data stored in the internal storage unit is extracted through the VDMA IP module, and the video data stored in the internal storage unit and the data are synthesized into new video data through the video synthesis unit; the first signal conversion unit converts the synthesized video into video data in RGB format; the backup unit backs up the synthesized new video data, sends it to the local display unit for display, and sends the synthesized video data to the DSP; the DSP transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display.

[0017] Furthermore, the first audio acquisition unit acquires audio data from the surgical end in real time, the second audio acquisition unit acquires audio data from the control end in real time, and communicates the acquired data with the DSP through the communication unit; the DSP compresses the audio data acquired by the first audio acquisition unit and sends it to the surgical end through the communication unit, and the surgical end voice output unit outputs the audio, the DSP decompresses the audio data acquired by the second audio acquisition unit and sends it to the remote control end through the communication unit, and the remote control end voice output unit outputs the audio; the data processing unit communicates with the external storage unit, the display unit and the voice output unit through the communication unit.

[0018] The advantages of the present invention are:

[0019] Integrate and display endoscopic video and surgical status information to reduce the number of displays; use FPGA and DSP integrated chips to synthesize endoscopic video and surgical status information, with high integration, small size, and low latency, reducing the probability of medical accidents caused by multi-screen observation by medical staff during surgery and missing important information, and reducing medical accidents caused by delays in intraoperative instructions; audio transmission solves the problem of inconvenient information exchange between medical staff at the control end and the execution end; audio and video backup during the operation facilitates subsequent accident investigation, teaching purposes, etc.; solves the problem of remote information transmission; solves the problem of audio and video transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A device for integrating and displaying surgical robot data provided by an embodiment of the present invention;

[0021] Figure 2 Another embodiment of the present invention provides a device for integrating and displaying surgical robot data;

[0022] Figure 3 A method for integrating and displaying surgical robot data provided by an embodiment of the present invention;

[0023] Figure 4 Another embodiment of the present invention provides a method for integrating and displaying surgical robot data. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described in more detail below in conjunction with the accompanying drawings. The present invention includes but is not limited to the following embodiments.

[0025] The endoscope is equipped with a video acquisition unit, which is used to capture the field of view that is difficult to see during surgery, such as the patient's abdominal cavity, the patient's uterus and other places where the field of view is obstructed. Endoscopes include gastroscopes, colonoscopes, cystoscopes, bronchoscopes, thoracoscopes, laparoscopes, laryngoscopes, nasopharyngeal mirrors, colposcopes, hysteroscopes, etc.

[0026] As attached Figure 1 As shown, a device for integrating and displaying surgical robot data of the present invention comprises: a data acquisition unit, a data processing unit, a display unit, an external storage unit, and a communication unit; wherein the data acquisition unit comprises a video acquisition unit and an information acquisition unit;

[0027] The data processing unit includes an FPGA and a DSP; the FPGA includes a first signal conversion unit, an internal storage unit, a video synthesis unit, and a backup unit, and the DSP includes a second signal conversion unit and an image conversion unit;

[0028] The first signal conversion unit is used to convert the data sent by the video acquisition unit and the data output by the FPGA received by the FPGA; the second signal conversion unit is used to convert the data sent by the information acquisition unit and the data output by the DSP received by the DSP; the image conversion unit is used to convert the data received by the DSP into image data; the video synthesis unit is used to synthesize the data received by the FPGA; the display unit is used to display the received data processed by the data processing unit, and the display unit includes a local display and / or a remote display;

[0029] The external storage unit is used to store data processed by the data processing unit, wherein the external storage unit and the DSP are connected via, for example, an IDE interface; the communication unit is used for data communication among the data acquisition unit, the data processing unit, the display unit, and the external storage unit, wherein the communication includes wireless communication and wired communication.

[0030] The video acquisition unit is installed on the endoscope at the surgical end, and is used to collect real-time video of the inside of the object being operated during surgery; the information acquisition unit is installed at the control end, and is used to collect real-time gesture instructions and / or typed instructions. Gesture instructions include instructions issued by gestures such as the motion state of the robotic arm and the position of the end of the robotic arm. Typed instructions include instructions issued by typing such as touch screen typing, keyboard typing, and button typing, and specifically include: information such as the type of surgery, type of instrument, and position of the end of the robotic arm. The position of the end of the robotic arm includes instructions such as the end depth and angle, which are convenient for direct display in the synthesized video. The displacement position angle of the robotic arm during surgery and the distance and angle required for displacement can be accurately determined without thinking, reducing the judgment error caused by only video display, and reducing the observation time of the controller watching the video.

[0031] Among them, the first signal conversion unit is used to convert the received video acquisition unit and the signal sent by the DSP into RGB signals; the second signal conversion unit is used to decode the signal received by the DSP and encode the signal sent by the DSP; the DSP processes the received instruction data into image data and sends it to the PFGA, and the internal storage unit realizes temporary storage of data and high-speed access; the video synthesis unit synthesizes the video data stored in the internal storage unit and the image data sent by the DSP; the first signal conversion unit converts the synthesized data into video data in RGB format; the backup unit backs up the video data, transmits the video data to the local display unit for display, and transmits the video data to the DSP; the DSP transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display. Among them, the local display displays the data synthesized by the video synthesis unit; the remote display displays the synthesized video sent by the receiving DSP; the external storage unit stores the synthesized video sent by the DSP.

[0032] The data collected by the video acquisition unit communicates with the FPGA, and the communication includes wired and wireless communication, such as communication through a wired HDMI interface, and the video data transmitted from the HDMI interface is converted into RGB signal data. The data collected by the information acquisition unit of the control end is transmitted to the processing end, and the processing end transmits the data to the data processing unit, and communicates with the DSP, and the communication includes wired and wireless communication, such as communication through a wired HDMI interface, and the DSP receives the command signal transmitted by the processing end. The DSP processes the command signal, for example, processes the command signal into image information, and sends it to the PFGA. The command signal includes gesture commands such as the operation status of the robot arm and the end posture of the robot arm, and the end posture of the robot arm includes commands such as the end depth and angle. The FPGA converts the signal of the data collected by the video acquisition unit, such as the HDMI interface, into an RGB signal through a signal conversion unit such as a DVI to RGB Decoder IP module. The RGB data is sent to a bus such as an AXI4 bus through a Video to AXI4 IP module. The VDMA (Video Direct Memory Access) IP module is used to realize the movement of a large amount of data between the internal storage unit and the bus-based video processing IP, and the high-speed access of the data on the bus and the data in the DDR memory is realized. The data processed by the data processing unit sent by the DSP, that is, the data processed into the information of the image, is stored in the internal storage unit through the GPMC bus IP, wherein the internal storage unit is used to temporarily store the operation data, and the internal storage unit is such as a DDR memory. The video synthesis unit synthesizes the video data stored in the internal storage unit and the image data sent by the DSP; the signal conversion unit such as the AXI4 to Video IP module converts the synthesized data into video data in RGB format; the backup unit backs up the video data. The FPGA transmits the synthesized video data to the local display unit for local display, and transmits the video data to the DSP; the DSP transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display, so as to facilitate the real-time viewing of remote medical staff, quickly obtain data information, and make judgments in time.

[0033] The surgical robot includes a surgical end, a processing end, and a control end. Figure 2As shown, another device for integrating and displaying surgical robot data of the present invention includes: a data acquisition unit, a data processing unit, a display unit, an external storage unit, a communication unit, and a voice output unit. The voice output unit is installed at the surgical end and the control end or at the surgical end and the remote control end, and communicates with the data processing unit to output voice; the information acquisition unit also includes an audio acquisition unit; the communication unit is used for data communication among the data acquisition unit, the data processing unit, the display unit, the external storage unit, and the voice output unit, and the communication includes wireless communication and wired communication. The data acquisition unit includes a video acquisition unit, an information acquisition unit, and an audio acquisition unit; the data processing unit includes a signal conversion unit, an internal storage unit, a video synthesis unit, and a backup unit; wherein the signal conversion unit is used to convert the data collected by the data acquisition unit and the data output by the data processing unit, and the video synthesis unit is used to synthesize the data collected by the data acquisition unit; the display unit is used to display the data processed by the data processing unit, including a local display and / or a remote display; the external storage unit is used to store the data processed by the data processing unit, wherein the external storage unit and the DSP are connected via, for example, an IDE interface; the communication unit is used for data communication among the data acquisition unit, the data processing unit, the display unit, the external storage unit, and the audio output unit, and the communication includes wireless communication and wired communication.

[0034] Among them, the video acquisition unit is installed on the endoscope at the surgical end, and is used to collect the video inside the object being operated during the operation; the information acquisition unit is installed at the control end, and is used to collect gesture instructions and / or typed instructions; the audio acquisition unit is used to collect audio data. Among them, gesture instructions include instructions issued by gestures such as the motion state of the robot arm and the position of the end of the robot arm, and typed instructions include instructions issued by typing such as touch screen typing, keyboard typing, and button typing, specifically including: information such as the type of surgery, type of instrument, and position of the end of the robot arm. The position of the end of the robot arm includes instructions such as the end depth and angle, which are convenient for direct display in the synthesized video. The displacement position angle of the robot arm during surgery and the distance and angle required to be displaced can be accurately determined without thinking, reducing the judgment error caused by only video display, and reducing the observation time of the controller watching the video.

[0035] Among them, the signal conversion unit is used to convert the signals collected by the received video acquisition unit and the information acquisition unit into RGB signals; the internal storage unit realizes temporary storage of data and high-speed access; the video synthesis unit includes synthesizing the data collected by the received video acquisition unit and the data collected by the information acquisition unit; the display unit displays the data synthesized by the video synthesis unit; the voice output unit outputs audio data, and the external storage unit stores the data synthesized by the video synthesis unit and the audio data.

[0036] The data processing unit includes FPGA and DSP. The data collected by the video acquisition unit communicates with the FPGA, and the communication includes wired and wireless communication, such as communicating through a wired HDMI interface, and converting the video data transmitted from the HDMI interface into RGB signal data.

[0037] The data collected by the information collection unit of the control end is transmitted to the processing end, and the processing end transmits the data to the data processing unit, and communicates with the DSP. The communication includes wired and wireless, such as communication through a wired HDMI interface, Ethernet or 5G network communication, and the DSP receives the command signal and voice signal transmitted by the processing end. The DSP processes the command signal, for example, processes the command signal into image information and sends it to the PFGA. At the same time, the DSP encodes and decodes the audio data, outputs the audio data through the voice output unit, and stores the audio data in the external storage unit. The command signal includes gesture command signals and typing command signals, wherein the command signal includes: gesture commands such as the operation status of the robot arm and the posture of the end of the robot arm, wherein the posture of the end of the robot arm includes commands such as the end depth and angle. The typing command includes commands issued by typing such as touch screen typing, keyboard typing, and button typing, specifically including: information such as the type of surgery, the type of instrument, and the position of the end of the robot arm, wherein the posture of the end of the robot arm includes commands such as the end depth and angle.

[0038] Among them, the FPGA converts the signal of the data collected by the video acquisition unit, such as the HDMI interface, into an RGB signal through a signal conversion unit, such as a DVI to RGB Decoder IP module. The RGB data is sent to a bus, such as an AXI4 bus, through a Video to AXI4 IP module. A large amount of data movement between the internal storage unit and the bus-based video processing IP is realized through a VDMA (Video Direct Memory Access) IP module, and high-speed access to data on the bus and data in the DDR memory is realized. The data processed by the data processing unit sent by the DSP, that is, the data processed into image information, is stored in the internal storage unit through the GPMC bus IP, wherein the internal storage unit is used to temporarily store calculation data, and the internal storage unit is such as a DDR memory. The video synthesis unit synthesizes the video data stored in the internal storage unit and the image data sent by the DSP; the signal conversion unit, such as the AXI4 to Video IP module, converts the synthesized data into video data in RGB format; the backup unit backs up the video data; the FPGA transmits the synthesized video data to the local display unit for local display, and transmits the video data to the DSP; the DSP transmits the synthesized video data and voice data to the external storage unit for storage, and encodes the synthesized video data and voice data, and transmits them to the remote display through the communication unit for remote display, so as to facilitate real-time viewing and real-time communication by remote medical staff, quickly obtain data information, make timely judgments, and provide timely guidance for surgery, thereby reducing the probability and frequency of medical accidents during surgery.

[0039] The surgical robot includes a surgical end, a processing end, a control end, and a remote control end. The local display is set at the control end, and the remote display is set at the remote control end, so as to facilitate the timely and simultaneous display of the synthesized data of the local display and the synthesized data of the remote display.

[0040] As attached Figure 3 As shown, a method for integrating and displaying surgical robot data of the present invention comprises the following steps:

[0041] S110: The data acquisition unit acquires data and communicates the acquired data with the data processing unit through the communication unit. The specific steps are: the video acquisition unit acquires video data and sends the acquired video data to the data processing unit through the communication unit; the information acquisition unit acquires gesture instructions and / or input instructions and sends the acquired data to the data processing unit through the communication unit.

[0042] S120: The data processing unit converts the data collected by the data collection unit and synthesizes and backs up the data. The specific steps are: the data processing unit receives the video data and data sent to the data processing unit through the communication unit, the signal conversion unit converts the video data and data into RGB signal data, and stores it in the internal storage unit, realizing a large amount of data movement between the internal storage unit and the bus-based video processing IP; extracting the data in the internal storage unit and synthesizing the video through the video synthesis unit, and backing up the synthesized video.

[0043] The data processing unit includes FPGA and DSP, and the specific steps include:

[0044] The video acquisition unit acquires video data and sends the acquired video data to the FPGA through the communication unit; the information acquisition unit acquires gesture instructions and / or typed instructions and sends the acquired instructions to the processing end, and the processing end transmits the data to the data processing unit and communicates with the DSP. The FPGA converts the signal of the video data received by the HDMI interface into an RGB signal through a signal conversion unit such as a DVI to RGB Decoder IP module; sends the RGB data to a bus such as an AXI4 bus through a Video to AXI4 IP module; stores the RGB data in an internal storage unit through a VDMA (Video Direct Memory Access) IP module; realizes the movement of a large amount of data between the storage unit in the system and the bus-based video processing IP; accesses the data on the bus such as the AXI4 bus and the DDR memory at a high speed. The DSP receives the instruction data, and processes the instruction data received by the DSP such as gesture instructions and / or typed instructions into image data, and sends it to the PFGA through a bus such as a GPMC bus.

[0045] Among them, gesture instructions include instructions issued through gestures, such as the movement state of the robot arm and the position of the end of the robot arm. Typed instructions include instructions issued through typing, such as touch screen typing, keyboard typing, and button typing, and specifically include: surgery type, instrument type, position of the end of the robot arm and other information. Among them, the position of the end of the robot arm includes instructions such as the end depth and angle, which are convenient for direct display in the synthesized video. The displacement position and angle of the robot arm during surgery and the distance and angle required to be displaced can be accurately determined without thinking, reducing the judgment error caused by video display only and reducing the observation time of the controller watching the video.

[0046] PFGA receives image data sent by DSP and stores it in an internal storage unit such as DDR memory; extracts video data and data stored in the internal storage unit through the VDMAIP module, and synthesizes new video data through the video synthesis unit; the signal conversion unit such as AXI4 to Video IP converts the synthesized video into video data in RGB format; the backup unit backs up the synthesized new video data, PFGA sends the synthesized video data to the local display unit for display, and sends the backed-up synthesized new video data to DSP.

[0047] S130: The data processing unit converts the synthesized data and communicates with the external storage unit and / or display unit through the communication unit. Specifically, the DSP receives the synthesized new video data sent by the backup unit, transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display.

[0048] As attached Figure 4 As shown, another method for integrating and displaying surgical robot data of the present invention comprises the following steps:

[0049] S210: The data acquisition unit acquires data, and communicates the acquired data with the data processing unit through the communication unit.

[0050] The specific steps are as follows: the video acquisition unit acquires video data and sends the acquired video data to the data processing unit through the communication unit; the information acquisition unit acquires gesture instructions and / or input instructions and sends the acquired data to the data processing unit through the communication unit, the information acquisition unit also includes an audio acquisition unit, the audio acquisition unit acquires the operation end and the control end or the operation end and the remote control end, the audio acquisition unit acquires the voice of the medical staff at the operation end and / or the voice of the medical staff at the control end, or the audio acquisition unit acquires the voice of the medical staff at the operation end and / or the voice of the medical staff at the remote control end, so as to facilitate mutual communication between medical staff, ensure real-time sharing of information, and reduce surgical failures. Among them, the voice output unit is installed at the operation end and the control end or at the operation end and the remote control end, communicates with the data processing unit, and is used to output voice; the information acquisition unit also includes an audio acquisition unit; the communication unit is used for data communication between the data acquisition unit, the data processing unit, the display unit, the external storage unit and the voice output unit, and the communication includes wireless communication and wired communication.

[0051] S220: The data processing unit converts the data collected by the data collection unit and synthesizes, backs up and encodes the data.

[0052] The specific steps are as follows: the data processing unit receives the video data, audio data and data sent to the data processing unit through the communication unit, the signal conversion unit converts the video data and data into RGB signal data, and stores them in the internal storage unit, realizing the movement of a large amount of data between the internal storage unit and the bus-based video processing IP; extracting the data in the internal storage unit and synthesizing the video through the video synthesis unit, backing up the synthesized video, and encoding the backed-up synthesized video and voice. The data processing unit includes FPGA and DSP, and the specific steps include:

[0053] The video acquisition unit acquires video data and sends the acquired video data to the FPGA through the communication unit; the information acquisition unit acquires gesture instructions and / or typed instructions and sends the acquired instructions to the processing end, and the processing end transmits the data to the data processing unit and communicates with the DSP. The FPGA converts the signal of the video data received by the HDMI interface into an RGB signal through a signal conversion unit such as a DVI to RGB Decoder IP module; sends the RGB data to a bus such as an AXI4 bus through a Video to AXI4 IP module; stores the RGB data in an internal storage unit through a VDMA (Video Direct Memory Access video register direct access and access) IP module; realizes a large amount of data movement between the storage unit in the system and the bus-based video processing IP; accesses the data on the bus such as the AXI4 bus and the DDR memory at a high speed. The DSP receives the instruction data, and processes the instruction data received by the DSP such as gesture instructions and / or typed instructions into image data, decodes the received audio data and sends it to an external storage unit for storage. The DSP sends the image data to the PFGA through a bus such as a GPMC bus.

[0054] Among them, gesture instructions include instructions issued through gestures, such as the movement state of the robot arm and the position of the end of the robot arm. Typed instructions include instructions issued through typing, such as touch screen typing, keyboard typing, and button typing, and specifically include: surgery type, instrument type, position of the end of the robot arm and other information. Among them, the position of the end of the robot arm includes instructions such as the end depth and angle, which are convenient for direct display in the synthesized video. The displacement position and angle of the robot arm during surgery and the distance and angle required to be displaced can be accurately determined without thinking, reducing the judgment error caused by video display only and reducing the observation time of the controller watching the video.

[0055] PFGA receives image data sent by DSP and stores it in an internal storage unit such as DDR memory; PFGA extracts video data and data stored in the internal storage unit through the VDMAIP module, and synthesizes new video data through the video synthesis unit; the signal conversion unit such as AXI4 to Video IP converts the synthesized video into video data in RGB format, and the backup unit backs up the synthesized new video data. PFGA sends the synthesized video data to the local display unit for display, and sends the backup synthesized new video data to DSP. DSP transmits the synthesized video data and voice data to the external storage unit for storage, encodes the synthesized video data and voice data, and transmits them to the remote display through the communication unit for remote display, so as to facilitate the remote medical staff to view and communicate in real time, quickly obtain data information, make judgments in time, and provide timely guidance for surgery, thereby reducing the probability and frequency of medical accidents during surgery.

[0056] The audio acquisition unit includes a first audio acquisition unit and a second audio acquisition unit. The first audio acquisition unit acquires audio data from the surgical end in real time, and the second audio acquisition unit acquires audio data from the control end in real time, and communicates the acquired data with the DSP through the communication unit; the DSP compresses the audio data acquired by the first audio acquisition unit and sends it to the surgical end through the communication unit, and the surgical end voice output unit outputs the audio. The DSP decompresses the audio data acquired by the second audio acquisition unit and sends it to the remote control end through the communication unit, and the remote control end voice output unit outputs the audio. The data processing unit communicates with the external storage unit, the display unit, and the voice output unit through the communication unit.

[0057] S230: The data processing unit converts the synthesized data and communicates with the external storage unit, the display unit and the voice output unit through the communication unit. Specifically including:

[0058] DSP combines audio data and synthesized video data, and sends them to an external storage unit, which stores data. PFGA sends the synthesized video to the display unit, which displays the synthesized video. PFGA sends the synthesized video data to the local display unit for display, and sends the backup synthesized new video data to DSP. DSP transmits the synthesized video data to the external storage unit for storage, encodes the synthesized video data and voice data, transmits them to the remote control end through the communication unit, displays them remotely on the remote display of the remote control end, and outputs them in the voice output unit of the remote control end. The voice acquisition unit collects the voice data of the remote control end, transmits them to the control end through the communication unit, and the DSP of the control end decodes them, stores the voice data, and outputs them in voice. It is convenient for remote medical staff to view and communicate in real time, quickly obtain data information, make judgments in time, and guide the operation in time, reducing the probability and frequency of medical accidents during surgery. Among them, the surgical robot includes a surgical end, a processing end, a control end, and a remote control end. The local display is, for example, arranged at the control terminal, and the remote display is, for example, arranged at the remote control terminal, so as to facilitate timely and simultaneous display of the synthesized data of the local display and the synthesized data of the remote display.

[0059] The present invention is not limited to the above-mentioned specific implementation modes. A person skilled in the art may adopt other various specific implementation modes to implement the present invention according to the embodiments and the contents of the drawings. Therefore, any design that adopts the design structure and concept of the present invention and makes some simple transformations or changes shall fall within the scope of protection of the present invention.

Claims

1. A device for integrating and displaying surgical robot data, characterized in that: include: Data acquisition unit, data processing unit, display unit, external storage unit, communication unit; The data acquisition unit includes a video acquisition unit and an information acquisition unit; The data processing unit includes FPGA and DSP; the FPGA includes a first signal conversion unit, an internal storage unit, a video synthesis unit, and a backup unit, and the DSP includes a second signal conversion unit and an image conversion unit; The first signal conversion unit is used to convert the data sent by the video acquisition unit and the data output by the FPGA received by the FPGA; the second signal conversion unit is used to convert the data sent by the information acquisition unit and the data output by the DSP received by the DSP; the image conversion unit is used to convert the data received by the DSP into image data; the video synthesis unit is used to synthesize the data received by the FPGA; the display unit is used to display the received data processed by the data processing unit, and the display unit includes a local display and / or a remote display; The external storage unit is used to store the data processed by the data processing unit; The communication unit is used for data communication among the data acquisition unit, the data processing unit, the display unit, and the external storage unit, and the communication includes wireless communication and wired communication; The data collected by the video acquisition unit communicates with the FPGA; the data collected by the information acquisition unit communicates with the DSP; The first signal conversion unit converts the signal of the data collected by the video acquisition unit into an RGB signal; the DSP processes the received data into image data and sends it to the PFGA; the image data sent by the DSP is stored in the internal storage unit; the video synthesis unit synthesizes the video data stored in the internal storage unit and the image data sent by the DSP; the first signal conversion unit converts the synthesized data into video data in RGB format; the backup unit backs up the video data, transmits the video data to the local display unit for display, and transmits the video data to the DSP; the DSP transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display.

2. The device for integrated display of surgical robot data according to claim 1, characterized in that: The video acquisition unit is installed on the endoscope at the surgical end and is used to acquire the video inside the operating object in real time; the information acquisition unit is installed at the control end and is used to acquire gesture instructions and / or typed instructions in real time.

3. The device for integrated display of surgical robot data according to claim 1, characterized in that: The first signal conversion unit is used to convert the received signal of the video acquisition unit into an RGB signal; the second signal conversion unit is used to decode the signal received by the DSP and encode the signal sent by the DSP; the internal storage unit realizes temporary storage of data and high-speed access; the video synthesis unit synthesizes the received data collected by the video acquisition unit and the data sent by the DSP; the local display displays the data synthesized by the video synthesis unit; The remote display displays the composite video sent by the DSP; the external storage unit stores the composite video sent by the DSP.

4. A device for integrating and displaying surgical robot data according to any one of claims 1 to 3, characterized in that: It also includes a voice output unit, which is installed on the surgical end and the control end or on the surgical end and the remote control end, communicates with the DSP, and is used to output voice; the information acquisition unit also includes an audio acquisition unit, and the audio acquisition unit includes a first audio acquisition unit and a second audio acquisition unit; the communication unit also includes data communication between the data acquisition unit, the data processing unit, the display unit, the external storage unit and the voice output unit, and the communication includes wireless communication and wired communication.

5. A method for integrating and displaying surgical robot data using the device according to any one of claims 1 to 4, characterized in that: include: The data acquisition unit acquires data and communicates the acquired data with the data processing unit through the communication unit; The data processing unit converts the data collected by the data collection unit and synthesizes and backs up the data; The data processing unit converts the synthesized data and communicates with the external storage unit and / or the display unit through the communication unit.

6. A method for integrating and displaying surgical robot data according to claim 5, characterized in that: The data acquisition unit acquires data, and communicates the acquired data with the data processing unit through the communication unit. The specific steps are as follows: The video acquisition unit acquires video data and sends the acquired video data to the FPGA through the communication unit; the information acquisition unit acquires gesture instructions and / or input instructions and sends the acquired data to the DSP through the communication unit.

7. The method for integrated display of surgical robot data according to claim 5, characterized in that: The data processing unit converts the data collected by the data collection unit and synthesizes and backs up the data. The specific steps are: The data processing unit receives the video data and data sent to the data processing unit through the communication unit, and the signal conversion unit converts the video data and data into RGB signal data and stores them in the internal storage unit, thereby realizing the movement of large amounts of data between the internal storage unit and the bus-based video processing IP; extracts the data in the internal storage unit and synthesizes the video through the video synthesis unit, and backs up the synthesized video.

8. The method for integrated display of surgical robot data according to claim 7, characterized in that: It also includes the FPGA converting the received video data signal into an RGB signal through the first signal conversion unit; sending the RGB data to the bus through the Video to AXI4IP module; and storing the RGB data in the internal storage unit through the VDMA (Video Direct Memory Access) IP module; Realize the movement of large amounts of data between the storage unit in the system and the bus-based video processing IP; the DSP receives the data collected by the information acquisition unit, processes the data received by the DSP into image data, and sends it to the PFGA through the GPMC bus; the PFGA receives the image data sent by the DSP and stores it in the internal storage unit; extracts the data stored in the internal storage unit through the VDMAIP module, and synthesizes the video data stored in the internal storage unit and the data into new video data through the video synthesis unit.

9. The method for integrated display of surgical robot data according to claim 7, characterized in that: It also includes the first signal conversion unit converting the synthesized video into video data in RGB format; the backup unit backs up the newly synthesized video data, sends it to the local display unit for display, and sends the synthesized video data to the DSP; the DSP transmits the synthesized video data to the external storage unit for storage, and transmits the synthesized video data to the remote display through the communication unit for remote display.

10. The method for integrating and displaying surgical robot data according to claim 5, characterized in that: The following steps are also included: The first audio acquisition unit acquires audio data from the surgical end in real time, and the second audio acquisition unit acquires audio data from the control end in real time, and communicates the acquired data with the DSP through the communication unit; The DSP compresses the audio data collected by the first audio collection unit and sends it to the surgery end through the communication unit, and the surgery end voice output unit outputs the audio; the DSP decompresses the audio data collected by the second audio collection unit and sends it to the remote control end through the communication unit, and the remote control end voice output unit outputs the audio; The data processing unit communicates with the external storage unit, the display unit and the voice output unit through the communication unit.

Images