Visual intraoperative video interaction system

Abstract

The invention, which relates to the technical field of video interaction, discloses a visual intra-operative video interaction system comprising a video acquisition module, a data processing module, a network communication module, a far-end teaching guidance module, a data storage module, a user interface module and a back-end service module. The video acquisition module is used for acquiring focus image information of a patient in an operation through a camera and an endoscope and capturing an operation process video in real time; and the data processing module is used for receiving, processing and compressing the video signals and carrying out image analysis and data integration at the same time. Through real-time monitoring and remote guidance, a remote expert can perform video communication with an operating room through a mobile device, time and occasions are not limited, the remote expert can watch an on-going operation anytime and anywhere, if the remote expert wants to invite an opposite side to perform operation communication, the remote expert can directly call the opposite side, and a surgeon can obtain immediate support under complex conditions; therefore, the safety and the effect of the operation process are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of video interaction, and in particular to a visual intraoperative video interaction system. Background Art

[0002] With the improvement of network technology, video transmission technology and data processing capabilities, the medical industry has gradually introduced modern information technology, which has made visualization and remote communication of the surgical process possible. With the continuous advancement of medical technology, the types and complexity of surgeries are increasing, and surgeons are facing more challenges. During surgery, quick response and efficient decision-making are crucial to patient safety.

[0003] Currently, surgeons may face difficulties in making decisions during surgery. The traditional model often lacks real-time guidance and advice from external experts, which may affect the effectiveness of the surgery and patient safety. It is impossible to communicate effectively with remote experts during the operation and to solve problems immediately. This makes it difficult for doctors to find solutions quickly in difficult situations, which makes it easy for errors and complications to occur, affecting the safety and health of patients. Summary of the invention

[0004] The purpose of the present invention is to provide a visualized intraoperative video interaction system to solve the problem that surgeons may face decision-making difficulties as mentioned in the above background technology. The traditional model often lacks real-time guidance and suggestions from external experts, which may affect the surgical effect and patient safety. It is impossible to communicate effectively with remote experts during the operation and solve the problems that arise immediately, making it impossible for doctors to quickly find solutions in difficult situations, which makes it easy for errors and complications to affect the safety and health of patients.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solutions: a visual intraoperative video interaction system, comprising a video acquisition module, a data processing module, a network communication module, a remote teaching guidance module, a data storage module, a user interface module and a backend service module;

[0006] The video acquisition module is used to collect the image information of the patient's lesion during surgery through a camera and an endoscope, and capture the video of the surgical process in real time;

[0007] The data processing module is used to receive, process and compress video signals, and simultaneously perform image analysis and data integration, and the processed video is transmitted through the network;

[0008] The network communication module uses WebSocket and Netty to establish a communication platform for real-time chat and video calls based on the 5G network, using a point-to-point exclusive communication link and real-time video transmission;

[0009] The remote teaching and guidance module is used for doctors and experts to participate in the surgical process using mobile devices at any location, and supports users to initiate surgical communication invitation calls at any time;

[0010] The data storage module is used to automatically save the video call content as a video file, supports playback and reading, and uses local terminal storage while backing up and retaining data in the cloud;

[0011] The user interface module is used for doctors and remote users to operate and interact, and supports users to play back saved video files;

[0012] The backend service module uses SpringBoot and MyBatis-Plus to implement facility construction for data processing, storage and implementation of business logic.

[0013] Preferably, the data processing module includes a video encoding module and an image processing module;

[0014] The video encoding module uses H.264 / H.265 encoding algorithm to compress the video stream and decode the received video stream for display and analysis. Through advanced video encoding and network communication technology, it ensures efficient transmission and processing of surgical video streams, greatly improving the system's response speed and user experience;

[0015] The image processing module is used to process, analyze and optimize real-time video streams and static images to improve video quality so that doctors can observe the surgical process more clearly.

[0016] Preferably, the network communication module includes a network protocol, a network server, a network security module and a network monitoring module;

[0017] The network protocol supports WebRTC, RTSP / RTMP and HTTP / HTTPS. WebRTC supports the real-time communication protocol for point-to-point video calls, RTSP / RTMP is used for streaming media transmission, HTTP / HTTPS is used for data request and interaction, and uses a point-to-point exclusive communication link for real-time video transmission to avoid being affected by other communication links, thereby ensuring the stability and security of data transmission;

[0018] The network server is used to process and exchange signaling information, establish and maintain P2P connections, forward and process video streams, optimize network transmission paths, and reduce data delays during transmission to achieve instant response and communication;

[0019] The network security module is used to encrypt data transmission and perform access control to ensure that only authenticated users can access the system, protect the privacy of data in transmission, and prevent unauthorized access and data leakage;

[0020] The network monitoring module is used to display network status, traffic usage, latency and connection quality, and track network activities and user behavior through log records to promptly identify and resolve potential network problems.

[0021] Preferably, the remote teaching and guidance module includes a real-time remote guidance module, a remote teaching module, a case analysis module and a surgical communication module;

[0022] The real-time remote guidance module is used for remote users to communicate with the operating room via video through mobile devices, which is not limited to time and place, and can watch the ongoing surgery anytime and anywhere;

[0023] The remote teaching module is used to help novice doctors obtain technical guidance from experienced experts through remote teaching, and to enable relevant medical personnel to participate in discussions and guidance at any time through remote teaching and real-time communication, thereby promoting more efficient knowledge sharing and teamwork;

[0024] The case analysis module is used for multiple participants to share perspectives and conduct case discussions;

[0025] The surgical communication module is used for doctors and experts to communicate about the surgery through live video broadcast, and they can directly call each other.

[0026] Preferably, the data storage module includes a local storage module and a cloud storage;

[0027] The local storage module is used to store data with high-frequency access requirements and automatically save the video call content as a video file;

[0028] The cloud storage is used to store videos and large files in the cloud, providing a backup of the video files in the cloud while saving them on the terminal.

[0029] Preferably, the user interface module includes a user interaction module, an information display module, a user management module and an interactive tool module;

[0030] The user interaction module is used to provide users with an intuitive operation interface, supporting users to perform various operations, including logging in, video viewing, record viewing and operation control;

[0031] The information display module is used to present surgery-related data, video streams and important notifications in a visual form, thereby improving the convenience of user decision-making and ensuring that users can obtain system feedback in a timely manner during the operation;

[0032] The user management module is used to provide user registration, login, and permission management functions to ensure security and personalized experience;

[0033] The interactive tool module is used to provide real-time text chat and annotation and comment functions, allowing users to annotate and comment on the video stream to facilitate expert guidance and teaching.

[0034] Preferably, the backend service module includes an application server and a database;

[0035] The application server is built with SpringBoot and MyBatis-Plus application framework to process requests from the front end and other modules, implement business logic processing, make the system easy to expand and integrate other applications or services, and provide possibilities for future technology iteration and function expansion;

[0036] The database supports a relational database or a non-relational database to store user data information.

[0037] Preferably, the backend service module also includes a user authentication module and an API gateway;

[0038] The user authentication module adopts a dual Token strategy to implement interface authentication and user authentication;

[0039] The API gateway is used to provide an API interface for front-end applications and other services, and supports data interaction with different systems or modules.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. In the present invention, through real-time monitoring and remote guidance, remote experts can communicate with the operating room via video through mobile devices, without limitation on time and place, and can watch the ongoing surgery anytime and anywhere. If you want to invite the other party to communicate about the surgery, you can directly call the other party. Surgeons can get immediate support in complex situations, thereby enhancing the safety and effectiveness of the surgical process.

[0042] 2. The present invention supports 5G network communication. SpringBoot and MyBatis-Plus are used in the background to realize project facility construction, WebSocket and Netty are used to build a chat platform, and a point-to-point exclusive communication link is used. Real-time video transmission is smooth and will not be affected by other communication links, ensuring the stability and security of data transmission. The back-end service module based on SpringBoot makes it easy to expand and integrate other applications or services in the system, providing possibilities for future technology iterations and function expansion. A dual Token strategy is used to realize interface authentication and user authentication, thereby improving the security of information interaction.

[0043] 3. In the present invention, through the guidance of remote experts, novice doctors can quickly learn and master surgical skills, improve the efficiency of medical talent training, make up for the shortcomings of traditional training methods, and enable relevant medical personnel to participate in discussions and guidance at any time through remote teaching and real-time communication modules, thereby promoting more efficient knowledge sharing and teamwork.

[0044] 4. In the present invention, the content of video calls can be automatically saved as video files and support playback. While the video files are saved on the terminal, a backup is left in the cloud to ensure the safe storage and backup of the video content. At the same time, the data access method is optimized, making the management and use of medical records more flexible. Through advanced video encoding and network communication technologies, the efficient transmission and processing of surgical video streams are ensured, greatly improving the system's response speed and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The system block diagram of a visualized intraoperative video interaction system of the present invention.

[0046] In the figure: 1. Video acquisition module; 2. Data processing module; 21. Video encoding module; 22. Image processing module; 3. Network communication module; 4. Remote teaching guidance module; 41. Real-time remote guidance module; 42. Remote teaching module; 43. Case analysis module; 44. Surgical communication module; 5. Data storage module; 51. Local storage module; 52. Cloud storage module; 6. User interface module; 61. User interaction module; 62. Information display module; 63. User management module; 64. Interactive tool module; 7. Back-end service module. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Reference Figure 1 As shown: a visual intraoperative video interaction system, comprising a video acquisition module 1, a data processing module 2, a network communication module 3, a remote teaching and guidance module 4, a data storage module 5, a user interface module 6 and a back-end service module 7;

[0049] The video acquisition module 1 is used to collect the image information of the patient's lesion during surgery through a camera and an endoscope, capture the video of the surgical process in real time, transmit the surgical scene and the endoscope image in the operating room to the outside world, and teach remote viewers;

[0050] The data processing module 2 is used to receive, process and compress video signals, and perform image analysis and data integration. The processed video is transmitted through the network. Advanced video encoding and network communication technologies are used to ensure efficient transmission and processing of surgical video streams, greatly improving the system's response speed and user experience. Specifically, it includes a video encoding module 21 and an image processing module 22.

[0051] The video encoding module 21 uses H.264 / H.265 encoding algorithm to compress the video stream, thereby reducing bandwidth usage, and decodes the received video stream for display and analysis;

[0052] The image processing module 22 is used to process, analyze and optimize the real-time video stream and static images to improve the video quality so that the doctor can observe the surgical process more clearly;

[0053] The network communication module 3 uses WebSocket and Netty to establish a communication platform for real-time chat and video calls based on the 5G network, adopts a point-to-point exclusive communication link, and real-time video transmission to avoid being affected by other communication links, ensuring the stability and security of data transmission; specifically, it includes network protocols, network servers, network security modules, and network monitoring modules;

[0054] The network protocol supports WebRTC, RTSP / RTMP and HTTP / HTTPS. WebRTC supports the real-time communication protocol for point-to-point video calls, which can achieve low-latency video and audio stream transmission. RTSP / RTMP is a protocol for streaming media transmission, which is suitable for real-time transmission and control of video streams. HTTP / HTTPS is used for data request and interaction to ensure the standardization and security of data exchange.

[0055] The network server is used to process and exchange signaling information, establish and maintain P2P connections, forward and process video streams, optimize network transmission paths, and reduce data delays during transmission to achieve instant response and communication;

[0056] The network security module is used to encrypt data transmission and perform access control to ensure that only authenticated users can access the system, protect the privacy of data in transmission, and prevent unauthorized access and data leakage;

[0057] The network monitoring module is used to display network status, traffic usage, latency, and connection quality, and track network activities and user behavior through log records to promptly identify and resolve potential network problems;

[0058] The remote teaching and guidance module 4 is used for doctors and experts to participate in the operation process using mobile devices at any location, and supports users to initiate a surgical communication invitation call at any time. Through real-time monitoring and remote guidance, surgeons can get immediate support in complex situations, thereby enhancing the safety and effectiveness of the operation process; specifically, it includes a real-time remote guidance module 41, a remote teaching module 42, a case analysis module 43 and a surgical communication module 44;

[0059] The real-time remote guidance module 41 is used for remote users to communicate with the operating room via video through mobile devices, which is not limited to time and place, and can watch the ongoing surgery anytime and anywhere;

[0060] The remote teaching module 42 is used to help novice doctors obtain technical guidance from experienced experts through remote teaching, and to enable relevant medical personnel to participate in discussions and guidance at any time through remote teaching and real-time communication, thereby promoting more efficient knowledge sharing and teamwork;

[0061] The case analysis module 43 is used for multiple participants to share perspectives and conduct case discussions;

[0062] The surgical communication module 44 is used for doctors and experts to communicate about the surgery through live video broadcast, and they can directly call each other;

[0063] The data storage module 5 is used to automatically save the video call content as a video file, supports playback and reading, uses local terminal storage and performs data backup and retention in the cloud, ensures the safe storage and backup of the video content, and optimizes the data access method, making the management and use of medical records more flexible; specifically, it includes a local storage module 51 and a cloud storage module 52;

[0064] The local storage module 51 is used to store data with high-frequency access requirements and automatically save the video call content as a video file;

[0065] The cloud storage module 52 is used to store videos and large files in the cloud, providing video files that are saved on the terminal while having a backup in the cloud;

[0066] The user interface module 6 is used for the doctor and the remote user to operate and interact, and supports the user to play back the saved video files; specifically, it includes a user interaction module 61, an information display module 62, a user management module 63 and an interactive tool module 64;

[0067] The user interaction module 61 is used to provide an intuitive operation interface for users, supporting users to perform various operations, including logging in, video viewing, record viewing and operation control;

[0068] The information display module 62 is used to present surgery-related data, video streams and important notifications in a visual form, thereby improving the convenience of user decision-making and ensuring that the user can obtain system feedback in a timely manner during the operation;

[0069] The user management module 63 is used to provide user registration, login, and permission management functions to ensure security and personalized experience;

[0070] The interactive tool module 64 is used to provide real-time text chat and annotation and comment functions, allowing users to annotate and comment on the video stream to facilitate expert guidance and teaching;

[0071] Backend service module 7 uses SpringBoot and MyBatis-Plus to implement facility construction for data processing, storage and business logic implementation, making the system easy to expand and integrate with other applications or services, providing possibilities for future technology iteration and function expansion; specifically, it includes application server, database, user authentication module and API gateway;

[0072] The application server is built with SpringBoot and MyBatis-Plus application framework to process requests from the front-end and other modules and implement business logic processing; the database supports relational database or non-relational database to store user data information to ensure data persistence and retrieval efficiency; the user authentication module adopts a dual Token strategy to implement interface authentication and user authentication to protect user data security; the API gateway is used to provide API interfaces for front-end applications and other services, support data interaction with different systems or modules, and simplify the interaction between clients and back-end services.

[0073] The usage method and working principle of the present invention are as follows: the image and video information of the operation process are captured in real time through the camera and endoscope, the operation scene and endoscope image in the operating room are transmitted to the outside world, the remote viewers are taught, the collected video signals are processed and compressed, image analysis and data integration are realized, and efficient video transmission is ensured. Based on the 5G network, a real-time communication platform is established through WebSocket and Netty, point-to-point video transmission is supported, and a point-to-point exclusive communication link is adopted. The real-time video transmission is smooth and will not be affected by other communication links, ensuring the stability and security of data transmission. The present invention allows doctors and experts to participate in the operation through mobile devices and provides real-time remote guidance. Surgeons can get immediate support in complex situations, thereby enhancing the safety and effectiveness of the operation process, supporting case analysis and surgical communication, and enhancing Training effect, automatically save video content through local storage and cloud storage and support playback, video call content can be automatically saved as video files, support playback, video files are saved on the terminal and backed up in the cloud at the same time, ensuring the safe storage and backup of video content, and optimizing data access methods, making the management and use of medical records more flexible, ensuring data security and accessibility, and providing users with a friendly operation interface, supporting real-time operation, information display and user management, enhancing user experience, and adopting a dual Token strategy to implement interface authentication and user authentication to improve interaction security. Remote users can communicate with the operating room via video through mobile devices, not limited to time and place, and can watch the ongoing surgery anytime and anywhere. If you want to invite the other party to communicate about the surgery, you can call the other party directly without notifying the other party in advance, and the other party does not need to wait.

[0074] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual intraoperative video interaction system, characterized by: It comprises a video acquisition module (1), a data processing module (2), a network communication module (3), a remote teaching and guidance module (4), a data storage module (5), a user interface module (6) and a back-end service module (7); The video acquisition module (1) is used to collect intraoperative lesion image information of the patient through a camera and an endoscope, and to capture the video of the surgical process in real time; The data processing module (2) is used to receive, process and compress video signals, and simultaneously perform image analysis and data integration, and the processed video is transmitted via a network; The network communication module (3) uses WebSocket and Netty to establish a communication platform for real-time chat and video calls based on the 5G network, adopts a point-to-point exclusive communication link, and real-time video transmission; The remote teaching and guidance module (4) is used for doctors and experts to participate in the surgical process using mobile devices at any location, and supports users to initiate surgical communication invitation calls at any time; The data storage module (5) is used to automatically save the video call content as a video file, supports playback and reading, and uses local terminal storage while backing up and retaining data in the cloud; The user interface module (6) is used for doctors and remote users to operate and interact, and supports users to play back saved video files; The backend service module (7) uses SpringBoot and MyBatis-Plus to implement facility construction for data processing, storage and implementation of business logic.

2. A visual intraoperative video interaction system according to claim 1, characterized in that: The data processing module (2) comprises a video encoding module (21) and an image processing module (22); The video encoding module (21) uses H.264 / H.265 encoding algorithm to compress the video stream and decodes the received video stream for display and analysis; The image processing module (22) is used to process, analyze and optimize real-time video streams and static images.

3. The visual intraoperative video interaction system according to claim 1, characterized in that: The network communication module (3) comprises a network protocol, a network server, a network security module and a network monitoring module; The network protocol supports WebRTC, RTSP / RTMP and HTTP / HTTPS. WebRTC supports the real-time communication protocol for peer-to-peer video calls, RTSP / RTMP is used for streaming media transmission, and HTTP / HTTPS is used for data request and interaction. The network server is used to process and exchange signaling information, establish and maintain P2P connections, and forward and process video streams; The network security module is used to encrypt data transmission and perform access control to ensure that only authenticated users can access the system; The network monitoring module is used to display network status, traffic usage, latency and connection quality, and track network activities and user behavior through log records.

4. The visual intraoperative video interaction system according to claim 1, characterized in that: The remote teaching and guidance module (4) comprises a real-time remote guidance module (41), a remote teaching module (42), a case analysis module (43) and a surgical communication module (44); The real-time remote guidance module (41) is used for remote users to communicate with the operating room via video through a mobile device, which is not limited to time and place, and can watch the ongoing surgery anytime and anywhere; The remote teaching module (42) is used to help novice doctors obtain technical guidance from experienced experts through remote teaching; The case analysis module (43) is used for multiple participants to share perspectives and conduct case discussions; The surgical communication module (44) is used for doctors and experts to communicate about the surgery through live video broadcasting, and they can just call each other directly.

5. The visual intraoperative video interaction system according to claim 1, characterized in that: The data storage module (5) comprises a local storage module (51) and a cloud storage module (52); The local storage module (51) is used to store data with high-frequency access requirements and automatically save the video call content as a video file; The cloud storage module (52) is used to store videos and large files in the cloud, so that the video files can be saved on the terminal while being backed up in the cloud.

6. The visual intraoperative video interaction system according to claim 1, characterized in that: The user interface module (6) comprises a user interaction module (61), an information display module (62), a user management module (63) and an interaction tool module (64); The user interaction module (61) is used to provide an intuitive operation interface for users, supporting users to perform various operations, including logging in, video viewing, record viewing and operation control; The information display module (62) is used to present surgery-related data, video streams and important notifications in a visual form; The user management module (63) is used to provide user registration, login, and authority management functions to ensure security and personalized experience; The interactive tool module (64) is used to provide real-time text chat as well as annotation and comment functions.

7. The visual intraoperative video interaction system according to claim 1, characterized in that: The backend service module (7) includes an application server and a database; The application server is built using SpringBoot and MyBatis-Plus application frameworks to process requests from the front end and other modules and implement business logic processing; The database supports a relational database or a non-relational database to store user data information.

8. The visual intraoperative video interaction system according to claim 7, characterized in that: The backend service module (7) also includes a user authentication module and an API gateway; The user authentication module adopts a dual Token strategy to implement interface authentication and user authentication; The API gateway is used to provide an API interface for front-end applications and other services, and supports data interaction with different systems or modules.

Images