A hypermedia recording method for intraoperative conversation

Abstract

The present invention relates to a hypermedia recording method for intraoperative conversations. The specific recording steps are: data acquisition → data processing → data network transmission → conversation exchange and recording. Data acquisition includes physical audio and video sources output by multiple medical imaging devices and audio acquisition devices, as well as preoperative examination and inspection information. Data processing involves flowing the collected data into independent stream pipes. The PC controls the reading of different video data streams based on identification information on each stream pipe, and can retrieve any audio and video signal at any time as the current image displayed by the PC. Data network transmission involves the PC compressing and encoding the original audio and video data frame stream into a network stream, which is then provided to a conversation terminal via the network for reading, thereby enabling mutual exchange of conversation content. The present invention can encode and transmit various surgical data in video form, and achieve multi-channel image switching and distribution through a single video signal, and can also store conversation content.

Description

Technical Field

[0001] The present invention relates to a recording method, in particular to a hypermedia recording method for intraoperative conversation. Background Art

[0002] In recent years, the digital transformation and upgrade of traditional hospital operating rooms has entered a fast-paced development, fueled by increasingly fierce competition. The breadth of informatization within the entire surgical department continues to expand, and new requirements for customized services tailored to specific medical users are constantly emerging. For example, the initial planning stages of many new digital operating rooms now include the requirement for intraoperative family conversations.

[0003] In the current common digital operating room solutions, most of them install cameras and microphones on the walls of the operating room, and then use encoding equipment to transmit local audio and video through the network to the remote family conversation room. The conversation room uses the same method to achieve full-duplex network communication.

[0004] Although this method can achieve contactless online conversations and improve efficiency, it still has the following shortcomings:

[0005] (1) The video source is a physical camera signal with a single source. More surgical-related information, such as preoperative examination and testing information, cannot be transmitted. If this is to be achieved, additional equipment must be added, which significantly increases the cost.

[0006] (2) The communication method between the two parties is limited to audio and video, and there are no more interactive methods. For medical staff, it is impossible to explain the detailed operation details to non-professional patient family members.

[0007] (3) The conversation process between the two parties cannot be fully recorded into a single file. Since the network coding is performed by independent devices, the various network streams cannot communicate with each other. Therefore, at present, there is either only one party's audio and video data file, or each is an independent recording file. When it is needed later, the content is incomplete and the coordination is poor, which loses its practical significance. Summary of the Invention

[0008] The purpose of the present invention is to provide a method for recording intraoperative conversation hypermedia that can not only encode and transmit various surgical data in video form, but also realize multi-channel screen switching and distribution through one video signal and store the conversation content.

[0009] To achieve the above-mentioned object, the technical solution of the present invention is: a method for recording intraoperative conversations through hypermedia, comprising a PC located in an operating room, a capture card connected to the PC, and a conversation terminal located in a remote conversation room, wherein the PC and the conversation terminal are connected via a network, the conversation terminal only captures one video signal source and one audio source, and the capture card has multiple capture channels, characterized in that the multiple capture channels of the capture card are respectively used to access multiple medical imaging devices and audio capture devices in the operating room.

[0010] The specific steps for recording the intraoperative conversation are:

[0011] Step 1: Data collection. The specific steps of data collection are: the physical audio and video sources output by multiple medical imaging devices and audio acquisition devices are sent to the PC through the acquisition card device, and the preoperative examination and inspection information is stored in the PC in the form of static images;

[0012] Step 2: Data processing. The specific steps of data processing are: performing audio and video separation logic processing on the multiple physical audio and video collected in step 1 in the PC, encapsulating and converting each audio and video signal and flowing it into an independent stream pipeline. Each stream pipeline receives the current real-time audio and video raw data frame stream. Each frame of data is identified by two key attributes: the corresponding memory address and size.

[0013] The static image in step 1 is converted into a format, and its memory format is kept consistent with the structure of the real-time video frame data in the stream pipeline. The PC controls the reading of different video data streams based on the identification information on each stream pipeline, and can retrieve any audio and video signal at any time as the current image displayed by the PC;

[0014] Step 3: Data network transmission. The specific steps of the data network transmission are as follows: the PC creates a video service engine based on the RTSP streaming protocol, the video service engine receives the current real-time output of the audio and video raw data frame stream in any one of the stream pipes in step 2, compresses and encodes it into a network stream, and provides it to the conversation terminal through the network for reading;

[0015] Step 4: Conversation and recording; The specific steps of conversation and recording are as follows: the conversation terminal in the remote conversation room receives the network stream output by the PC network, and at the same time, the locally collected audio and video data is processed in the data processing method of step 2 to form a data composite stream and transmit it to the PC end via the network. At this time, the PC and the conversation terminal each call the corresponding network stream decoding engine to separate the received network stream and restore the audio data and video data, and output them to the speakers and playback display of the PC in the operating room and the speakers and playback display of the conversation terminal respectively. Through information exchange, the doctor in the operating room and the family members in the remote conversation room can communicate with each other.

[0016] The PC end in the operating room contains two channels of audio data and video data, one local and one remote, and the PC end synthesizes the two channels of audio data and video data into one signal and stores it in the PC for subsequent review and call.

[0017] In the above technical solution, the specific steps for format conversion of the static image in step 2 are: the PC in the operating room accesses the third-party Web service interface to obtain preoperative examination and inspection information, and the preoperative examination and inspection information is presented on the PC in the form of a static text data list or image format, and the PC copies and stores it in a bmp bitmap format, and each bitmap is uniquely identified by its corresponding memory address and address length, so that the bitmap format maintains structural consistency with the real-time video frame data in the stream pipeline.

[0018] In the above technical solution, in the data processing of step 2, the PC creates a video screen annotation engine, creates a transparent overlay layer in the stream pipeline and places it above the real-time video screen frame, draws a pattern logo on the transparent overlay layer, and uses the video screen annotation engine to render the drawn pattern logo into each video frame in real time so that it is superimposed with the original video signal. Finally, the PC can call any video data signal with the drawn pattern logo as the current screen displayed by the PC.

[0019] In the above technical solution, the drawn pattern logo is a combination of one or several different logo patterns selected from the group consisting of a straight line logo, a free line segment logo, a circular logo and a triangular logo.

[0020] In the above technical solution, the real-time audio and video frames contained in the video service engine in step three are compressed into an audio and video composite RTSP stream through H264 and AAC encoding, and a stream pull address in URL format is provided for decoding and playback by the conversation terminal in the remote conversation room.

[0021] In the above technical solution, the PC end in step four synthesizes the two channels of audio data and video data into one signal and stores it in the PC in .mp4 video file format. At this time, the PC creates a multi-screen synthesis engine and combines the two channels of video data into a new video source signal after multi-screen combination, so that the audio and video data at both ends of the conversation scene are processed and converted into one video source and two audio sources. Then, a mixed video processing engine is created to mix the two audio sources, and the compression encoding module is called to realize the writing of audio and video data into a standard file and store it in the PC.

[0022] The present invention has the following positive effects: after adopting the intraoperative conversation hypermedia recording method of the present invention, since the multiple acquisition channels of the acquisition card device of the present invention are respectively used to access multiple medical imaging devices and audio acquisition devices in the operating room, the specific recording steps of the intraoperative conversation are: data acquisition → data processing → data network transmission → conversation exchange and recording, wherein the data acquisition includes the physical audio and video sources output by the multiple medical imaging devices and audio acquisition devices, as well as preoperative examination and inspection information; the data processing is to flow the collected data into independent stream pipes respectively; the PC realizes the reading control of different video data streams according to the identification information on each stream pipe, and can call any audio and video signal as the current picture displayed by the PC at any time; the data network transmission is that the PC compresses and encodes the original audio and video data frame stream into a network stream, and provides it to the conversation terminal for reading via the network; the PC and the conversation terminal realize mutual communication of the conversation content, and the PC end in the operating room contains two-way audio data and video data of the local and remote ends, and the PC end synthesizes the two-way audio data and video data into one signal and stores it in the PC for subsequent reference and call;

[0023] Therefore, the advantages of the present invention are:

[0024] (1) Expand the existing single video image of the imaging device in the intraoperative conversation scene into a richer hypermedia resource, which can fully communicate the surgical situation with the patient's family;

[0025] (2) During the conversation, users can add annotation information to the real-time image source and synchronize it to the video in real time and send it to the other party as an auxiliary means of explanation;

[0026] (3) The multi-channel video source input adopts a "director's console"-like operation. The terminal switches and controls different signal sources on demand, and only outputs one network video stream throughout the entire process. This not only ensures surgical privacy but also reduces the business complexity of the software, while also saving network bandwidth and improving flexibility.

[0027] (4) Through a unique audio and video processing framework, the encoding and decoding process can be processed and controlled more flexibly, thereby achieving full and persistent recording of the network audio and video data of both parties. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and given embodiments, but the present invention is not limited thereto.

[0030] like Figure 1 As shown, a method for recording intraoperative conversations through hypermedia includes a PC located in an operating room, a capture card connected to the PC, and a conversation terminal located in a remote conversation room. The PC and the conversation terminal are connected via a network. The conversation terminal only captures one video signal source and one audio source. The capture card has multiple capture channels, each of which is used to access multiple medical imaging devices and audio capture devices in the operating room.

[0031] The specific steps for recording the intraoperative conversation are:

[0032] Step 1: Data collection. The specific steps of data collection are: the physical audio and video sources output by multiple medical imaging devices and audio acquisition devices are sent to the PC through the acquisition card device, and the preoperative examination and inspection information is stored in the PC in the form of static images;

[0033] Step 2: Data processing. The specific steps of data processing are: performing audio and video separation logic processing on the multiple physical audio and video collected in step 1 in the PC, encapsulating and converting each audio and video signal and flowing it into an independent stream pipeline. Each stream pipeline receives the current real-time audio and video raw data frame stream. Each frame of data is identified by two key attributes: the corresponding memory address and size.

[0034] The static image in step 1 is converted into a format, and its memory format is kept consistent with the structure of the real-time video frame data in the stream pipeline. The PC controls the reading of different video data streams based on the identification information on each stream pipeline, and can retrieve any audio and video signal at any time as the current image displayed by the PC;

[0035] The specific steps of converting the format of the static image are as follows: the PC in the operating room accesses a third-party Web service interface to obtain preoperative examination and inspection information, and the preoperative examination and inspection information is presented on the PC in a static text data list or image format, and the PC copies and stores it in a bmp bitmap format, and each bitmap is uniquely identified by its corresponding memory address and address length, so that the bitmap format maintains a consistent structure with the real-time video frame data in the streaming pipeline;

[0036] The PC creates a video image annotation engine, creates a transparent overlay in the stream pipeline and places it above the real-time video image frame, draws a pattern identifier on the transparent overlay, and uses the video image annotation engine to render the drawn pattern identifier into each video frame in real time so that it is superimposed on the original video signal. Finally, the PC can call any video data signal with the drawn pattern identifier as the current image displayed by the PC, where the drawn pattern identifier is one or a combination of different identifier patterns such as a straight line identifier, a free line segment identifier, a circular identifier, and a triangular identifier;

[0037] Step 3: Data network transmission. The specific steps of the data network transmission are as follows: the PC creates a video service engine based on the RTSP streaming protocol. The video service engine receives the current real-time output of the audio and video raw data frame stream in any stream pipe in step 2, compresses and encodes it into a network stream, and provides it to the conversation terminal through the network for reading.

[0038] The real-time audio and video frames contained in the video service engine are compressed into audio and video composite RTSP streams through H264 and AAC encoding, and a stream pull address in URL format is provided for decoding and playback by the conversation terminal in the remote conversation room;

[0039] Step 4: Conversation and recording; The specific steps of conversation and recording are as follows: the conversation terminal in the remote conversation room receives the network stream output by the PC network, and at the same time, the locally collected audio and video data is processed in the data processing method of step 2 to form a data composite stream and transmit it to the PC end via the network. At this time, the PC and the conversation terminal each call the corresponding network stream decoding engine to separate the received network stream and restore the audio data and video data, and output them to the speakers and playback display of the PC in the operating room and the speakers and playback display of the conversation terminal respectively. Through information exchange, the doctor in the operating room and the family members in the remote conversation room can communicate with each other.

[0040] The PC in the operating room contains two channels of audio data and video data from the local and remote ends, and the PC combines the two channels of audio data and video data into one signal and stores it in the PC for subsequent review and call;

[0041] In the step 4, the PC side synthesizes the two channels of audio data and video data into one signal and stores it in the .mp4 video file format in the PC. At this time, the PC creates a multi-screen synthesis engine and combines the two channels of video data into a new video source signal after multi-screen combination. The audio and video data at both ends of the conversation scene are processed and converted into one video source and two audio sources. Then, a mixed video processing engine is created to mix the two audio sources, and the compression encoding module is called to realize the writing of audio and video data into a standard file and store it in the PC.

[0042] It can be seen that the specific recording steps of the intraoperative conversation of the present invention are: data acquisition → data processing → data network transmission → conversation exchange and recording, wherein data acquisition includes physical audio and video sources output by multiple medical imaging devices and audio acquisition devices, as well as preoperative examination and testing information, data processing is to flow the collected data into independent stream pipes respectively, and the PC realizes the reading control of different video data streams according to the identification information on each stream pipe, and can call any audio and video signal as the current picture displayed by the PC at any time, and data network transmission is created by the PC as a video service engine based on the RTSP streaming protocol, and the PC compresses and encodes the original audio and video data frame stream into a network stream, and provides it to the conversation terminal for reading through the network, so as to realize the mutual communication of the conversation content between the doctor in the operating room and the patient in the remote conversation room, and the PC end in the operating room contains two-way audio data and video data of the local and remote ends, and the PC end synthesizes the two-way audio data and video data into one signal and stores it in the PC for subsequent reference and calling.

[0043] Therefore, the advantages of the present invention are:

[0044] (1) Expand the existing single video image of the imaging device in the intraoperative conversation scene into a richer hypermedia resource, which can fully communicate the surgical situation with the patient's family;

[0045] (2) During the conversation, users can add annotation information to the real-time image source and synchronize it to the video in real time and send it to the other party as an auxiliary means of explanation;

[0046] (3) The multi-channel video source input adopts a "director's console"-like operation. The terminal switches and controls different signal sources on demand, and only outputs one network video stream throughout the entire process. This not only ensures surgical privacy but also reduces the business complexity of the software, while also saving network bandwidth and improving flexibility.

[0047] (4) Through a unique audio and video processing framework, the encoding and decoding process can be processed and controlled more flexibly, thereby achieving full and persistent recording of the network audio and video data of both parties.

[0048] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A method for recording intraoperative conversations through hypermedia, comprising a PC located in an operating room, a capture card connected to the PC, and a conversation terminal located in a remote conversation room. The PC and the conversation terminal are connected via a network. The conversation terminal only captures one video signal source and one audio source. The capture card has multiple capture channels. The method is characterized by: The multiple acquisition channels of the acquisition card device are used to access multiple medical imaging devices and audio acquisition devices in the operating room. The specific steps for recording the intraoperative conversation are: Step 1: Data collection. The specific steps of data collection are: the physical audio and video sources output by multiple medical imaging devices and audio acquisition devices are sent to the PC through the acquisition card device, and the preoperative examination and inspection information is stored in the PC in the form of static images; Step 2: Data processing. The specific steps of data processing are: performing audio and video separation logic processing on the multiple physical audio and video collected in step 1 in the PC, encapsulating and converting each audio and video signal and flowing it into an independent stream pipeline. Each stream pipeline receives the current real-time audio and video raw data frame stream. Each frame of data is identified by two key attributes: the corresponding memory address and size. The static image in step 1 is converted into a format, and its memory format is kept consistent with the structure of the real-time video frame data in the stream pipeline. The PC controls the reading of different video data streams based on the identification information on each stream pipeline, and can retrieve any audio and video signal as the current picture displayed by the PC at any time. The specific steps of converting the format of the static image in this step are as follows: the PC in the operating room accesses a third-party Web service interface to obtain preoperative examination and inspection information, and the preoperative examination and inspection information is presented on the PC in a static text data list or image format. The PC copies and stores the information in a bmp bitmap format, and each bitmap is uniquely identified by its corresponding memory address and address length, so that the bitmap format maintains a consistent structure with the real-time video frame data in the streaming pipeline; Step 3: Data network transmission. The specific steps of the data network transmission are as follows: the PC creates a video service engine based on the RTSP streaming protocol. The video service engine receives the current real-time output of the audio and video raw data frame stream in any stream pipe in step 2, compresses and encodes it into a network stream, and provides it to the conversation terminal through the network for reading. In this step, the real-time audio and video frames contained in the video service engine are compressed into an audio and video composite RTSP stream through H264 and AAC encoding, and a stream pull address in URL format is provided for decoding and playback by the conversation terminal in the remote conversation room; Step 4: Conversation and recording; The specific steps of conversation and recording are as follows: the conversation terminal in the remote conversation room receives the network stream output by the PC network, and at the same time, the locally collected audio and video data is processed in the data processing method of step 2 to form a data composite stream and transmit it to the PC end via the network. At this time, the PC and the conversation terminal each call the corresponding network stream decoding engine to separate the received network stream and restore the audio data and video data, and output them to the speakers and playback display of the PC in the operating room and the speakers and playback display of the conversation terminal respectively. Through information exchange, the doctor in the operating room and the family members in the remote conversation room can communicate with each other. The PC end in the operating room contains two channels of audio data and video data, one local and one remote, and the PC end synthesizes the two channels of audio data and video data into one signal and stores it in the PC for subsequent review and call.

2. The method for recording intraoperative conversations via hypermedia according to claim 1, wherein: In the data processing of step 2, the PC creates a video screen annotation engine, creates a transparent overlay in the stream pipeline and places it above the real-time video screen frame, draws a pattern logo on the transparent overlay, and uses the video screen annotation engine to render the drawn pattern logo into each video frame in real time so that it is superimposed with the original video signal. Finally, the PC can call any video data signal with the drawn pattern logo as the current screen displayed by the PC.

3. The method for recording intraoperative conversations via hypermedia according to claim 2, characterized in that: The drawn pattern logo is a combination of one or several different logo patterns selected from the group consisting of a straight line logo, a free line segment logo, a circular logo and a triangular logo.

4. The method for recording intraoperative conversations via hypermedia according to claim 1, wherein: In the step 4, the PC side synthesizes the two channels of audio data and video data into one signal and stores it in the .mp4 video file format in the PC. At this time, the PC creates a multi-screen synthesis engine and combines the two channels of video data into a new video source signal after multi-screen combination. The audio and video data at both ends of the conversation scene are processed and converted into one video source and two audio sources. Then, a mixed video processing engine is created to mix the two audio sources, and the compression encoding module is called to realize the writing of audio and video data into a standard file and store it in the PC.

Images