Operation video labeling system, operation video playing system and method, and storage medium

Abstract

The invention discloses an operation video labeling system, an operation video playing system and method, and a storage medium, and the system comprises at least one medical device which is used for providing a user interface of the medical device or a picture signal of an operation visual field; the lumped control equipment is used for providing a picture signal of a user interface of the lumped control equipment; the at least one camera device is used for providing a picture signal of an operation site; the processor is used for acquiring a picture signal from at least one of the at least one medical device, the lumped control device and the at least one camera device in an operation process and generating a video about the operation process; and the video processing module is also used for generating annotation information of the video based on the acquired setting information under the condition that any type of setting information of the operation site is acquired.

Description

Technical Field

[0001] The present application belongs to the field of medical equipment technology, and in particular, relates to a surgical video annotation system, a surgical video playback system and method, and a storage medium. Background Art

[0002] In order to meet the needs of surgical teaching, the surgical scene will be recorded during the operation. The recorded screen includes the panoramic screen of the operating room, the screen of the surgical field, the display interface of the monitor, the laparoscope screen, and so on. An operation usually lasts for several hours. The duration of the surgical video is relatively long, and the content of the screen in the surgical video changes relatively little during the entire duration of the surgical video. For example, during the entire duration of the surgical video, the laparoscope screen always shows the surgical operation in the abdominal cavity. Therefore, at this stage, if you need to review the surgical video at a certain moment in the surgical process, you need to repeatedly drag the progress bar of the video playback, and it will take multiple operations to locate the target playback position, which seriously affects the efficiency of reviewing the surgical video. Summary of the invention

[0003] To solve related technical problems, the embodiments of the present application provide a surgical video annotation system, a surgical video playback system and method, and a storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application embodiment provides a surgical video annotation system, including:

[0006] At least one medical device, used to provide a user interface or a picture signal of an operating field of the medical device;

[0007] A lumped control device, used for providing a picture signal of a user interface of the lumped control device;

[0008] At least one camera device, used to provide a picture signal of the surgical site;

[0009] A processor is used to acquire image signals from at least one of the at least one medical device, the centralized control device and the at least one camera device during the surgical process to generate a video about the surgical process; and is also used to generate annotation information for the video based on the acquired setting information when any type of setting information of the surgical site is acquired.

[0010] The present application also provides a surgical video playback system, including a display and a processor:

[0011] The processor is used to obtain a video of the surgical process and output it to the display, and obtain all or part of the category annotation information of the video, and when the video is played to a corresponding playback position, synchronously output the annotation information of the video associated with the corresponding playback position to the display; wherein the annotation information is generated based on setting information of the corresponding category of the surgical site;

[0012] The display is used to display the video and / or the annotation information of the video.

[0013] The present application also provides a surgical video annotation method, including:

[0014] During a surgical procedure, acquiring a picture signal from at least one of at least one medical device, a centralized control device, and at least one camera device to generate a video of the surgical procedure;

[0015] When any type of setting information of the surgical site is obtained, the annotation information of the video is generated based on the obtained setting information; wherein,

[0016] The marking information is generated based on setting information corresponding to the category of the surgical site.

[0017] The present application also provides a surgical video playback method, including:

[0018] Acquire the video of the surgical process and output it to the display;

[0019] Acquire all or part of the category labeling information of the video, and when the video is played to the corresponding playback position, synchronously output the labeling information of the video associated with the corresponding playback position to the display; wherein,

[0020] The marking information is generated based on setting information corresponding to the category of the surgical site.

[0021] An embodiment of the present application also provides a storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any of the above-mentioned surgical video annotation methods are implemented, or the steps of any of the above-mentioned surgical video playback methods are implemented.

[0022] In an embodiment of the present application, at least one medical device is used to provide a picture signal of the user interface or operation field of the medical device, the centralized control device is used to provide a picture signal of the user interface of the centralized control device, and at least one camera device is used to provide a picture signal of the operation site. The processor generates a video of the operation process based on at least one of the above picture signals, and, in the case of obtaining any type of setting information of the operation site, the processor generates the annotation information of the video based on the obtained setting information to complete the annotation of the operation video. Based on this, when playing the operation video, the processor outputs the video of the operation process to the display on the one hand, and obtains all or part of the category annotation information of the video on the other hand, and when the video is played to the corresponding playback position, the annotation information of the video associated with the corresponding playback position is synchronously output to the display. In this way, the display also displays the annotation information corresponding to the video while playing the video, which is convenient for the user to quickly locate the playback progress of the video based on the annotation information, thereby improving the efficiency of reviewing the operation video. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the structure of a surgical video annotation system according to an embodiment of the present application;

[0024] Figure 2 This is a flowchart of a surgical video annotation method implementation method according to an embodiment of the present application;

[0025] Figure 3 This is a schematic diagram of the structure of a surgical video playback system according to an embodiment of the present application;

[0026] Figure 4 This is an example diagram of the surgical video playback interface of the embodiment of the present application;

[0027] Figure 5 This is a flow chart of a surgical video playback method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Medical students need to undergo long-term training and teaching to be able to perform various operations proficiently. In daily training and teaching in hospitals, surgical demonstration is the most intuitive teaching method widely used. In order to meet the needs of surgical demonstration, the surgical scene will be recorded during the operation. The recorded screen includes the panoramic screen of the operating room, the surgical field screen, the display interface of the monitor, the laparoscope screen, and so on. An operation usually lasts for several hours. The duration of the surgical video is long, and the content of the screen in the surgical video changes relatively little during the entire duration of the surgical video. For example, during the entire duration of the surgical video, the laparoscope screen always shows the surgical operation in the abdominal cavity. Therefore, at this stage, if you need to review the surgical video at a certain moment in the surgical process, you need to repeatedly drag the progress bar of the video playback, and it will take multiple operations to locate the target playback position, which seriously affects the efficiency of reviewing the surgical video.

[0029] Based on this, the embodiment of the present application provides a surgical video annotation system, a surgical video playback system and method, and a storage medium, wherein at least one medical device is used to provide a screen signal of a user interface or an operation field of view of the medical device, a centralized control device is used to provide a screen signal of a user interface of the centralized control device, and at least one camera device is used to provide a screen signal of a surgical site, and the processor generates a video of the surgical process based on at least one of the above screen signals, and, in the case of obtaining any type of setting information of the surgical site, the processor generates the annotation information of the video based on the obtained setting information to complete the annotation of the surgical video. Based on this, when playing the surgical video, the processor outputs the video of the surgical process to the display on the one hand, and obtains all or part of the category annotation information of the video on the other hand, and when the video is played to the corresponding playback position, the annotation information of the video associated with the corresponding playback position is synchronously output to the display, so that the display also displays the annotation information corresponding to the video while playing the video, which is convenient for the user to quickly locate the playback progress of the video based on the annotation information, thereby improving the efficiency of reviewing the surgical video.

[0030] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0031] Figure 1 FIG. 4 shows a schematic diagram of the system architecture of a surgical video annotation system provided in an embodiment of the present application. Figure 1 As shown, the surgical video annotation system 1 includes:

[0032] At least one medical device 11, used to provide a picture signal of a user interface or an operation field of view of the medical device 11;

[0033] The lumped control device 12 is used to provide a picture signal of a user interface of the lumped control device 12;

[0034] At least one camera device 13 is used to provide a picture signal of the operation site.

[0035] Among them, all medical devices 11 and / or at least one camera device 13 deployed in the operating room can be communicated and connected through the centralized control device 12, and the centralized control device 12 can obtain the real-time operating status of these medical devices 11. Optionally, the centralized control device can centrally control some or all of the devices. The device form of the centralized control device 12 can be a fixed terminal, or it can also be a mobile terminal, or it can also be a combination of a fixed terminal and a mobile terminal. Optionally, the centralized control device 12 can be deployed in the same operating room with the medical devices 11. Medical equipment 11 includes but is not limited to operating tables, operating lights, pneumoperitoneum machines, energy platforms, ultrasonic knives, electric knives, monitors, laparoscopes, etc. In actual application, according to the communication mode, communication protocol or communication interface supported by the medical device 11, the centralized control device 12 can communicate with each medical device 11 through wired communication or wireless communication. Exemplarily, the centralized control device 12 can be connected to a local area network at the same time as all medical devices 11 in the operating room to achieve a safer and more reliable communication connection. Based on the communication connection between the centralized control device 12 and the medical device 11, the centralized control device 12 and the medical device 11 can transmit instructions or data to each other, including but not limited to the centralized control device 12 issuing instructions to the medical device 11, the medical device 11 returning a response to the centralized control device 12, or the medical device 11 reporting data to the centralized control device 12.

[0036] In an embodiment of the present application, both the medical device 11 and the centralized control device 12 serve as video signal sources for providing image signals during the surgical process. The medical device 11 may be a device equipped with a display screen, and the display screen displays a user interface of the medical device 11, which is used to display the operating status of the medical device 11, the data collection status, or provide an operation interface for operating the medical device 11. For example, the medical device 11 may be a monitor. Alternatively, the medical device 11 may be a device equipped with a camera, and the camera on the medical device 11 is used to capture the operating field of the medical device 11. For example, the medical device 11 may be a laparoscope, and the camera configured on the laparoscope can capture the operating field in the abdominal cavity. The centralized control device 12 provides a picture signal of its own user interface, which may be an operation interface for controlling each medical device 11.

[0037] The camera device 13 includes, but is not limited to, a camera device for photographing the surgical field area and a camera device for photographing the panoramic area of ​​the operating room. The camera device for photographing the surgical field area is usually deployed on the head or front of the body of the surgeon, and can clearly show the surgical field of the surgeon. The camera device for photographing the panoramic area of ​​the operating room is usually deployed at a specific location in the operating room, such as on the ceiling or on the wall, and usually uses a wide-angle lens, and the video image provided can show the panoramic view of the operating room.

[0038] In addition, the surgical video annotation system 1 also includes:

[0039] The processor 14 is used to obtain image signals from at least one of the at least one medical device, the centralized control device and the at least one camera device during the operation process to generate a video about the operation process; and is also used to generate annotation information for the video based on the acquired setting information when any type of setting information of the operation site is acquired.

[0040] Here, each of at least one medical device 11, a centralized control device 12, and at least one camera device 13 is used as a video signal source. In actual application, in order to meet the needs of surgical teaching, sometimes one of the video signal sources is selected to generate a surgical video, and sometimes multiple video signal sources are selected to be combined to generate a surgical video through a picture-in-picture split screen method, so that multiple picture signals can be displayed simultaneously on the same screen. In an embodiment of the present application, the processor 14 is used to record the video of one or more video signal sources to generate a surgical video. In the process of generating a surgical video, the video provided by the video signal source can be encoded and compressed, or it can be encoded and compressed without being compressed to directly generate a lossless surgical video.

[0041] In an embodiment of the present application, in addition to generating a surgical video, when any type of setting information of the surgical scene is obtained, the processor 14 is also used to generate video annotation information based on the obtained one or more types of setting information, and the generated annotation information is associated with the video through a timeline.

[0042] Here, the setting information is at least one of the following:

[0043] The person making the sound and / or the content of the sound at the surgical site;

[0044] Set alarm messages issued by medical equipment;

[0045] Setting operations triggered in setting devices in operating rooms;

[0046] Information that characterizes the progress of a surgery.

[0047] In the embodiment of the present application, the processor 14 adopts different marking methods for the surgical video according to different setting information. The following describes the categories of setting information listed above respectively:

[0048] For the case where the setting information is the person who speaks and / or the content of the speech at the operation site, correspondingly, in one embodiment, the system further includes:

[0049] An audio acquisition device is used to collect sound signals at the surgical site in real time during the surgical process;

[0050] Correspondingly, the processor is used to generate the first annotation information and / or the second annotation information of the video when the person who speaks and / or the content of the sound at the operation site are identified based on the sound signal.

[0051] Among them, the first annotation information represents the person who speaks; the second annotation information includes information determined based on the sound content; the first annotation information and / or the second annotation information are associated with the first playback position of the video; the first playback position represents the playback position corresponding to the sound time corresponding to the person who speaks and / or the sound content.

[0052] In actual application, the sound signals collected by the audio acquisition device during the operation include the voices of the medical staff in the operating room. Therefore, the identity of the person making the sound can be identified based on voiceprint recognition technology, or the content of the sound can be identified through speech recognition technology, thereby generating annotation information representing the person making the sound and / or the content of the sound, and the annotation information is associated with the corresponding playback position of the surgical video according to the time of the sound. For example, if the sound is made at 10:40 in the morning, then the corresponding annotation information is associated with the screen content at 10:40 in the morning in the surgical video.

[0053] Since the voices of medical staff can usually represent important nodes in the surgical process, for example, the surgeon will instruct the start of the operation, suture the incision and other important nodes by speaking at the surgical site, based on the above scheme, the important nodes in the surgical process can be marked in the surgical video based on the sound signals at the surgical site, which makes it convenient to quickly locate the important nodes in the surgical process for viewing during the teaching process, thereby improving the operational efficiency of video playback.

[0054] Correspondingly, in response to the alarm information issued by the set medical device, the processor generates third annotation information for the video when monitoring that the set medical device issues the alarm information.

[0055] Among them, the third annotation information represents the alarm information issued by the set medical device; the third annotation information is associated with the second playback position of the video; the second playback position represents the playback position corresponding to the issuance time of the alarm information.

[0056] In the surgical scenario, abnormal monitoring values ​​of the patient's physiological indicators and abnormal operating status of the medical equipment will trigger the alarm of the relevant medical equipment to indicate that the surgical process has reached a critical moment. For example, when the patient's blood pressure or blood oxygen index does not meet expectations during the operation, there may be surgical risks. Usually, at this time, medical staff will take corresponding emergency measures to prevent the situation from worsening. Therefore, this type of risk response is usually also a part that needs to be reviewed during the surgical teaching process. Therefore, here, based on the alarm information of the set medical equipment, the annotation information is generated, and the annotation information is associated with the corresponding playback position of the surgical video according to the time when the medical equipment sends the alarm information. It is also possible to realize the annotation of important nodes in the surgical process, which is convenient for quickly locating the important nodes in the surgical process for viewing during the teaching process.

[0057] Regarding the case where the setting information is a setting operation triggered by a setting device in an operating room, correspondingly, in one embodiment, the processor generates fourth annotation information for the video when detecting that the setting device in the operating room triggers the setting operation.

[0058] The fourth annotation information is used to describe the setting operation; the fourth annotation information is associated with a third playback position of the video; and the third playback position represents a playback position corresponding to a triggering time of the setting operation.

[0059] Here, the setting device of the operating room can be any electronic device in the operating room, and the triggered setting operation is usually an operation that can represent the node of the surgical process. In actual application, the setting operation of setting the electronic device includes but is not limited to:

[0060] 1. Operations related to the video recording function of the video signal source during the surgical process, such as starting recording, pausing recording, resuming recording, taking pictures, turning on the microphone, turning off the microphone, etc.

[0061] 2. Operations performed on related surgical equipment during the operation, such as turning on the laparoscope, turning on the operating light to ambient light, starting inflation of the pneumoperitoneum machine, stimulating the ultrasonic scalpel, taking laparoscopic photos, turning off the operating light, etc.

[0062] In other words, it can be understood that all the equipment used during the operation and all the operations that can be performed by the equipment can be used as annotation information to annotate the surgical video, and the annotation information can be associated with the corresponding playback position of the surgical video according to the time when the set operation is triggered. In this way, when performing subsequent teaching playback, the operation events of all relevant equipment during the operation can be used as indexes to achieve rapid positioning of the surgical video.

[0063] Correspondingly, with respect to the information characterizing the surgical progress, in one embodiment, the processor generates fifth annotation information of the video when a signal characterizing the surgical progress and / or changes in the operating room environment is detected.

[0064] Among them, the fifth annotation information is used to describe the corresponding surgical process and / or operating room environment changes; the fifth annotation information is associated with the fourth playback position of the video; the fourth playback position represents the playback position corresponding to the emission time of the signal.

[0065] Here, the information characterizing the surgical process includes, but is not limited to: preoperative preparation, surgery in progress, anesthesia, suturing, someone entering the operating room, someone leaving the operating room, equipment entering the operating room, equipment leaving the operating room, and so on.

[0066] In actual application, information representing the progress of the operation can be pushed by third-party software. For example, when someone enters or leaves the operating room, the sensor deployed on the operating room door can sense and push relevant information; or, information representing the progress of the operation can also be issued by medical staff through voice.

[0067] In an embodiment of the present application, the processor generates a surgical video based on one or more video signal sources, and after completing the annotation of the surgical video, when performing subsequent surgical teaching, the annotation information of the surgical video can be synchronously output based on the playback progress of the surgical video while playing the surgical video. On this basis, the surgical video can be quickly positioned using the annotation information as an index.

[0068] In one embodiment, the system further comprises:

[0069] A video storage module, used for storing the video;

[0070] An information storage module, used for the annotation information of the video;

[0071] A display, used to display the video and / or the annotation information of the video;

[0072] The processor is also used to obtain the video from the video storage module and output it to the display, and to obtain all or part of the category labeling information of the video from the information storage module, and when the video is played to the corresponding playback position, synchronously output the labeling information of the video to the display.

[0073] Here, when the surgical video is associated with annotation information corresponding to one or more types of setting information, the annotation information corresponding to one type of setting information can be output while the surgical video is being played, or the annotation information corresponding to multiple types of setting information can be output simultaneously. Alternatively, the annotation information corresponding to one or more types of setting information can be displayed separately, and the user can use the annotation information corresponding to one type of setting information as an index to search for the annotation information, and quickly locate the surgical video based on the annotation information found.

[0074] Based on this, in one embodiment, the processor is further configured to:

[0075] Outputting to the display all or part of the annotation information of the categories of the video stored in the information storage module; detecting the annotation information selected by the user in the output annotation information, and jumping the video to the playback position corresponding to the annotation information selected by the user; and / or,

[0076] The information storage module is searched based on the input query conditions, and the annotation information that meets the query conditions is output; and the video is jumped to the playback position corresponding to the output annotation information.

[0077] In actual application, users can view the annotation information of surgical videos according to the categories of set information. In addition, they can also query the annotation information by keywords. For example, they can enter the name of a doctor or the name of a medical device to search and obtain the voice content related to the doctor or the surgical node mark related to the medical device. Furthermore, when the user clicks on the corresponding annotation information, the surgical video can quickly locate the playback progress associated with the annotation information, so that the user can quickly view the associated surgical video.

[0078] In one embodiment, the processor is further configured to start the function of generating the annotation information in response to the first instruction, and to stop the function of generating the annotation information in response to the second instruction.

[0079] In an embodiment of the present application, the processor can simultaneously execute the operations of generating a surgical video and obtaining the annotation information corresponding to the surgical video, that is, while obtaining the picture signal from the video signal source, once the setting information of the surgical site is obtained, the corresponding annotation information is instantly generated based on the obtained setting information, and the generated annotation information is instantly associated with the surgical video on the timeline. In actual application, the above-mentioned function of real-time annotation of surgical videos can be turned on or off according to user needs. Specifically, the methods of turning on or off the real-time annotation function include but are not limited to: buttons, voice commands, gesture commands, third-party commands, such as issuing commands through mobile phone applications, etc.

[0080] Corresponding to the surgical video annotation system described in the above embodiment, the present application embodiment also provides a surgical video annotation method, which is applied to a processor in the surgical video annotation system. Figure 2 , methods include:

[0081] Step 201: During a surgical procedure, a picture signal is acquired from at least one of at least one medical device, a centralized control device, and at least one camera device to generate a video about the surgical procedure.

[0082] Step 202: when any type of setting information of the operation scene is obtained, the annotation information of the video is generated based on the obtained setting information.

[0083] The marking information is generated based on setting information of a corresponding category of the surgical site.

[0084] In one embodiment, the setting information is at least one of the following:

[0085] The person making the sound and / or the content of the sound at the surgical site;

[0086] Set alarm messages issued by medical equipment;

[0087] Setting operations triggered in setting devices in operating rooms;

[0088] Information that characterizes the progress of a surgery.

[0089] In one embodiment, generating the annotation information of the video based on the acquired setting information includes:

[0090] In the case where the person making the sound and / or the sound content at the operation scene are identified based on the sound signal at the operation scene, the first annotation information and / or the second annotation information of the video is generated; wherein,

[0091] The first annotation information represents the person who speaks; the second annotation information includes information determined based on the sound content; the first annotation information and / or the second annotation information are associated with the first playback position of the video; the first playback position represents the playback position corresponding to the sound time corresponding to the person who speaks and / or the sound content.

[0092] In one embodiment, generating the annotation information of the video based on the acquired setting information includes:

[0093] When the set medical device issues an alarm, third annotation information of the video is generated; wherein:

[0094] The third annotation information represents the alarm information issued by the set medical device; the third annotation information is associated with the second playback position of the video; the second playback position represents the playback position corresponding to the issuance time of the alarm information.

[0095] In one embodiment, generating the annotation information of the video based on the acquired setting information includes:

[0096] When it is detected that the setting device in the operating room triggers a setting operation, fourth annotation information of the video is generated; wherein,

[0097] The fourth annotation information is used to describe the setting operation; the fourth annotation information is associated with a third playback position of the video; and the third playback position represents a playback position corresponding to a triggering time of the setting operation.

[0098] In one embodiment, generating the annotation information of the video based on the acquired setting information includes:

[0099] When a signal representing a change in the surgical process and / or the operating room environment is detected, fifth annotation information of the video is generated; wherein,

[0100] The fifth annotation information is used to describe the corresponding surgical process and / or operating room environment changes; the fifth annotation information is associated with the fourth playback position of the video; the fourth playback position represents the playback position corresponding to the time when the signal is emitted.

[0101] The specific implementation of the surgical video annotation method in the embodiment of the present application can refer to the relevant description of the surgical video annotation system embodiment above, which will not be repeated here.

[0102] Figure 3 A schematic diagram of the system architecture of a surgical video playback system provided in an embodiment of the present application is shown. Figure 3 As shown, the surgical video playback system 3 includes: a display 31 and a processor 32.

[0103] The processor 32 is used to obtain the video of the surgical process and output it to the display, and obtain all or part of the category annotation information of the video, and when the video is played to the corresponding playback position, synchronously output the annotation information of the video associated with the corresponding playback position to the display; wherein the annotation information is generated based on the setting information of the corresponding category of the surgical site;

[0104] The display 31 is used to display the video and / or the annotation information of the video.

[0105] In the surgical video playback system 3 , one or more displays 31 may be deployed.

[0106] Here, the surgical video and / or the annotation information corresponding to the surgical video can be stored locally in the surgical video playback system or on a remote server. Therefore, the processor 32 obtains the video and all or part of the annotation information of the categories of the video from the server or the local storage of the surgical video playback system.

[0107] In the case where the surgical video and / or the annotation information corresponding to the surgical video can be stored locally in the surgical video playback system, the system further includes:

[0108] The video storage module is used to store the video;

[0109] The information storage module is used to store the annotation information of the video;

[0110] The processor block obtains the video from the video storage module, and obtains labeling information of all or part of the categories of the video from the information storage module.

[0111] In one embodiment, the setting information is at least one of the following:

[0112] The person making the sound and / or the content of the sound at the surgical site;

[0113] Set alarm messages issued by medical equipment;

[0114] Setting operations triggered in setting devices in operating rooms;

[0115] Information that characterizes the progress of the surgery.

[0116] In one embodiment, the processor is further configured to:

[0117] Outputting all or part of the annotation information of the video category to the display; detecting the annotation information selected by the user in the output annotation information, and jumping the video to the playback position corresponding to the annotation information selected by the user; and / or,

[0118] The search is performed based on the input query conditions, and the annotation information that meets the query conditions is output; and the video is jumped to the playback position corresponding to the output annotation information.

[0119] In one embodiment, the display also displays a first play progress bar of the video when playing the video.

[0120] Each annotation information of the video is associated with a first identifier at a corresponding playback position of the first playback progress bar, which is used to mark that the annotation information exists at the corresponding playback position.

[0121] Here, the annotation information corresponding to one or more types of setting information associated with the surgical video is displayed on the playback progress bar. When the user clicks on any of the annotation information, the surgical video will jump to the playback position associated with the annotation information for playback. In addition, in actual application, when the mouse hovers over the annotation information, the corresponding annotation information content will also be displayed, so that the user can easily know the corresponding surgical node through the annotation information.

[0122] In one embodiment, the display further displays at least two second play progress bars of the video when playing the video; wherein,

[0123] Each second playback progress bar is associated with a category of annotation information; each annotation information under each category is associated with a second identifier at a corresponding playback position of the corresponding second playback progress bar, which is used to mark the presence of annotation information at the corresponding playback position.

[0124] Reference Figure 4 An example of a player is given, in which 41 is a playback window for playing surgical videos, 42 to 45 are playback progress bars of the surgical videos. For example, playback progress bar A may correspond to the label information representing the identity of the person making the sound, playback progress bar B may represent the alarm information of the monitor, playback progress bar C may represent related operations of the laparoscope, such as turning on the laparoscope, taking pictures, etc., and playback progress bar D may represent changes in the operating room environment, such as someone entering the operating room, someone leaving the operating room, etc.

[0125] In this way, when playing the surgical video, the user can select the playback progress bar corresponding to a certain type of setting information.

[0126] In one embodiment, the identifiers corresponding to different types of annotation information are displayed in a differentiated manner according to a setting method; the setting method includes at least one of the following:

[0127] The color of the first logo;

[0128] The shape of the first logo;

[0129] The size of the first logo;

[0130] Transparency of the first logo;

[0131] The characters or images in the first logo.

[0132] In this embodiment, different categories of annotation information are distinguished and displayed through different identifiers. In this way, no matter whether multiple categories of annotation information are displayed through multiple playback progress bars or multiple categories of annotation information are displayed through one playback progress bar, different categories of annotation information can be distinguished and displayed very intuitively, which is convenient for users to quickly pay attention to the category of annotation information they are interested in, thereby speeding up the speed of video positioning.

[0133] Corresponding to the surgical video playback system described in the above embodiment, the present application embodiment also provides a surgical video playback method, which is applied to a processor in the surgical video playback system. Figure 5 , methods include:

[0134] Step 501: Acquire the video of the surgical process and output it to the display.

[0135] Step 502: Obtain all or part of the category labeling information of the video, and when the video is played to a corresponding playback position, synchronously output the labeling information of the video associated with the corresponding playback position to the display.

[0136] The marking information is generated based on setting information of a corresponding category of the surgical site.

[0137] In one embodiment, the setting information is at least one of the following:

[0138] The person making the sound and / or the content of the sound at the surgical site;

[0139] Set alarm messages issued by medical equipment;

[0140] Setting operations triggered in setting devices in operating rooms;

[0141] Information that characterizes the progress of a surgery.

[0142] In one embodiment, the method further comprises:

[0143] Outputting all or part of the annotation information of the video category to the display; detecting the annotation information selected by the user in the output annotation information, and jumping the video to the playback position corresponding to the annotation information selected by the user; and / or,

[0144] The search is performed based on the input query conditions, and the annotation information that meets the query conditions is output; and the video is jumped to the playback position corresponding to the output annotation information.

[0145] In one embodiment, the method further comprises:

[0146] When playing the video, a first play progress bar of the video is displayed on the play interface; wherein,

[0147] Each annotation information of the video is associated with a first identifier at a corresponding playback position of the first playback progress bar, which is used to mark that the annotation information exists at the corresponding playback position.

[0148] In one embodiment, the method further comprises:

[0149] When playing the video, at least two second play progress bars of the video are displayed on the play interface; wherein,

[0150] Each second playback progress bar is associated with a category of annotation information; each annotation information under each category is associated with a second identifier at a corresponding playback position of the corresponding second playback progress bar, which is used to mark the presence of annotation information at the corresponding playback position.

[0151] In one embodiment, the identifiers corresponding to different types of annotation information are displayed in a differentiated manner according to a setting method; the setting method includes at least one of the following:

[0152] The color of the first logo;

[0153] The shape of the first logo;

[0154] The size of the first logo;

[0155] Transparency of the first logo;

[0156] The characters or images in the first logo.

[0157] The specific implementation of the relevant steps of the surgical video playback system and the surgical video playback method in the embodiment of the present application can refer to the relevant description of the surgical video playback system embodiment above, and will not be repeated here.

[0158] The present application also provides a storage medium for storing various types of data to support Figure 1 or Figure 4 Operation of the processor in the embodiments. Examples of such data include: any computer program for operating on the processor.

[0159] In the above-mentioned embodiment of the present application, the processor may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The above-mentioned processor can implement or execute the disclosed methods, steps and logic block diagrams in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In conjunction with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, and the steps of the above-mentioned method are completed in conjunction with its hardware.

[0160] It can be understood that the storage medium of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0161] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0162] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0163] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A surgical video annotation system, characterized in that: include: At least one medical device, used to provide a user interface or a picture signal of an operating field of the medical device; A lumped control device, used for providing a picture signal of a user interface of the lumped control device; At least one camera device, used to provide a picture signal of the surgical site; A processor is used to acquire image signals from at least one of the at least one medical device, the centralized control device and the at least one camera device during the surgical process to generate a video about the surgical process; and is also used to generate annotation information for the video based on the acquired setting information when any type of setting information of the surgical site is acquired.

2. The system according to claim 1, characterized in that The setting information is at least one of the following: The person making the sound and / or the content of the sound at the surgical site; Set alarm messages issued by medical equipment; Setting operations triggered in setting devices in operating rooms; Information that characterizes the progress of a surgery.

3. The system according to claim 1 or 2, characterized in that: The system further comprises: An audio acquisition device is used to collect sound signals at the surgical site in real time during the surgical process; Correspondingly, the processor is used to generate the first annotation information and / or the second annotation information of the video when the person making the sound and / or the sound content at the operation site are identified based on the sound signal; wherein, The first annotation information represents the person who speaks; the second annotation information includes information determined based on the sound content; the first annotation information and / or the second annotation information are associated with the first playback position of the video; the first playback position represents the playback position corresponding to the sound time corresponding to the person who speaks and / or the sound content.

4. The system according to claim 1 or 2, characterized in that: The processor generates third annotation information of the video when detecting that the set medical device issues an alarm; wherein, The third annotation information represents the alarm information issued by the set medical device; the third annotation information is associated with the second playback position of the video; the second playback position represents the playback position corresponding to the issuance time of the alarm information.

5. The system according to claim 1 or 2, characterized in that: The processor generates fourth annotation information of the video when monitoring that the setting device in the operating room triggers the setting operation; wherein, The fourth annotation information is used to describe the setting operation; the fourth annotation information is associated with a third playback position of the video; and the third playback position represents a playback position corresponding to a triggering time of the setting operation.

6. The system according to claim 1 or 2, characterized in that: The processor generates fifth annotation information of the video when monitoring a signal representing a change in the surgical process and / or the operating room environment; wherein, The fifth annotation information is used to describe the corresponding surgical process and / or operating room environment changes; the fifth annotation information is associated with the fourth playback position of the video; the fourth playback position represents the playback position corresponding to the time when the signal is emitted.

7. The system according to claim 1, characterized in that The system further comprises: A video storage module, used for storing the video; An information storage module, used for the annotation information of the video; A display, used to display the video and / or the annotation information of the video; The processor is also used to obtain the video from the video storage module and output it to the display, and to obtain all or part of the category labeling information of the video from the information storage module, and when the video is played to the corresponding playback position, synchronously output the labeling information of the video to the display.

8. The system according to claim 7, characterized in that The processor is further configured to: Outputting all or part of the annotation information of the categories of the video stored in the information storage module to the display; detecting the annotation information selected by the user in the output annotation information, and jumping the video to the playback position corresponding to the annotation information selected by the user; and / or, The information storage module is searched based on the input query conditions, and the annotation information that meets the query conditions is output; and the video is jumped to the playback position corresponding to the output annotation information.

9. The system according to claim 1, characterized in that The processor is further configured to start the function of generating the annotation information in response to the first instruction, and to stop the function of generating the annotation information in response to the second instruction.

10. A surgical video playback system, characterized in that: Including display and processor: The processor is used to obtain a video of the surgical process and output it to the display, and obtain all or part of the category annotation information of the video, and when the video is played to a corresponding playback position, synchronously output the annotation information of the video associated with the corresponding playback position to the display; wherein the annotation information is generated based on setting information of the corresponding category of the surgical site; The display is used to display the video and / or the annotation information of the video.

11. The system according to claim 10, characterized in that The processor obtains the video and labeling information of all or part of the categories of the video from the server.

12. The system according to claim 10, characterized in that The system further comprises: The video storage module is used to store the video; The information storage module is used to store the annotation information of the video; The processor block obtains the video from the video storage module, and obtains labeling information of all or part of the categories of the video from the information storage module.

13. The system according to claim 11, characterized in that The setting information is at least one of the following: The person making the sound and / or the content of the sound at the surgical site; Set alarm messages issued by medical equipment; Setting operations triggered in setting devices in operating rooms; Information that characterizes the progress of a surgery.

14. The system according to claim 12 or 13, characterized in that The processor is further configured to: Outputting all or part of the annotation information of the category of the video to the display; detecting the annotation information selected by the user in the output annotation information, and jumping the video to the playback position corresponding to the annotation information selected by the user; and / or, The search is performed based on the input query conditions, and the annotation information that meets the query conditions is output; and the video is jumped to the playback position corresponding to the output annotation information.

15. The system according to claim 10, characterized in that The display also displays a first playback progress bar of the video when playing the video; wherein, Each annotation information of the video is associated with a first identifier at a corresponding playback position of the first playback progress bar, which is used to mark that the annotation information exists at the corresponding playback position.

16. The system according to claim 10, characterized in that The display also displays at least two second play progress bars of the video when playing the video; wherein, Each second playback progress bar is associated with a category of annotation information; each annotation information under each category is associated with a second identifier at a corresponding playback position of the corresponding second playback progress bar, which is used to mark the presence of annotation information at the corresponding playback position.

17. The system according to claim 15 or 16, characterized in that The identifiers corresponding to different types of annotation information are displayed in a differentiated manner according to a setting method; the setting method includes at least one of the following: The color of the first logo; The shape of the first logo; The size of the first logo; Transparency of the first logo; The characters or images in the first logo.

18. A surgical video annotation method, characterized in that: include: During a surgical procedure, acquiring a picture signal from at least one of at least one medical device, a centralized control device, and at least one camera device to generate a video of the surgical procedure; When any type of setting information of the surgical site is obtained, the annotation information of the video is generated based on the obtained setting information; wherein, The marking information is generated based on setting information corresponding to the category of the surgical site.

19. The method according to claim 18, characterized in that The setting information is at least one of the following: The person making the sound and / or the content of the sound at the surgical site; Set alarm messages issued by medical equipment; Setting operations triggered in setting devices in operating rooms; Information that characterizes the progress of a surgery.

20. The method according to claim 19, characterized in that The generating the annotation information of the video based on the acquired setting information includes: In the case where the person making the sound and / or the sound content at the operation scene are identified based on the sound signal at the operation scene, the first annotation information and / or the second annotation information of the video is generated; wherein, The first annotation information represents the person who speaks; the second annotation information includes information determined based on the sound content; the first annotation information and / or the second annotation information are associated with the first playback position of the video; the first playback position represents the playback position corresponding to the sound time corresponding to the person who speaks and / or the sound content.

21. The method according to claim 19, characterized in that The generating the annotation information of the video based on the acquired setting information includes: When the set medical device issues an alarm, third annotation information of the video is generated; wherein: The third annotation information represents the alarm information issued by the set medical device; the third annotation information is associated with the second playback position of the video; the second playback position represents the playback position corresponding to the issuance time of the alarm information.

22. The method according to claim 19, characterized in that The generating the annotation information of the video based on the acquired setting information includes: When it is detected that the setting device in the operating room triggers a setting operation, fourth annotation information of the video is generated; wherein, The fourth annotation information is used to describe the setting operation; the fourth annotation information is associated with a third playback position of the video; and the third playback position represents a playback position corresponding to a triggering time of the setting operation.

23. The method according to claim 19, characterized in that The generating the annotation information of the video based on the acquired setting information includes: When a signal representing a change in the surgical process and / or the operating room environment is detected, fifth annotation information of the video is generated; wherein, The fifth annotation information is used to describe the corresponding surgical process and / or operating room environment changes; the fifth annotation information is associated with the fourth playback position of the video; the fourth playback position represents the playback position corresponding to the time when the signal is emitted.

24. A surgical video playback method, characterized in that: include: Acquire the video of the surgical process and output it to the display; Acquire all or part of the category labeling information of the video, and when the video is played to the corresponding playback position, synchronously output the labeling information of the video associated with the corresponding playback position to the display; wherein, The marking information is generated based on setting information corresponding to the category of the surgical site.

25. The method according to claim 24, characterized in that The setting information is at least one of the following: The person making the sound and / or the content of the sound at the surgical site; Set alarm messages issued by medical equipment; Setting operations triggered in setting devices in operating rooms; Information that characterizes the progress of a surgery.

26. The method according to claim 24 or 25, characterized in that The method further comprises: Outputting all or part of the annotation information of the video category to the display; detecting the annotation information selected by the user in the output annotation information, and jumping the video to the playback position corresponding to the annotation information selected by the user; and / or, The search is performed based on the input query conditions, and the annotation information that meets the query conditions is output; and the video is jumped to the playback position corresponding to the output annotation information.

27. The method according to claim 24, characterized in that The method further comprises: When playing the video, a first play progress bar of the video is displayed on the play interface; wherein, Each annotation information of the video is associated with a first identifier at a corresponding playback position of the first playback progress bar, which is used to mark that the annotation information exists at the corresponding playback position.

28. The method according to claim 24, characterized in that The method further comprises: When playing the video, at least two second play progress bars of the video are displayed on the play interface; wherein, Each second playback progress bar is associated with a category of annotation information; each annotation information under each category is associated with a second identifier at a corresponding playback position of the corresponding second playback progress bar, which is used to mark the presence of annotation information at the corresponding playback position.

29. The method according to claim 27 or 28, characterized in that The identifiers corresponding to different types of annotation information are displayed in a differentiated manner according to a setting method; the setting method includes at least one of the following: The color of the first logo; The shape of the first logo; The size of the first logo; Transparency of the first logo; The characters or images in the first logo.

30. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method described in any one of claims 18 to 23, or implements the steps of the method described in any one of claims 27 to 28.

Images