Postoperative report output method and system for extracorporeal circulation doctor split-screen operation

Abstract

The invention provides a postoperative report output method and system for an extracorporeal circulation doctor split-screen operation. The method comprises the steps that the operation stage of a main operation is recognized based on an operation image; acquiring first interactive behavior data between an extracorporeal circulation doctor and the split-screen equipment, second interactive behavior data between the extracorporeal circulation doctor and a surgeon, and physiological monitoring data of a patient; inputting the operation image, the operation stage, the first interaction behavior data, the second interaction behavior data and the physiological monitoring data into a trained interference detection large model, so that the interference detection large model maps the first interaction behavior data to virtual standing posture data of an extracorporeal circulation doctor based on a mapping relation, when it is predicted that the surgeon is in an operation concentration state based on the virtual standing posture data, the operation image, the operation stage and the physiological monitoring data, whether the extracorporeal circulation doctor interferes with the surgeon or not is detected based on the second interaction behavior data; and outputting a postoperative report, wherein the postoperative report comprises an interference detection result output by the interference detection large model.

Description

Technical Field

[0001] This application relates to the technical field of medical devices, and more particularly, to a method and system for outputting postoperative reports for off-pump cardiologist split-screen surgery. Background Art

[0002] Cardiopulmonary bypass is a life support technology that uses artificial devices to drain venous blood returning to the heart outside the body, perform gas exchange artificially, adjust the temperature and filter it, and then return it to the body's arterial system. During cardiopulmonary bypass, since artificial devices replace the body's functions, it is also called cardiopulmonary bypass, and the cardiopulmonary bypass machine is also called an artificial heart-lung machine. For the main surgery that requires cardiopulmonary bypass surgery assistance, after the operation, the surgeon and / or the cardiologist often need to analyze the video manually to conduct a review to improve the surgical level. However, such a review method takes a long time for doctors, and the review efficiency and effect are not good. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a method and system for outputting postoperative reports for off-pump cardiologist split-screen surgery, so as to improve the postoperative review efficiency and review effect.

[0004] The embodiments of this application provide a method for outputting postoperative reports for off-pump cardiologist split-screen surgery. The off-pump cardiologist split-screen surgery is used to assist the main surgery; each surgical stage of the main surgery is monitored through a split-screen device for real-time surgical monitoring by the cardiologist and a main control device; the method includes: Identifying the surgical stage of the main surgery based on the surgical images collected by the main control device; Obtaining the first interaction behavior data between the cardiologist and the split-screen device, the second interaction behavior data between the cardiologist and the surgeon, and the physiological monitoring data of the patient; Inputting the surgical images, the surgical stage, the first interaction behavior data, the second interaction behavior data, and the physiological monitoring data into a trained interference detection large model, so that the interference detection large model maps the first interaction behavior data to the virtual standing posture data of the cardiologist based on a preset mapping relationship, and when predicting that the surgeon is in a focused surgical state based on the virtual standing posture data, the surgical images, the surgical stage, and the physiological monitoring data, detecting whether the cardiologist interferes with the surgeon based on the second interaction behavior data; the training data of the interference detection large model includes historical surgical videos with surgical stage labels and surgical state labels, historical physiological monitoring data, and historical standing posture data of the cardiologist; Outputting a postoperative report, where the postoperative report includes the interference detection result output by the interference detection large model.

[0005] By using a large model to predict whether the surgeon is in a surgical concentration state based on surgical images, the first interaction behavior data of the extracorporeal circulation doctor and the split-screen device, the physiological monitoring data of the patient, and the pre-established mapping relationship, and detecting whether the extracorporeal circulation doctor interferes with the surgeon when the surgeon is concentrated, and then generating a postoperative report. In this way, it is beneficial for the surgeon and the extracorporeal circulation doctor to use the postoperative report for postoperative summary and review, thereby improving the proficiency in using the split-screen device and the main control device in subsequent surgeries. Brief Description of the Drawings

[0006] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0007] Figure 1 It is a schematic flowchart of a method for outputting a postoperative report of an extracorporeal circulation doctor's split-screen surgery provided by an embodiment of the present application; Figure 2 It is a hardware structure diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0008] The following will describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application.

[0009] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0010] Before introducing this solution, the application scenario of this solution will be introduced first. Cardiopulmonary bypass surgery is used to assist the main surgery to be carried out together, rather than being carried out alone. Its core function is to temporarily replace the functions of the heart and lungs and provide operating conditions for the main surgery that requires cardiac arrest or cardiopulmonary function suspension. The main surgery includes, but is not limited to, cardiac surgery, lung transplantation surgery, great vessel surgery, and complex tumor resection surgery, etc. Among them, cardiopulmonary bypass surgery includes multiple circulation stages, such as, but not limited to, aortic occlusion, perfusion, rewarming, and root exhaust, etc. And the main surgery also includes multiple surgical stages. The assistance of cardiopulmonary bypass surgery to the main surgery is reflected in that one circulation stage of cardiopulmonary bypass surgery corresponds to one or more surgical stages of the main surgery, or multiple circulation stages correspond to one surgical stage of the main surgery. When the main surgery enters the next surgical stage, the cardiopulmonary bypass surgery should promptly enter the next circulation stage to assist the main surgery.

[0011] In this application, for the main surgery that needs to be assisted by cardiopulmonary bypass surgery, a split-screen device for real-time surgical monitoring by cardiopulmonary bypass doctors (hereinafter referred to as the split-screen device) and a main control device can be used. Among them, the split-screen device and the main control device can be respectively set in different surgical areas. For example, the split-screen device and the main control device can be set in different surgical areas in the same operating room, or can be set in different operating rooms. The split-screen device and the main control device are used to monitor multiple surgical stages of the main surgery. Specifically, the main control device can collect real-time pictures of the main surgery, and cardiopulmonary bypass doctors can timely master the surgical progress of the main surgery through the split-screen device without frequent communication with the surgeon. The split-screen device and the main control device can be communicatively connected by wireless communication or by wired connection. The following introduces these two devices with the wireless communication method between the split-screen device and the main control device.

[0012] The split-screen device for real-time surgical monitoring by cardiopulmonary bypass doctors specifically includes: A wireless transmission module, including a wireless receiver and a wireless transmitter. Its specific function is to receive information sent by wireless devices or transmit the current information to devices that can receive wireless signals.

[0013] A loudspeaker, whose specific function is to emit a warning sound to remind the operator. The operator includes cardiopulmonary bypass doctors and / or surgeons.

[0014] A display screen, including a display interface and multiple buttons. Its specific function is to display the current picture to the operator in a live broadcast manner or provide multiple buttons for the operator to touch and select.

[0015] The multiple buttons on the display screen include function and parameter settings, Live, Focus +, Focus -, Magnification +, Magnification -, Auto - calibration, Auto - focus, Video Recording (On / Off), Microphone (On / Off), Field of View Center, and Laser. In addition, it may also include Clear All Drawings, Undo Drawing, and Drawing buttons.

[0016] The function and parameter settings are used to set the screen parameters and functions of the display screen. The Live function allows the operator to enter or exit the live broadcast of the current display interface. The Focus +, Focus -, Magnification +, and Magnification - buttons are used for the operator to manually adjust the camera focal length of the main control device to adjust the clarity of the display interface. The Auto - focus function is a button for the main control device to automatically adjust the camera focal length to adjust the clarity of the display area. The Clear All Drawings function is used to clear the traces of the user's drawing on the display screen. The Undo Drawing function is used to undo the traces of the current user's drawing on the display screen. The Drawing function is a button for the user to draw on the display screen. The Laser function is a button to turn on the laser emitter of the main control device. The Microphone (On / Off) function is a button for the user to turn on / off the microphone of the current display device.

[0017] The main control device can obtain the images captured by the camera, connect to the wireless receiver of the split - screen device through the wireless projection module, and project the captured images onto the display screen of the split - screen device; the operator selects the buttons on the display screen of the split - screen device by touching the screen, the split - screen device obtains the selection, sends the selected content to the main control device through the wireless transmitter, and the built - in program of the main control device automatically completes the corresponding operations.

[0018] In addition to the split - screen device and the main control device for real - time surgical monitoring by extracorporeal circulation doctors, a server can also be set up to communicate with the split - screen device and the main control device for real - time surgical monitoring by extracorporeal circulation doctors. The postoperative report output method provided in this application can be executed by any one of the split - screen device, the main control device, and the server, or jointly executed by multiple devices.

[0019] Before the operation, the operator can use the split - screen device and the main control device for real - time surgical monitoring by extracorporeal circulation doctors for preoperative settings, specifically including: The operator, using the main control device, turns on the laser emitter. The laser emitter projects a cross calibration cursor. The operator obtains the projection position of the cross calibration cursor and manually moves the camera of the main control device to perform shooting position calibration. After calibration, the operator uses the buttons on the display screen of the split screen device to adjust the focal length and magnification of the camera of the main control device, and adjusts the clarity of the picture by viewing the picture on the display screen of the split screen device. During the operation, the operator can view the operation picture in real time in a live broadcast manner through the display screen of the split screen device to understand the current operation site and operation progress.

[0020] In addition, during the operation, the main control device can automatically generate an operation record sheet. The operation record sheet specifically includes: liquid / drug name, pre recharge, whether in circulation, total amount, gas flow rate, perfusion flow rate, arterial pressure, venous pressure, heparin in vivo, Tn, TR, HCT, SvO2, K+, Lac, rSO2, etc. The main control device can read the data corresponding to each physiological monitoring index in the operation record sheet from the split screen device for real-time operation monitoring by the extracorporeal circulation doctor to obtain physiological monitoring data. The built-in program of the CPU of the main control device forms key value pairs with the physiological monitoring data and the physiological monitoring indexes, stores them and fills them in the operation record sheet.

[0021] In addition, during the operation, the main control device can obtain operation process related data from the server, use each process name as the operation process name, remove specific process special symbols, segment according to semicolons and carriage returns, form key value pairs of steps, and use them as the operation step table of the operation tutorial, and transmit them to the memory card of the main control device for storage. The operation tutorial specifically includes: an operation step table and an operation instruction record table. The operation step table includes the operation process name, operation steps and corresponding operations. The operation instruction record table includes instruction text segments.

[0022] For example, the operation step table of extracorporeal circulation surgery can be expressed as: "Operation process name: I. Preparation work.

[0023] Step 1: Prepare consumables, including oxygenator, tubing set, cardioplegia device, cardioplegia solution, HTK solution, del Nido solution, perfusion needle, left heart suction tube, carbon dioxide insufflation tube, air filter, etc.; Step 2: Select an appropriate cannula size according to the patient's physical condition (body mass, blood vessel diameter, CT results), cannulate the femoral artery, femoral vein, and superior vena cava through neck puncture; Step 3: Obtain crystalloid solution, place it in the extracorporeal circulation machine, and perform priming; Step 4: The doctor injects 3 mg / kg of heparin into the femoral vein; Step 5: If the ACT > 300 s and the mean arterial pressure ≈ 80 mmHg, adjust the extracorporeal circulation rate to 0.5 - 0.8 L / min and remind the doctor to cannulate the arterial vessel; Step 6: Obtain surgical forceps, surgical scissors, and the venous end tube of the oxygenator. Clamp the venous end tube of the oxygenator with the surgical forceps and cut the tube with the surgical scissors; Step 7: Open the main pump protection cover, manually adjust the tightness of the pump tube so that the water column drops at a speed of 1 - 2 cm / min, or the pump pressure is 200 mmHg and does not drop by more than 20 mmHg in 30 s. If the patient's blood pressure is low, remind the operator to turn on the extracorporeal circulation machine as soon as possible.

[0024] Surgical procedure name: II. Power on.

[0025] Step 1: Obtain the extracorporeal circulation power-on time and record it on the operation record sheet; Step 2: The operator turns on the gas source, slowly turns on the main pump and loosens the superior vena cava tubing, checks the oxygenator liquid level, and adjusts the pump flow rate to 60 - 80 mL per kilogram of body weight; Step 3: Obtain the negative pressure of the control panel. If the pressure value is between 20 - 30 mmHg, lower the perfusion system to 2°C and cool the patient's core temperature at a rate of 0.7 - 1.5°C / min; Step 4: Obtain the pump pressure, temperature of the extracorporeal circulation machine, and the patient's blood pressure. If the pump pressure is below 200 mmHg, the temperature is 31°C - 32°C, and the blood pressure is between 50 - 80 mmHg, it is safe. If the pump pressure is 200 - 250 mmHg, be highly vigilant. If the pump pressure exceeds 250 mmHg, remind the doctor to check the arterial line.

[0026] Surgical procedure name: III. Cross-clamp the aorta.

[0027] Step 1: Obtain the patient's blood pressure, surgical forceps, the circulation pipeline of the perfusion system and the oxygenator. According to the patient's blood pressure, use the surgical forceps to clamp the circulation pipeline between the perfusion system and the oxygenator; Step 2: Obtain the perfusion tube, loosen the buckle of the perfusion tube, wait for the doctor's notice, and the operator quickly drains 0.3 L / min of perfusion fluid into the perfusion tube and then stops the operation, waiting for further instructions from the doctor; Step 3: Obtain the nasopharyngeal temperature. If the nasopharyngeal temperature reaches 34 °C, remind the doctor that the surgical range has been reached, obtain the left heart drainage pump, turn on the left heart drainage pump, and adjust its flow rate to 0.2 L / min; Step 4: Observe the drainage situation of the oxygenator when the superior and inferior vena cava are blocked. If it is normal, report to the doctor the drainage status after the venous block; Step 5: Obtain the doctor's instruction to reduce the flow rate or block, set the main pump flow rate to 1 L / min. If the block is completed, increase the main pump flow rate to 60 - 80 mL / kg.

[0028] Surgical procedure name: IV. Perfusion.

[0029] Step 1: Obtain the patient's aortic valve. If the aortic valve is moderately stenotic or more, patients with aortic regurgitation should perform left and right perfusion after opening the aorta. Otherwise, perfusion can be carried out immediately after blocking; Step 2: If performing antegrade perfusion, obtain the perfusion flow rate and perfusion pump pressure, increase the perfusion flow rate. If the perfusion pump pressure increases abnormally, reduce the perfusion flow rate until the perfusion pump pressure is 200 mmHg. Otherwise, increase the perfusion flow rate to 0.32 L / min and the perfusion pump pressure to 250 mmHg; If performing retrograde perfusion, obtain the perfusion pump pressure and maintain the perfusion pump pressure at 25 - 50 mmHg; Step 3: After perfusion, obtain the high perfusion pathway, low perfusion pathway, surgical forceps, and perfusion dose. Use the surgical forceps to clamp the high perfusion pathway, open the low perfusion pathway, and adjust the perfusion dose to normal; Step 4: Obtain the ACT and blood gas analysis results. According to the results, obtain the dosages of potassium and sodium bicarbonate needed; Step 5: Obtain the patient's bladder temperature, control the patient's bladder temperature at 30 °C - 32 °C, use blood perfusion fluid for blocking, wait for 30 minutes and then remind the doctor of the perfusion time, obtain the perfusion system, cool down the perfusion system, obtain the time for re-perfusion. If the time for re-perfusion is 6 - 8 minutes, re-check the blood gas or ACT; Step 6: Obtain the surgical time. If the surgical time is 30 minutes, re-check the blood gas. If the surgical time is 60 minutes, re-check the ACT; Step 7: Obtain the patient's HCT (hematocrit), blood potassium, and fluid volume. If the HCT (hematocrit) is too low, the blood potassium is too high, or the fluid volume is too much, perform routine ultrafiltration.

[0030] Surgical procedure name: V. Rewarming.

[0031] Step 1: Obtain the doctor's instructions, surgical stage, and oxygen concentration. If the doctor's instructions are for rewarming or the surgical stage is the finishing stage, increase the oxygen concentration to 70%. Step 2: Obtain the blood gas condition, patient's body temperature, water temperature, and rewarming rate. According to the blood gas condition, set the patient's body temperature at 35 - 38 °C, the water temperature less than 42 °C, and the difference from the patient's body temperature not exceeding 8 °C. Control the rewarming rate at 0.2 - 0.5 °C / min.

[0032] Surgical procedure name: VI. Root venting.

[0033] Step 1: If reverse aspiration for venting, obtain the surgical forceps, oxygenator, perfusion system, and blood reservoir. Using the surgical forceps, clamp the bypass of the oxygenator leading to the perfusion system and loosen the passage between the blood reservoir and the perfusion system. If in the pre-loosening state, slowly aspirate gas at a rate of 0.05 - 0.1 L / min; otherwise, the root back-aspiration rate is 0.4 L / min and the left heart back-aspiration rate is 0.2 - 3 L / min. Step 2: If active venting, obtain the three-way connector on the perfusion needle and connect the third suction using the three-way connector on the perfusion needle.

[0034] Surgical procedure name: VII. Aorta opening.

[0035] Step 1: Obtain the doctor's instructions and the main pump flow rate. If the doctor's instructions are to reduce the flow rate, reduce the main pump flow rate to below 1 L / min. If the doctor's instructions are to loosen the occlusion clamp to open, add the main pump flow rate and record the occlusion time. Step 2: Obtain the patient's blood pressure, heart re-beating condition, main pump flow rate, and the temperature of the temperature control machine. Adjust the main pump flow rate so that the patient's blood pressure is below 100 mmHg and adjust the temperature control machine to 38 °C. Step 3: Obtain the time to complete aorta opening and the blood gas analysis results. If the aorta opening time is 5 min, adjust the electrolytes and oxygen flow rate using the blood gas analysis results.

[0036] Surgical procedure name: VIII. Shut down.

[0037] Step 1: Obtain transesophageal echocardiogram, post - bypass time, aortic cross - clamp time, CPB main pump flow, blood volume, CVP, hepatic temperature, nasopharyngeal temperature, HCT, blood gas, and electrolytes. If there is no cardiac structural abnormality in the transesophageal echocardiogram, if the post - bypass time is 1 / 4 to 1 / 3 of the aortic cross - clamp time, if the CPB main pump flow can maintain a satisfactory arterial pressure, if the blood volume is basically replenished and the CVP is satisfactory, if the hepatic temperature > 35°C, the nasopharyngeal temperature is 36°C - 37°C, if the HCT > 28%, and if the blood gas and electrolytes are basically normal, and if one of these conditions is met, then we can stop the machine; Step 2: Obtain the patient's blood pressure, cardiac resuscitation status, main pump flow, and the temperature of the temperature - changing machine, and adjust the main pump flow so that the patient's blood pressure is below 100 mmHg, and adjust the temperature - changing machine to 38°C.

[0038] Surgical procedure name: IX. Autologous blood transfusion.

[0039] Step 1: Obtain the doctor's instruction. If the instruction is venous back - draw, then use the venous pipeline blood to back - draw into the blood storage tank, and turn off the alarm switch on the control panel. Obtain the liquid level and the main pump flow, observe whether the liquid level drops, and adjust the main pump flow; Step 2: Obtain the doctor's instruction. If the instruction is back - draw, then use the main pump to return the blood in the arterial pipeline to the blood storage tank. Obtain the surgical forceps, arterial pipeline, and the direction of the main pump. Use the surgical forceps to clamp the arterial pipeline, and adjust the direction of the main pump to rotate counterclockwise. After obtaining the surgical procedure list, keywords similar to "notify", "remind", "instruction", "issue", etc. can be extracted from the surgical procedure list through text extraction technology, and the relevant text segments of the keywords can be entered into the surgical instruction record table in the form of key - value pairs.

[0040] For example, in the "Obtain the doctor's issued reduced - flow or occlusion instruction, set the main pump flow to 1 L / min, and if the occlusion is completed, then increase the main pump flow to 60 - 80 mL / kg" recorded in the above example, extract the keyword "issued", and obtain the relevant text segment "reduced - flow or occlusion instruction" and record it in the instruction history record table.

[0041] In addition, during the surgical process, the microphone of the main control device can also obtain the voice instructions issued by the doctor and record the instruction time. The main control device can use speech recognition technology to convert the voice instructions into text instructions. If the confidence level of the text instructions is lower than the threshold, then partial correction can be performed through text correction technology. The CPU of the main control device obtains the surgical instruction record table, and performs fuzzy matching between each instruction record in the surgical instruction record table and the transcribed text instructions to obtain the most - matched instruction record, and updates the surgical instruction record table based on the instruction time.

[0042] If the voice command issued by the doctor again, and the time interval between its command time and the time of the previous voice command is within 20 minutes, then through text similarity technology, judge the similarity between the previous voice command and the current voice command. If the similarity is higher than the set threshold, it is determined that the previous voice command is invalid, and only the command issued by the current operator is taken.

[0043] For example: The operating doctor issues the command "Turn on the extracorporeal circulation machine", record the time of the command: 2022-10-10 10:11:30, transcribe the command through voice recognition technology, and combine natural language processing technology to obtain the command "Turn on the extracorporeal circulation machine". Subsequently, the operating doctor issues the command "Turn on the extracorporeal circulation", record the time of the command: 2022-10-10 10:12:30, transcribe the command through voice recognition technology, calculate the time interval between the two commands as 1 minute, which is less than the threshold of 20 minutes, and combine text similarity technology to obtain the similarity between the two commands as 0.85, and delete the record of the previous command.

[0044] So far, by using the split-screen device for real-time surgical monitoring of the extracorporeal circulation doctor and the main control device to monitor each surgical stage of the main operation, and the split-screen device and the main control device can be set in different surgical areas respectively. The extracorporeal circulation doctor can view the surgical progress of the main operation in the split-screen device, without crowding around the operating table of the main operation, which can ensure that the extracorporeal circulation doctor can timely master the surgical progress while avoiding the main surgeon being unable to concentrate on the operation.

[0045] Furthermore, during the process of using the split-screen device and the main control device to monitor the main operation, it may be due to the unfamiliar use of the two newly introduced devices by the extracorporeal circulation doctor or the main surgeon, resulting in possible mistakes or errors during the operation, such as the surgical process of the main operation being interrupted. In order to better review the surgical process of the main operation with the split-screen device and the main control device introduced, relevant data collected during the operation by the split-screen device and the main device can be used to generate corresponding postoperative reports, which is beneficial for the main surgeon and the extracorporeal circulation doctor to use the postoperative reports for postoperative summary and review, thereby improving the proficiency in using these two devices in subsequent operations. For this reason, a method for outputting a postoperative report for the extracorporeal circulation doctor's split-screen operation provided by this application is applied to any one or more of the split-screen device, the main control device, and the server for real-time surgical monitoring of the extracorporeal circulation doctor. The method includes steps 110-step 140 as Figure 1 shown.

[0046] Step 110: Identify the surgical stage of the main operation based on the surgical images collected by the main control device.

[0047] As described above, the master device can use a camera to collect the images of the main operation, and the images include surgical images. The surgical stage of the main operation can be recognized from the surgical images by using image recognition technology. Among them, after the main operation is completed, the main operation video collected by the master device can be obtained, and the surgical images at preset time intervals or preset frame numbers in the main operation video can be recognized to determine the surgical stages corresponding to the respective surgical images.

[0048] Step 120: Obtain the first interaction behavior data of the extracorporeal circulation doctor and the split-screen device, the second interaction behavior data of the extracorporeal circulation doctor and the surgeon, and the physiological monitoring data of the patient.

[0049] Exemplarily, the split-screen device can collect the operation data of the extracorporeal circulation doctor on the split-screen device to form the first interaction behavior data. In addition, the second interaction behavior data of the extracorporeal circulation doctor and the surgeon can be collected by the split-screen device, the master device, or other image acquisition devices installed in the operating room. The second interaction behavior data is used to characterize whether there is a conversation behavior between the extracorporeal circulation doctor and the surgeon. In addition, the master device can obtain the physiological monitoring data corresponding to each physiological monitoring index from the operation record sheet. The physiological monitoring data can be obtained from the split-screen device and then sent to the master device.

[0050] Step 130: Input the surgical images, surgical stages, first interaction behavior data, second interaction behavior data, and physiological monitoring data into the trained interference detection large model, so that the interference detection large model maps the first interaction behavior data to the virtual standing posture data of the extracorporeal circulation doctor based on a preset mapping relationship, and when predicting that the surgeon is in a surgical focused state based on the virtual standing posture data, surgical images, surgical stages, and physiological monitoring data, detect whether the extracorporeal circulation doctor interferes with the surgeon based on the second interaction behavior data; the training data of the interference detection large model includes historical surgical videos with surgical stage labels and surgical state labels, historical physiological monitoring data, and historical standing posture data of the extracorporeal circulation doctor.

[0051] Step 140: Output a postoperative report, and the postoperative report includes the interference detection result output by the interference detection large model.

[0052] As an example, the input process of the model parameters can include: generating a first prompt word based on the surgical images, surgical stages, first interaction behavior data, second interaction behavior data, physiological monitoring data, and a preset first prompt word template, and inputting the first prompt word into the trained interference detection large model.

[0053] It is understandable that before using the split-screen device and the master control device to monitor the main surgery, the extracorporeal circulation doctor and the surgeon will stand around the operating table. The surgeon performs the main surgery. During the critical stages of the surgery, the extracorporeal circulation doctor may observe the progress of the main surgery and may communicate with the surgeon. During the non-critical stages, the extracorporeal circulation doctor may leave the operating table. Therefore, the historical standing posture data of the extracorporeal circulation doctor during historical surgeries can be obtained. Specifically, during historical surgeries, the standing posture images of the extracorporeal circulation doctor can be collected. Subsequently, tools such as OpenPose or MediaPipe are used to parse the standing posture images, and the standing posture model data is extracted from the standing posture images. The standing posture model data includes skeletal key points. The skeletal key points include, but are not limited to, the skeletal key points corresponding to the head, shoulders, hands, and feet. Subsequently, the relative position changes of the extracorporeal circulation doctor can be extracted from the standing posture model data, including approaching the surgeon, moving away from the surgeon, raising the head, and lowering the head, etc. And the movement trajectory is calculated based on the relative position changes of the extracorporeal circulation doctor at each moment, such as the movement trajectory of the extracorporeal circulation doctor approaching the extracorporeal circulation machine, or the movement trajectory of how to move away from the surgeon, etc. And the movement trajectory of the extracorporeal circulation doctor can be recognized through a deep learning model, such as a 3D convolutional neural network (3D-CNN) model. The deep learning model can extract action features from the movement trajectory in the time dimension, such as actions like "approaching", "moving away", "raising the head", etc. Subsequently, a long short-term memory network (LSTM) or a bidirectional LSTM (Bi-LSTM) network is applied to recognize the behavior pattern of the extracorporeal circulation doctor according to the continuous action sequence. For example, when the doctor's action sequence suddenly shows "approaching" or "standing straight", it can be inferred through pattern learning that the surgery may have entered a critical stage.

[0054] In the training stage of the interference detection large model, the training data includes historical surgery videos with surgery stage labels and surgery status labels, historical physiological monitoring data, and historical standing posture data. The surgery status labels include a concentration label and a non-concentration label. In this way, what the interference detection large model learns is the surgery status corresponding to different physiological monitoring data and different standing postures of the extracorporeal circulation doctor at each surgery stage.

[0055] However, after using the split-screen device and the master control device to monitor the main surgery, the extracorporeal circulation doctor can directly view the progress of the main surgery through the split-screen device. And the split-screen device provides a microphone, and the extracorporeal circulation doctor and the surgeon can communicate through the split-screen device, and the extracorporeal circulation doctor no longer needs to stand around the operating table. Therefore, in the usage stage of the interference detection large model, the standing posture data of the extracorporeal circulation doctor is lacking as model input. To solve this problem, in this embodiment, a mapping relationship between the first interaction behavior data and the standing posture data is established in advance. The mapping relationship is shown in Table 1.

[0056] Table 1 Thus, when the surgical image, surgical stage, first interaction behavior data, second interaction behavior data, and physiological monitoring data are input into the trained interference detection large model, the interference detection large model can call the pre-established mapping relationship. First, it maps the input first interaction behavior data to the virtual standing posture data of the extracorporeal circulation doctor using the mapping relationship. The reason it is virtual standing posture data is that the standing posture data of the extracorporeal circulation doctor is not actually collected at the surgical site. Instead, based on the first interaction behavior data, it is speculated how the extracorporeal circulation doctor might stand at the surgical site if the split screen device and the main control device were not used. Then, the interference detection large model predicts the surgical state of the surgeon based on the virtual standing posture data, surgical image, surgical stage, and physiological monitoring data. The surgical state includes a surgical focused state and a surgical unfocused state. If it is predicted that the surgeon is in the surgical focused state, then the interference detection large model will further detect whether the extracorporeal circulation doctor interferes with the surgeon based on the second interaction behavior data. If it is predicted that the surgeon is in the surgical unfocused state, even if the second interaction behavior data indicates that there is a conversation behavior between the extracorporeal circulation doctor and the surgeon, it can be considered that such a conversation will not interfere with the surgeon. Therefore, the next frame of the surgical image can be obtained, and steps 110 - 130 can be returned for execution. Finally, the interference detection result output by the interference detection large model can be obtained.

[0057] After traversing the main surgical video, a postoperative report can be output. The postoperative report includes the interference detection result output by the interference detection large model. Of course, in addition to the interference detection result, the postoperative report can also include other content. As an optional example, if the interference detection result indicates that the extracorporeal circulation doctor interferes with the surgeon during the operation, the detection result can be highlighted in the postoperative report.

[0058] Optionally, a virtual image of the extracorporeal circulation doctor can also be generated based on the virtual standing posture data, and the virtual image can be superimposed and displayed on the surgical screen in the extracorporeal circulation surgery monitoring device. At the same time, gestures or voices can be used for perspective switching to view the virtual image of the extracorporeal circulation doctor from different angles.

[0059] It can be seen that in this embodiment, the large model is used to predict whether the surgeon is in a surgical focused state based on the surgical image, the first interaction behavior data of the extracorporeal circulation doctor and the split screen device, the physiological monitoring data of the patient, and the pre-established mapping relationship, and to detect whether the extracorporeal circulation doctor interferes with the surgeon when the surgeon is focused, and then generate a postoperative report. Thus, it is beneficial for the surgeon and the extracorporeal circulation doctor to use the postoperative report for postoperative summary and review, thereby improving the proficiency in using the split screen device and the main control device in subsequent surgeries.

[0060] The following provides a detailed introduction to steps 110 - 140.

[0061] According to some embodiments of the present application, before performing step 110, steps S1 - S4.2 may be performed first.

[0062] Step S1: Obtain the picture collected by the master device.

[0063] Exemplarily, the master device may use a camera to collect pictures, thereby obtaining the main surgical video. It can be understood that each frame of the picture in the main surgical video is not necessarily a surgical image. The picture may include surgical images, pre - surgical images, and post - surgical images. Therefore, the picture collected by the master device needs to go through the following steps to determine whether it is a surgical image.

[0064] Step S2: If it is determined that the picture has the image content of a surgical incision using an image classification algorithm, determine that the picture is a candidate image.

[0065] Exemplarily, an image classification algorithm can be combined to determine whether there is the image content of a surgical incision in the picture. If it is determined that the picture does not have the image content of a surgical incision, it is determined that the picture is not a surgical image, and the next picture is obtained from the main surgical video and returned to execute step S2.

[0066] If the picture has the image content of a surgical incision, obtain the confidence level of the image classification result. If the confidence level is greater than or equal to a preset third confidence threshold, for example, 0.8, it is determined that the picture is a candidate picture, and it is determined whether the candidate picture is a surgical picture by performing step S3.

[0067] If the confidence level is less than the third confidence threshold, the current picture can be manually assisted to determine whether it is a surgical picture. At the same time, the manual assistance determination result of the picture can be collected as a training data set for subsequent training of the image classification algorithm.

[0068] Step S3: Use an image segmentation algorithm to segment the entity name, image position, and segmentation credibility of the object from the candidate image; the object includes incisions, blood, doctor's hands, patient organs, and surgical tools.

[0069] Exemplarily, an image segmentation algorithm such as AutoSam can be used to segment the entity name, image position, and segmentation confidence of an object from a candidate image. The five major types of objects obtained include incisions, blood, the doctor's hand, patient organs (such as the femoral artery, femoral vein, myocardium), and surgical tools (such as a left heart suction tube, a carbon dioxide insufflation tube, a scalpel). That is, it is possible to segment from the candidate image an incision and its image position and segmentation confidence, blood and its image position and segmentation confidence, the doctor's hand and its image position and segmentation confidence, patient organs and their image positions and segmentation confidence, and surgical tools and their image positions and segmentation confidence.

[0070] Step S4.1: When the ratio of the first image area of the blood in the candidate image to the second image area of the incision in the candidate image is less than a preset ratio threshold, if the first segmentation confidence of the patient organ and the second segmentation confidence of the surgical tool in the candidate image are greater than a first confidence threshold, then determine that the candidate image is the surgical image; if the first segmentation confidence or the second segmentation confidence is less than the first confidence threshold, then obtain the next candidate image and return to execute step S3 until a surgical image is determined.

[0071] Exemplarily, the pixel position of the blood in the candidate image can be obtained, and based on the pixel position, the first image area of the blood in the candidate image can be calculated. And the pixel position of the incision in the candidate image can be obtained, and based on this pixel position, the second image area of the incision in the candidate image can be calculated. The specific process of calculating the image position based on the pixel position can be referred to the related technology and will not be elaborated here. After obtaining the first image area and the second image area, it is possible to determine that the area percentage of the blood occupying the incision in the candidate image is the ratio of the first image area to the second image area.

[0072] If the area percentage of the blood occupying the incision is less than a preset ratio threshold, for example, 50%, then continue to determine whether the first segmentation confidence of the patient organ and the second segmentation confidence of the surgical tool in the candidate image are both greater than a preset first confidence threshold, for example, 0.8. When the area percentage of the blood occupying the incision is less than the preset ratio threshold and both the first segmentation confidence and the second segmentation confidence are greater than the first confidence threshold, determine that the candidate image is the surgical image and the surgical image is credible.

[0073] When the area percentage of the blood occupying the incision is less than the preset ratio threshold and the first segmentation confidence is less than the first confidence threshold, then obtain the next candidate image through the above steps S1 - S2 and immediately return to execute step S3 until a surgical image is determined.

[0074] In the case where the area percentage of the incision occupied by blood is less than a preset ratio threshold and the second segmentation confidence is less than the first confidence threshold, the next candidate image is obtained through the above steps S1 - S2, and step S3 is immediately returned for execution until the surgical image is determined.

[0075] Step S4.2: In the case where the ratio of the area of the first image to the area of the second image is greater than the ratio threshold, the next candidate image is obtained at a preset interval time, and step S3 is returned for execution until the surgical image is determined.

[0076] Exemplarily, if the area percentage of the incision occupied by blood is greater than the ratio threshold, at this time, regardless of whether the first segmentation confidence of the patient's organ or the second segmentation confidence of the surgical tool is greater than the first confidence threshold, the next candidate image is obtained through the above steps S1 - S2, and step S3 is returned for execution at a preset time interval, such as 10s, until the surgical image is determined.

[0077] In addition, after obtaining the surgical image, image classification technology can be combined to determine whether the surgical tool in the surgical image belongs to a dangerous tool. If there is a dangerous tool in the surgical image, the surgical state is determined to be a dangerous state; otherwise, the surgical state is determined to be a safe state.

[0078] It can be seen that in this embodiment, first, the image classification algorithm is used to classify the real - time images with surgical incision image content from multiple frames as possible candidate images. Subsequently, the image segmentation algorithm is used to identify the surgical incision, blood, doctor's hand, patient's organ, and surgical tool from the candidate images, and based on the image area ratio between the incision and the blood, and the segmentation confidence of the patient's organ and the surgical tool, it is further determined whether the candidate image is a credible surgical image, thus determining a credible surgical image from the frames by combining multiple algorithms and multiple dimensions, improving the accuracy of subsequent interference detection.

[0079] Based on any of the above - mentioned embodiments, regarding identifying the surgical stage based on the surgical image in step 110, it may specifically include steps 111 - 116.

[0080] Step 111: Determine the association result between the target instruction in the surgical instruction record table and the surgical image, and the instruction value of the target instruction.

[0081] Among them, the process of obtaining and updating the surgical instruction record table is as described in the above embodiments and will not be elaborated here. The surgical instruction record table records multiple surgical instructions, their instruction values, and instruction times. The target instruction in the surgical instruction record table can refer to the instruction with the closest instruction time to the current time. Specifically, the gap between the instruction time of the target instruction and the current time can be judged. If the time gap exceeds a preset time threshold, such as 60 minutes, then the association result between the target instruction and the surgical image is irrelevant; if the time gap does not exceed the time threshold, then the association result between the target instruction and the surgical image is relevant.

[0082] Step 112: Input the association result, the instruction value, the entity name and segmentation credibility of the object in the surgical image, the surgical image, and the physiological monitoring data into the random forest model to obtain the first surgical stage output by the random forest model.

[0083] Subsequently, the association result obtained above, the instruction value of the target instruction, the entity name and segmentation credibility of the object in the surgical image, the surgical image, and the physiological monitoring data can be input into the trained random forest model. The random forest model is used to predict the first surgical stage by combining the current data obtained from each data source.

[0084] For example, the target instruction obtained from the surgical instruction record table is "reduce flow rate", and the evaluation result of the association between the target instruction and the surgical image is "associated", and the instruction value of the target instruction is "reduce flow rate". The entity name and confidence of the organ obtained in step S3 are: aorta, 0.87. And the obtained physiological monitoring data includes: air flow rate 0.2 L / min, arterial pressure 98 mmHg. The above data is processed through the random forest machine learning algorithm to obtain the classification probabilities of 9 surgical stages, including: preparation: 0.12, power on: 0.12, aortic occlusion: 0.14, perfusion: 0.23, rewarming: 0.23, root exhaust: 0.33, aortic opening: 0.92, shutdown: 0.43, autologous blood transfusion: 0.34. Determine the surgical stage with the highest classification probability as the first surgical stage, that is, "aortic opening".

[0085] Step 113: Input the embedding vector of the surgical image into the ResNet model to obtain a set of preset number of video frames determined from the surgical tutorial videos in the order of similarity from high to low between the embedding vector of the surgical image and the embedding vectors of the pre-stored surgical tutorial videos in the ResNet model; wherein, the surgical tutorial videos carry surgical action labels, and the ResNet model is trained using the surgical tutorial videos.

[0086] Step 114: Determine the similarity between the target video frame with the highest similarity in the set of video frames and the target video frame, and determine the second surgical stage based on the surgical action label carried by the target video frame.

[0087] Exemplarily, the ResNet model is used to predict the surgical stage based on the surgical image. Regarding the training process of the ResNet model, the surgical tutorial videos stored in the server can be read, and the video frames with action labels are extracted from the surgical tutorial videos as the training samples of the ResNet model. By calculating the loss of the similarity of the same label samples, the ResNet model can identify several video frames with the highest similarity to the embedding vector of the surgical image, and after the operation, the operator can confirm the correctness of this image recognition. The correct surgical image and the successfully matched video image can be added to the model training set as a new sample pair.

[0088] Subsequently, after the embedding vector of the surgical image is input into the ResNet model, the ResNet model can calculate the similarity between the embedding vector of the surgical image and the embedding vectors of the pre-stored surgical tutorial videos. And in the order of similarity from high to low, a preset number of video frames are obtained from the surgical video tutorial to form a set of video frames. Since the video frames carry surgical action labels, the target video frame with the highest similarity can be determined from the set of video frames, and the surgical action label carried by the target video frame is determined as the second surgical stage, and the similarity of the target video frame is obtained at the same time.

[0089] Subsequently, by comparing the size relationship between the similarity of the target video frame and the preset similarity threshold, step 115 or step 116 is selected to be executed.

[0090] Step 115: If the similarity of the target video frame exceeds the preset similarity threshold, determine the second surgical stage as the surgical stage.

[0091] Exemplarily, if the similarity of the target video frame exceeds the preset similarity threshold, such as 0.9, it means that the target video frame is very similar to the surgical image corresponding to the second surgical stage in the surgical video tutorial. At this time, it can be considered that the recognition result of the second surgical stage is credible. Therefore, the second surgical stage can be determined as the surgical stage.

[0092] Step 116: If the similarity of the target video frame does not exceed the similarity threshold, determine the first surgical stage as the surgical stage.

[0093] Exemplarily, if the similarity of the target video frame does not exceed the similarity threshold, such as 0.9, it means that the target video frame is not very similar to the surgical image corresponding to the second surgical stage in the surgical video tutorial. At this time, the first surgical stage determined by the random forest can be taken as the surgical stage.

[0094] For example, if the embedding vector of the surgical image is input into the ResNet model, the ResNet model calculates the top 10 video frames with the highest similarity to the embedding vector of the surgical image and the corresponding similarities as follows: venous cannulation: 0.12, antegrade perfusion: 0.12, retrograde perfusion: 0.14, perfusion and exhaust: 0.23, rewarming: 0.23, aspiration and exhaust: 0.33, aortic opening: 0.92, adjusting the cooling machine: 0.43, active exhaust: 0.34, autologous blood transfusion: 0.33. Assuming that the similarity threshold is 0.9, it can be seen that the similarity of "aortic opening" exceeds the similarity threshold. Therefore, "aortic opening" is determined as the surgical stage, and its confidence level is obtained. On the contrary, assuming that the similarity threshold is 0.95, then the similarities of the above 10 video frames do not exceed the similarity threshold. At this time, the first surgical stage output by the random forest model is used as the surgical stage.

[0095] It can be seen that in this embodiment, the random forest model and the ResNet model respectively predict the surgical stage based on the surgical image, and then determine whether to adopt the output result of the random forest model or the ResNet model as the surgical stage based on the similarity of the first surgical stage output by the ResNet model. Through the joint prediction of the two models, the prediction accuracy of the surgical stage can be improved, thereby ensuring the accuracy of subsequent interference detection.

[0096] Based on any of the above embodiments, regarding identifying the surgical stage in step 110, it may specifically include steps 117 - 118.

[0097] Step 117: Obtain the initial surgical stage output after classifying the surgical image by the image classification model; the surgical stage recognition model includes a random forest model and / or a ResNet model; wherein, the confidence level of the initial surgical stage is less than a preset second confidence threshold.

[0098] In some embodiments, the first surgical stage output by the random forest model can be obtained through the above steps 111 - 112 as the initial surgical stage, and the confidence level of the first surgical stage is used as the confidence level of the initial surgical stage, and / or the second surgical stage output by the ResNet model can be obtained through the above steps 113 - 114 as the second initial surgical stage, and the confidence level of the second initial surgical stage is used as the confidence level of the initial surgical stage. That is, the initial surgical stage can include one or two results.

[0099] In the case where the confidence level in the initial surgical stage is less than the second confidence threshold, it indicates that neither the random forest model nor the ResNet model can accurately predict the surgical stage. At this time, a large model with a larger number of parameters and stronger learning ability can be used to identify the surgical stage.

[0100] Step 118: Input the initial surgical stage, the surgical image, the first interaction behavior data, and the physiological monitoring data into the trained large surgical stage recognition model, so that the large surgical stage recognition model determines whether the initial surgical stage is credible based on the surgical image, the physiological monitoring data, and the first interaction behavior data, and determines the surgical stage based on the surgical image, the physiological monitoring data, and the first interaction behavior data when the initial surgical stage is not credible.

[0101] As an example, the initial surgical stage, the surgical image, the first interaction behavior data, and the physiological monitoring data can be input into a preset second prompt template to generate a second prompt, and the second prompt is input into the large surgical stage recognition model. The following is one exemplary example of the second prompt.

[0102] "Role: You are a real-time surgical assistance system, responsible for analyzing surgical images, process data, physiological parameters, as well as the split-screen interaction behavior and standing posture data of the extracorporeal circulation doctor, and assisting in judging the surgical stage corresponding to the surgical image.

[0103] Task: I need you to judge the surgical stage based on the current surgical image information, physiological monitoring data, and the trained model. Through image recognition and process analysis, provide confirmation information for the current stage. At the same time, please combine the interaction operations of the extracorporeal circulation doctor on the split-screen display (such as camera adjustment, laser marking, voice interaction, etc.) and the corresponding standing posture mapping information to assist in judging the current surgical state and stage recognition.

[0104] Input parameters: surgical video frame; initial surgical stage (obtained through an image classification model); real-time physiological monitoring data of the patient (blood pressure, heart rate, blood oxygen, etc.); split-screen interaction behavior data of the extracorporeal circulation doctor (such as: adjusting the camera focus / magnification, turning on / off the microphone, using laser marking, manual graffiti, undoing / clearing graffiti, exiting the live interface, adjusting the display parameters, etc.).

[0105] Reference logic: Use an image recognition model (such as ResNet) to analyze the surgical video frame and calculate the similarity with the stored surgical tutorial video. Based on the similarity and the initial surgical stage (such as perfusion, air exhaust, etc.), determine the current surgical stage. If the confidence level exceeds the set threshold (such as 0.9), confirm that the initial surgical stage is the surgical stage of the current surgical image; if it is lower than the threshold, initiate other analyses (such as physiological monitoring data such as incision bleeding and heart contraction state).

[0106] Exemplary examples of using extracorporeal circulation doctor interaction behavior data and virtual standing posture data to assist in surgical stage identification are as follows: 1) If the interaction behavior is to adjust the camera focus (focus + / focus -), the mapped virtual standing posture is that the doctor approaches the surgeon and looks down to observe key details. At this time, if the similarity between the surgical image and the video of the "opening the aorta" step is higher than 0.9, and it is detected that the doctor frequently adjusts the camera focus (indicating approaching to observe), it is determined that the current surgical stage is "opening the aorta", and the "perfusion" stage is about to be entered.

[0107] 2) If the interaction behavior is to adjust the camera magnification (magnification + / magnification -), the mapped virtual standing posture is that the doctor stands farther away to observe the overall situation. At this time, if the similarity between the surgical image and the video of the "perfusion" step is higher than 0.9, and it is detected that the doctor adjusts the magnification, it can be determined that the current surgical stage is perfusion.

[0108] 3) If the interaction behavior is to turn on autofocus, the mapped virtual standing posture is that the doctor maintains a stable standing posture and waits for the key points of the surgery to appear, indicating that the doctor is in a stable monitoring state. At this time, if it is determined that the current surgical image matches the perfusion stage by combining image recognition data, the system maintains the current stage and does not need to trigger a stage transition.

[0109] 4) If the interaction behavior is to turn on / off the microphone, the mapped virtual standing posture is that the extracorporeal circulation doctor communicates frequently with the surgeon, possibly accompanied by approaching movements. At this time, if it is determined that it is a critical moment (such as before perfusion start or before preparation for extracorporeal circulation withdrawal) by combining surgical image recognition and it is detected that the microphone is frequently turned on and off in a short period of time, this behavior should be noted, indicating an increased risk of stage transition.

[0110] 5) If the interaction behavior is to use laser to mark the image, the mapped virtual standing posture is that the doctor points at the surgical area with a finger to mark abnormal points. If the extracorporeal circulation doctor uses laser to mark on the split screen and is accompanied by voice communication, it can be considered as a high-risk signal, indicating that the next stage is about to enter.

[0111] 6) If the interaction behavior is manual graffiti marking, undo / clear graffiti, the mapped virtual standing posture is that the extracorporeal circulation doctor repeatedly evaluates the risk area through gestures and modification prompts. At this time, if it is determined by combining image recognition that this behavior frequently appears in the "aorta occlusion" or "root air exhaust" stage, it may indicate a risk adjustment period and predict a stage transition in advance.

[0112] 7) If the interaction behavior is to exit the surgical live broadcast interface, the mapped virtual standing posture is that the doctor leaves the side of the surgeon in charge, indicating that the current stage has ended or the risk has decreased. For example, after perfusion is completed, it enters the shutdown stage. However, if exiting during a critical stage, the system needs to confirm whether to conduct a handover of responsibilities.

[0113] 8) If the interaction behavior is to adjust the display screen parameters (brightness, contrast), the mapped virtual standing posture is that the doctor observes the operating environment of the surgeon in charge to optimize visual information. This operation is used to ensure the visibility of information. If adjusted frequently during a critical stage, it may imply insufficient visual information, which may affect the stage judgment. The system needs to re-correct the current state in combination with the results of image recognition.

[0114] Example of special situation: Example 1: The system detects that the similarity between the surgical video and the video of the "opening the aorta" step is higher than 0.9. At the same time, it detects that the doctor frequently adjusts the camera focus (indicating approaching for observation) and the standing posture data supports this behavior. Then it is confirmed that the current stage is "opening the aorta". Prompt: "The doctor is closely observing the critical area. Please continue to monitor relevant physiological data in real time (such as heart function, blood pressure, etc.)." And predict "The possibility of entering the next stage (such as perfusion) is 85%, and it is expected to switch after 45 seconds".

[0115] Example 2: If the similarity is lower than 0.9, it is detected that the extracorporeal circulation doctor uses a laser to mark the abnormal area and is accompanied by voice communication. At the same time, the standing posture data shows that the doctor is looking down to observe. Then the current surgical stage is inconsistent. The system will judge whether it is in the transition stage through methods such as analyzing the blood flow at the incision to further confirm the surgical steps.

[0116] Output: Output whether the current surgical stage is consistent. If it is consistent, no reminder is given. If it is inconsistent, it is judged whether to give an immediate reminder through surgical image recognition and the standing posture data of the extracorporeal circulation doctor (it is not allowed to disturb during the critical stage, and an immediate reminder is given when in the stage transition). Provide a prediction of the stage transition (such as "The possibility of entering the next stage is 85%, and it is expected to switch after 45 seconds"). If potential risks are recognized, output a warning of abnormal physiological data (such as "The ACT value is too low, please adjust immediately"), and an additional reminder based on the interaction behavior of the extracorporeal circulation doctor (such as "The doctor keeps approaching the display screen, please confirm the key parameters"). It can be seen that in this embodiment, the ordinary model and the large model are successively used to predict the surgical stage. When the confidence level of the output result of the ordinary image classification model is relatively high, it can achieve the rapid prediction of the current surgical stage with less computational resource occupancy. When the confidence level of the prediction result of the ordinary image classification model is insufficient, the large model is then called for prediction to ensure the prediction accuracy of the surgical stage.

[0117] In addition, based on any of the above embodiments, the method further includes the following steps: Detect whether the surgical stage of the main surgery matches the circulation stage of the extracorporeal circulation surgery at the same moment, and obtain a stage matching detection result. Wherein, the postoperative report also includes the stage matching detection result.

[0118] As described above, the extracorporeal circulation surgery assists the main surgery to be carried out together. When the main surgery enters the next surgical stage, the extracorporeal circulation surgery should promptly enter the next circulation stage to assist the main surgery. At this time, ensuring that the circulation stage of the extracorporeal circulation surgery matches the surgical stage of the main surgery is of great significance. Therefore, in postoperative analysis, it is also possible to detect whether the surgical stage of the main surgery matches the circulation stage of the extracorporeal circulation surgery at the same moment, and obtain a stage matching detection result. Specifically, a corresponding relationship between the surgical stage and the circulation stage can be preset. After obtaining the main surgery video, multiple surgical images of the main surgery at multiple moments can be obtained from the main surgery video, and the surgical stage can be recognized from the surgical images through any of the above embodiments. In addition, since the circulation stage of the extracorporeal circulation surgery can be obtained from the data recorded by the split screen device. Thus, the surgical stage of the main surgery and the circulation stage of the extracorporeal circulation surgery at the same moment can be obtained. Based on the preset corresponding relationship, it can be determined whether the surgical stage and the circulation stage match, and a stage matching detection result can be obtained, and the stage matching detection result is recorded in the output postoperative report. As an optional example, if the stage matching detection result indicates that the surgical stage and the circulation stage do not match, the stage matching detection result is highlighted in the postoperative report.

[0119] In this way, by detecting whether each surgical stage in the main surgery matches the circulation stage of the extracorporeal circulation surgery after the operation, it is beneficial to review the entire surgical process, promptly discover problems with stage mismatches, contribute to tracing the source of the problems, and improve the safety of subsequent surgeries.

[0120] In addition, on the basis of any of the above embodiments, the method may further include step 151-step 152. As an example, the surgical images at every preset time or preset number of frames in the main surgery video collected by the master device can be executed with step 151-step 152 to periodically evaluate the urgency of the surgery. As another example, it is possible to evaluate whether the area ratio of the blood in the surgical image is greater than a preset ratio threshold for the surgical images at every preset time or preset number of frames in the main surgery video. The ratio threshold is, for example, 50%. If the area ratio of the blood in the surgical image is less than the ratio threshold, it indicates that the surgical risk is relatively low. On the contrary, if the area ratio of the blood in the surgical image is greater than the ratio threshold, it indicates that the patient has more bleeding and the surgical risk is relatively high. At this time, the urgency of the surgery can be further judged by executing step 151-step 155, and a postoperative report can be given.

[0121] Step 151: Obtain the historical surgical tutorials related to the surgical stage from the master device.

[0122] Step 152: Input the surgical stage, the related historical surgical tutorials, the first interaction behavior data, and the physiological monitoring data into the trained medical multi-modal large model, so that the medical multi-modal large model uses text extraction technology to extract the numerical index conditions related to the patient's physical signs from the historical surgical tutorials, obtain the comparison result of whether the physiological monitoring data meets the corresponding numerical index conditions, and evaluate the surgical urgency based on the surgical stage, the comparison result, and the first interaction behavior, to obtain the urgency evaluation result output by the medical multi-modal large model.

[0123] Exemplarily, the master device can obtain the historical surgical tutorials related to the surgical stage, and then input the surgical stage and its corresponding historical surgical tutorials, the first interaction behavior data between the extracorporeal circulation doctor and the split screen device, and the physiological monitoring data into the trained medical multi-modal large model.

[0124] Among them, during the training process of the medical multi-modal large model, the surgical stage related to the surgery, the comparison result of the patient's physiological monitoring data and indicators, and the historical standing posture data of the extracorporeal circulation doctor can be labeled with surgical urgency (such as low, medium, high) to construct multi-modal training samples. Taking the medical multi-modal large model BioMedGPT model as an example, the prepared multi-modal training samples are used to train the model, and the model comprehensively judges the urgency of the current surgery according to the input multi-modal information. And after the surgery, the operator can score the correctness of the model's urgency determination for this time to continuously optimize the large model.

[0125] When performing Step 152, the trained medical multi-modal large model can first use text extraction technology to extract the numerical index conditions related to the patient's physical signs from the input historical surgical tutorials. Subsequently, the numerical index conditions can be further refined. Specifically, the numerical index conditions can be divided into two categories: The first category, if there are keywords "maintain", " / ", "control", "too low", or "too high" in the digital index conditions, then 30 consecutive frames of physiological monitoring data need to be comprehensively statistically analyzed as the result. The second category, if the keyword does not exist in the digital index conditions, only one frame of physiological monitoring data is required as the result.

[0126] Numerical index conditions related to patient signs include: 1) Heart rate. When the heart restarts, the expected heart rate is 60 - 80 bpm (beats per minute). If the heart rate > 110 bpm, it is judged that there may be excessive cardiac function tension or arrhythmia, and a reminder is generated and the doctor is advised to evaluate cardiac drug support. If the heart rate < 50 bpm, it indicates that there may be weak cardiac contraction or poor heart restart, and it is recommended to adjust the electrolyte level or use a cardiac pacemaker for support. 2) Blood pressure. During the extracorporeal circulation withdrawal phase, the ideal blood pressure range is 90 - 140 mmHg (systolic pressure) and 60 - 90 mmHg (diastolic pressure). If the systolic pressure < 80 mmHg, it is determined that the patient may have hypovolemia or insufficient cardiac contractile function, and it is recommended to increase fluid replacement or use drugs to strengthen cardiac contractility; if the systolic pressure > 160 mmHg, it indicates that there may be excessive vascular resistance, and it is recommended to use antihypertensive drugs or adjust the parameters of the extracorporeal circulation machine to reduce the cardiac load. 3) Oxygen saturation should be maintained at 95% - 100%. If < 90%, it is determined that there may be insufficient oxygen supply, pulmonary function disorder, or abnormal extracorporeal circulation oxygenation. Remind the doctor to check the oxygen supply device or adjust the breathing parameters, prompt the doctor to perform a blood gas analysis to confirm whether carbon dioxide excretion is normal, and adjust the ventilation support according to the results. 4) Hematocrit (HCT). During cardiac surgery, the ideal range of HCT is 30% - 45%. If HCT < 22%, it indicates that there may be excessive bleeding or dilution effect, and it is recommended to supplement red blood cell products or adjust the flow rate of extracorporeal circulation and the proportion of fluid input, and prompt the doctor to evaluate whether additional hemostatic measures are needed or adjust the surgical strategy. 5) Extracorporeal circulation flow rate is generally 2.4 - 2.8 L / min / m² (based on the patient's body surface area). If the flow rate < 2.0 L / min / m², it may lead to insufficient organ perfusion, prompt to check for pipeline blockage or pump function, recommend adjusting the flow rate setting or using bypass measures to increase the flow rate, and remind to monitor the perfusion of vital organs to ensure that severe ischemia does not occur. 6) Lactate level is generally < 2 mmol / L. If lactate > 4 mmol / L, it indicates metabolic acidosis. Analyze possible causes such as hypoperfusion, hypoxia, or infection, generate a reminder and recommend fluid supplementation, improvement of oxygen supply, or checking for potential infection sources. If lactate continues to rise, it is recommended to strengthen circulatory support or adjust the surgical process to reduce the metabolic burden.

[0127] After extracting the numerical index conditions related to the patient's physical signs, the medical multi-modal large model can determine whether the patient's physiological monitoring data meets the numerical index conditions and obtain a comparison result. The comparison result includes two cases: meeting the conditions and not meeting the conditions. Subsequently, the medical multi-modal large model can evaluate the urgency of the surgical stage based on the surgical stage, the comparison result, and the first interaction behavior. As an example, the medical multi-modal large model can first map the input first interaction behavior data to the virtual standing posture data of the extracorporeal circulation doctor using a mapping relationship (such as Table 1), and then evaluate the urgency of the surgical stage based on the virtual standing posture data, the surgical stage, and the comparison result. Among them, the urgency includes low risk, medium risk, and high risk. Low risk means that all target physiological monitoring data are within the normal range, with small fluctuations and tending to be stable. Medium risk means that 1-2 target physiological monitoring data deviate from the normal range, but there is no obvious worsening trend. High risk means that multiple physiological monitoring data deviate from the normal value, indicating potential surgical or physiological emergencies. In this way, the postoperative report can also include the urgency assessment result output by the medical multi-modal large model. Optionally, if the urgency assessment result is high risk, the urgency assessment result can be highlighted.

[0128] As an example, the medical multi-modal large model can be called using prompt engineering to evaluate the surgical urgency. The prompts of the medical multi-modal large model can be as follows: "Role positioning: You are an intelligent surgical risk summary system, responsible for comprehensively reviewing and analyzing the entire surgical process after the surgery and generating a detailed surgical risk summary report. The report needs to cover all stages of the surgery (such as perfusion, rewarming, root venting, etc.), highlight key risk events and emergencies for subsequent risk management and improvement.

[0129] Task description: Please conduct a comprehensive analysis based on the data of the entire surgical process. The data includes but is not limited to: physiological monitoring data (such as blood pressure, heart rate, HCT value, etc.); surgical stage information (obtained through a stage recognition model); interaction behavior data between the extracorporeal circulation doctor and the split-screen device (such as adjusting the camera, marking, undoing, exiting the interface, etc.), historical surgical video data (including bone key point detection, FAISS+ time index matching data).

[0130] Please divide the entire surgical process into different stages (such as perfusion, rewarming, root venting, etc.), describe the risk situations of each stage in detail in the report, and particularly highlight those parts with higher risks and emergencies. The report should include the following content: Risk level Low risk (1-3 points): Adjust the camera magnification to optimize the viewing angle.

[0131] Medium risk (4 - 6 points): Repeatedly adjust the focus and turn on the microphone to communicate with the surgeon.

[0132] High risk (7 - 10 points): Laser mark the key areas and communicate verbally, which may involve emergency adjustments.

[0133] Interactive behaviors of extracorporeal circulation doctors and risk assessment logic: 1) If the interactive behavior is to adjust the camera focus (focus + / focus -), the mapped virtual stance is that the doctor approaches the surgeon and looks down to observe key details (such as perfusion tubing, pump speed indication). If adjusted frequently, it may indicate abnormal perfusion.

[0134] 2) If the interactive behavior is to adjust the camera magnification (magnification + / magnification -), the mapped virtual stance is that the doctor stands farther away to observe the overall situation. For example, it may be assessing the overall perfusion status. If it occurs during the key stage switch, it may mean an increased risk (such as the start of hypothermic circulation).

[0135] 3) If the interactive behavior is to turn on / off the microphone, the mapped virtual stance is that the extracorporeal circulation doctor communicates frequently with the surgeon, possibly accompanied by approaching movements. If switched on and off multiple times in a short period, it may mean an emergency situation, such as the need to manually adjust the flow rate or oxygenation parameters.

[0136] 4) If the interactive behavior is to use the laser to mark the screen, the mapped virtual stance is that the doctor points at the surgical area with a finger to mark the abnormal point. If followed by verbal communication immediately after marking, it may indicate a risk at that point (such as a twisted perfusion tubing or the appearance of bubbles).

[0137] 5) If the interactive behavior is to manually scribble and mark or cancel the scribble, the mapped virtual stance is that the extracorporeal circulation doctor repeatedly assesses the risk area through gestures and modified prompts. If modified or cancelled multiple times at this time, it may mean continuous risk assessment (such as the low blood pressure alarm not being lifted), and further analysis is required in combination with physiological monitoring data.

[0138] 6) If the interactive behavior is to clear all scribbles, the mapped virtual stance is that the extracorporeal circulation doctor no longer pays attention to the previously marked risk points, which may mean a system false alarm or the doctor believes the risk has been lifted.

[0139] 7) If the interactive behavior is to exit the surgical live broadcast interface, the mapped virtual stance is that the doctor leaves the side of the surgeon, indicating that the current stage has ended or the risk has decreased. For example, after perfusion is completed and entering the shutdown stage; however, if exiting during a key stage, the system needs to confirm whether to conduct a handover of responsibilities.

[0140] 8) If the interaction behavior is to adjust the display screen parameters (brightness, contrast), the mapped virtual standing posture is for the doctor to observe the operating environment of the surgeon, optimizing visual information. This operation is used to ensure information visibility. If adjusted frequently during critical stages, it may imply insufficient visual information and may affect stage judgment.

[0141] 9) If the interaction behavior is to exit the surgical live broadcast interface, the mapped virtual standing posture is for the extracorporeal circulation doctor to leave, indicating a reduced surgical risk at this time. If exiting during a critical stage, it may be necessary to confirm whether responsibilities are handed over.

[0142] Overall risk assessment Overall risk level (such as low risk, medium risk, high risk).

[0143] Overview of the risk trend during the entire surgical process.

[0144] Stage risk analysis The surgical process is divided into multiple stages (such as perfusion, rewarming, root air exhaustion, etc.), and the physiological index performance, doctor's interaction behavior, and potential risk points in each stage are described separately. Abnormal indicators (such as HCT below the safety threshold, abnormal heart rate, etc.) and key doctor operations (such as frequently adjusting the camera, laser marking abnormal areas, etc.) that occur in each stage are described in detail, and emergency risk events are highlighted.

[0145] Key risk event record List important risk events during the surgical process, including the time of occurrence, the abnormal physiological data triggered, and doctor operations. For emergencies, they should be marked in a prominent way (for example: using red font or other visual cues).

[0146] Improvement suggestions Based on the risk events and data analysis of each stage, corresponding risk management and surgical improvement suggestions are put forward. Provide targeted measures to prevent similar events from happening again.

[0147] Output example:

Surgical Risk Summary Report

[0148] 2. Stage risk analysis - **Perfusion stage**: - Physiological monitoring data shows that the HCT value was once below the safety threshold.

[0149] - In the doctor's operation record, the camera focus was adjusted multiple times and the abnormal areas were marked with laser.

[0150] - **Emergency highlight**: This stage has a relatively high risk and requires key attention.

[0151] - **Rewarming stage**: - The heart rate increased slightly, but did not exceed the dangerous threshold.

[0152] - The doctor's interaction behavior was normal and the risk was relatively stable.

[0153] - **Root exhaust stage**: - Short-term abnormal fluctuations were detected, accompanied by the doctor's cancellation and re-marking operations.

[0154] - **Emergency highlight**: It is recommended to conduct a detailed review of the physiological data and equipment status at this stage.

[0155] 3. Highlight key risk event records - **Event 1**: The HCT decreased sharply at a certain moment during the perfusion stage - Abnormal index: HCT is lower than the safety threshold - Doctor's operation: Frequently adjust the camera and mark with laser - Countermeasures: Adjust the drug dosage and equipment parameters - **Status: Emergency (controlled)** - **Event 2**: Short-term abnormal heart rate was detected during the root exhaust stage - Abnormal index: Abnormal heart rate fluctuation - Doctor's operation: Cancel and re-mark - **Status: Emergency (further monitoring required)** 4. Improvement suggestions - Strengthen the real-time recording and review of physiological data during each stage of the operation.

[0156] - For high-risk stages, it is recommended to formulate more strict monitoring and emergency measures. In addition, based on any of the above embodiments, the postoperative report includes the interference detection results that interfere with the output of the large model for interference detection, the stage matching detection results between the surgical stage of the main operation and the circulation stage of the extracorporeal circulation operation at the same moment, and the emergency degree evaluation results output by the medical multi-modal large model. As an example, the postoperative report can be generated and output based on prompt engineering and the large model. The following is an exemplary example of the prompt for assisting in generating and outputting the postoperative report: "You are an intelligent surgical stage summary report system, responsible for reviewing and comprehensively analyzing the transitions between various stages during the entire surgical process after the operation. You need to generate a detailed summary report by combining physiological data, video images, historical trends, and the doctor's interaction behavior data, and highlight the key risks and risk warning content.

[0157] Task description: Please generate a detailed surgical stage summary report based on the data and model prediction results of the entire surgical process. The report should include: 1) Overall assessment: A summary of the overall stage transitions during the operation, including the overall risk level and deviation analysis of the transition timing. 2) Detailed stage analysis: Divide the surgical process into multiple stages (e.g., perfusion, rewarming, root venting, etc.), and summarize the actual transition timing, key physiological data, and doctor's interaction behavior for each stage. For the risk signals (such as abnormal heart rate, blood flow fluctuations, frequent camera adjustments, abnormal laser markings, etc.) that occur in each stage, describe them in detail and highlight the risk warning content. 3) Highlighting of risk warnings: For the detected risk events, clearly mark the risk warning information (e.g., using a prominent red font or other visual cues), explain the risk reasons and the corresponding safety hazards. 4) Improvement suggestions: Based on the data analysis of each stage, provide targeted improvement suggestions for risk management and equipment regulation to help optimize the stage transition control in subsequent surgeries.

[0158] Input parameters: Video data of the entire surgical process (for stage identification and analysis); Physiological monitoring data (such as blood pressure, heart rate, extracorporeal circulation flow rate, blood oxygen, etc.); Data of each stage (e.g., perfusion, rewarming, root venting, etc.); Interaction behavior data of the extracorporeal circulation doctor with the split-screen device (such as camera adjustment, laser marking, voice interaction, exiting the interface, etc.); Historical surgical data and model prediction results (such as the results of LSTM / GRU model analyzing the transition timing).

[0159] Reference logic: Use a data analysis model (such as LSTM / GRU) to review the predicted transition timing and the actual transition timing of each stage, and record the deviation between the two. Analyze the physiological monitoring data and the doctor's interaction behavior of each stage, locate the key risk events, and highlight the risk warnings (e.g., "Risk warning: Abnormal increase in heart rate!"). Summarize the risk indicators of each stage, evaluate the overall stage transition risk, and put forward targeted improvement suggestions Example of special situation: Example logic: Interaction behavior of the extracorporeal circulation doctor and stage prediction The extracorporeal circulation doctor's operation on the split screen: Adjust the camera focus (focus + / focus -) Possible stage transition signal: Pay attention to the key surgical site. The LSTM model predicts the transition timing: It may be 45 seconds after approaching the transition stage. Reminder method: The system gives a "The doctor is closely watching" 30 seconds in advance Cardiopulmonary bypass doctor's split-screen operation: Frequently use laser marking. Possible stage transition signal: Emphasize that abnormalities may occur in a certain area. LSTM model predicts the transition timing: After 30 seconds. Reminder method: High-risk alarm "Please check the marked area". Cardiopulmonary bypass doctor's split-screen operation: Turn on / off the microphone. Possible stage transition signal: May discuss with the surgeon. LSTM model predicts the transition timing: After 45 seconds. Reminder method: Audio notification "Please confirm the transition timing". Cardiopulmonary bypass doctor's split-screen operation: Exit the monitoring interface. Possible stage transition signal: May indicate that the current stage has stabilized. LSTM model predicts the transition timing: Immediately. Reminder method: Green prompt "Stage transition completed". Output example:

Surgical stage summary report

[0160] 2. Detailed stage analysis (highlight risk warnings) - Perfusion stage: - Predicted transition timing: After 45 seconds; Actual transition timing: After 50 seconds.

[0161] - Key physiological data: The HCT value decreased once; Doctor's operation: Frequent adjustment of the camera.

[0162] - **Risk warning: Abnormal increase in heart rate, high risk!** (Highlight risk warning) - Rewarming stage: - Predicted transition timing: After 60 seconds; Actual transition timing: After 58 seconds.

[0163] - Key physiological data: Blood flow is stable but the flow rate fluctuates slightly; Doctor's operation: Laser marking of abnormal areas.

[0164] - **Risk warning: Incisional blood flow exceeds expectations, please pay attention to monitoring!** (Highlight risk warning) - Root air exhaust stage: - Predicted transition timing: Immediately; Actual transition timing: Immediately.

[0165] - Key physiological data: All indicators are stable; Doctor's operation: No abnormal operation.

[0166] - Risk situation: Normal, no risk warning.

[0167] 3. Improvement suggestions - Strengthen the linkage monitoring of physiological data and doctors' interaction behaviors at each stage to narrow the deviation between prediction and actual transition timing.

[0168] - For high-risk warning stages (such as perfusion and rewarming stages), it is recommended to strengthen equipment calibration and real-time data review to ensure timely identification and handling of risk events.

[0169] - It is recommended to regularly review and summarize reports to optimize the surgical process and risk management strategies. Based on any of the above embodiments, the present application further provides a postoperative report output system for off-pump cardiologist split-screen surgery. The system includes a split-screen device, a main control device, and a server that are communicatively connected for real-time monitoring of off-pump cardiologist surgery; the split-screen device and the main control device are used to monitor each surgical stage of the off-pump cardiologist surgery; the split-screen device and the main control device are respectively arranged in different surgical areas; wherein, The split-screen device for real-time monitoring of off-pump cardiologist surgery is used to monitor each of the surgical stages and collect physiological monitoring data of the patient; The main control device is used to monitor each of the surgical stages and collect the main surgical video, and the main surgical video includes surgical images; The server is used to identify the surgical stage of the main surgery based on the surgical images; and is used to input the surgical images, the surgical stages, the first interaction behavior data, the second interaction behavior data, and the physiological monitoring data into a trained interference detection large model, so that the interference detection large model maps the first interaction behavior data to the virtual standing posture data of the off-pump cardiologist based on a preset mapping relationship, and when predicting that the surgeon is in a focused surgical state based on the virtual standing posture data, the surgical images, the surgical stages, and the physiological monitoring data, detect whether the off-pump cardiologist interferes with the surgeon based on the second interaction behavior data; the training data of the interference detection large model includes historical surgical videos with surgical stage labels and surgical status labels, historical physiological monitoring data, and historical standing posture data of the off-pump cardiologist; and is used to output a postoperative report, and the postoperative report includes the interference detection result output by the interference detection large model.

[0170] The implementation processes of the functions and roles of each module in the above device are specifically detailed in the implementation processes of the corresponding steps in the above method, and will not be repeated here.

[0171] Based on the method for outputting postoperative reports of off - pump coronary artery bypass surgery described in any of the above embodiments, the present application also provides a computer program product, which includes one or more computer programs or instructions. The computer programs or instructions can be stored in a computer - readable storage medium or transmitted from one computer - readable storage medium to another. When the computer program is executed by a processor, it implements the method for outputting postoperative reports of off - pump coronary artery bypass surgery described in any of the above embodiments.

[0172] Based on the method for outputting postoperative reports of off - pump coronary artery bypass surgery described in any of the above embodiments, the present application also provides Figure 2 a schematic structural diagram of an electronic device as shown in Figure 2 . At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non - volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non - volatile memory into the memory and then runs it to implement the method for outputting postoperative reports of off - pump coronary artery bypass surgery described in any of the above embodiments.

[0173] The present application also provides a computer storage medium storing a computer program, which can be used to execute the method for outputting postoperative reports of off - pump coronary artery bypass surgery described in any of the above embodiments when executed by a processor.

[0174] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware - based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0175] In addition, in each embodiment of the present application, the functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0176] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0177] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0178] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0179] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

Claims

1. A method for outputting a postoperative report of an extracorporeal circulation doctor's split-screen surgery, characterized in that, Cardiopulmonary bypass surgery is used to assist the main surgery; during each surgical stage of the main surgery, monitoring is carried out through a split-screen device for real-time surgical monitoring by a cardiopulmonary bypass doctor and a master control device; the method includes: Identifying the surgical stage of the main surgery based on the surgical images collected by the master control device; Obtaining first interaction behavior data of the cardiopulmonary bypass doctor and the split-screen device, second interaction behavior data of the cardiopulmonary bypass doctor and the surgeon, and physiological monitoring data of the patient; Inputting the surgical images, the surgical stage, the first interaction behavior data, the second interaction behavior data, and the physiological monitoring data into a trained interference detection large model, so that the interference detection large model maps the first interaction behavior data to the virtual standing posture data of the cardiopulmonary bypass doctor based on a preset mapping relationship, and when predicting that the surgeon is in a surgical concentration state based on the virtual standing posture data, the surgical images, the surgical stage, and the physiological monitoring data, detecting whether the cardiopulmonary bypass doctor interferes with the surgeon based on the second interaction behavior data; the training data of the interference detection large model includes historical surgical videos with surgical stage labels and surgical status labels, historical physiological monitoring data, and historical standing posture data of the cardiopulmonary bypass doctor; Outputting a postoperative report, where the postoperative report includes the interference detection result output by the interference detection large model.

2. The method according to claim 1, wherein Before identifying the surgical stage of the main surgery based on the surgical images collected by the master control device, the method further includes: Step S1: Obtaining the screen collected by the master control device; Step S2: If it is determined by an image classification algorithm that the screen contains image content of a surgical incision, determining the screen as a candidate image; Step S3: Using an image segmentation algorithm to segment the entity name, image position, and segmentation credibility of an object from the candidate image; the objects include incisions, blood, doctor's hands, patient organs, and surgical tools; Step S4.1: In the case where the ratio of the first image area of the blood in the candidate image to the second image area of the incision in the candidate image is less than a preset ratio threshold, if the first segmentation confidence of the patient organ and the second segmentation confidence of the surgical tool in the candidate image are greater than a first confidence threshold, determining the candidate image as the surgical image; if the first segmentation confidence or the second segmentation confidence is less than the first confidence threshold, obtaining the next candidate image and returning to execute Step S3 until a surgical image is determined; Step S4.2: In the case where the ratio of the first image area to the second image area is greater than the ratio threshold, obtaining the next candidate image at a preset interval time and returning to execute Step S3 until a surgical image is determined.

3. The method according to claim 2, characterized in that Identifying the surgical stage of the main surgery based on the surgical images collected by the master control device includes: Determining the association result between the target instruction in the surgical instruction record table and the surgical image, and the instruction value of the target instruction; Input the association result, the instruction value, the entity name of the object in the surgical image and the segmentation credibility, the surgical image, and the physiological monitoring data into a random forest model to obtain the first surgical stage output by the random forest model; Input the embedding vector of the surgical image into a ResNet model to obtain a set of a preset number of video frames determined from the surgical tutorial video in the order of decreasing similarity between the embedding vector of the surgical image and the embedding vectors of the pre-stored surgical tutorial videos by the ResNet model; wherein, the surgical tutorial video carries surgical action labels, and the ResNet model is trained using the surgical tutorial video; Determine the target video frame with the highest similarity and the similarity of the target video frame from the set of video frames, and determine the second surgical stage based on the surgical action label carried by the target video frame; If the similarity of the target video frame exceeds a preset similarity threshold, determine that the second surgical stage is the current surgical stage; If the similarity of the target video frame does not exceed the similarity threshold, determine that the first surgical stage is the current surgical stage.

4. The method according to any one of claims 1 to 3, characterized in that, The identifying the surgical stage of the main surgery based on the surgical image collected by the master device includes: Obtain the initial surgical stage output after classifying the surgical image by an image classification model; the surgical stage identification model includes a random forest model and / or a ResNet model; wherein, the confidence level of the initial surgical stage is less than a preset second confidence threshold; Input the initial surgical stage, the surgical image, the first interaction behavior data, and the physiological monitoring data into a trained large surgical stage identification model, so that the large surgical stage identification model determines whether the initial surgical stage is credible based on the surgical image, the physiological monitoring data, and the first interaction behavior data, and determines the surgical stage based on the surgical image, the physiological monitoring data, and the first interaction behavior data when the initial surgical stage is not credible.

5. The method according to claim 1, wherein The extracorporeal circulation surgery includes multiple circulation stages, the main surgery includes multiple surgical stages, and each circulation stage is matched with one or more of the surgical stages; the method further includes: Detect whether the surgical stage of the main surgery matches the circulation stage of the extracorporeal circulation surgery at the same moment to obtain a stage matching detection result; The postoperative report further includes the stage matching detection result.

6. The method according to claim 1, wherein The method further includes: Obtain the historical surgical tutorial related to the surgical stage from the master device; Input the surgical stage, the related historical surgical tutorial, the first interaction behavior data, and the physiological monitoring data into a trained medical multi-modal large model, so that the medical multi-modal large model uses text extraction technology to extract numerical index conditions related to the patient's physical signs from the historical surgical tutorial, obtain a comparison result of whether the physiological monitoring data meets the corresponding numerical index conditions, and evaluate the surgical urgency based on the surgical stage, the comparison result, and the first interaction behavior, to obtain the urgency evaluation result output by the medical multi-modal large model; Wherein, the postoperative report further includes the urgency evaluation result.

7. An electronic device, characterized in that, The electronic device includes: A processor; A memory for storing instructions executable by the processor; Wherein, when the processor calls the executable instructions, it implements the operations of the method according to any one of claims 1-6.

8. A postoperative report output system for extracorporeal circulation doctors' split-screen surgery, characterized in that, The system includes a split-screen device for real-time surgical monitoring by extracorporeal circulation doctors, a main control device, and a server that are communicatively connected; the extracorporeal circulation surgery is used to assist the main surgery; wherein, The split-screen device for real-time surgical monitoring by extracorporeal circulation doctors is used to monitor each surgical stage and collect the physiological monitoring data of the patient; The main control device is used to monitor each surgical stage and collect surgical images; The server is used to identify the surgical stage of the main surgery based on the surgical images; And is used to obtain the first interaction behavior data between the extracorporeal circulation doctor and the split-screen device, the second interaction behavior data between the extracorporeal circulation doctor and the surgeon, and the physiological monitoring data; And is used to input the surgical images, the surgical stage, the first interaction behavior data, the second interaction behavior data, and the physiological monitoring data into a trained interference detection large model, so that the interference detection large model maps the first interaction behavior data to the virtual standing posture data of the extracorporeal circulation doctor based on a preset mapping relationship, and when predicting that the surgeon is in a surgical focused state based on the virtual standing posture data, the surgical images, the surgical stage, and the physiological monitoring data, detect whether the extracorporeal circulation doctor interferes with the surgeon based on the second interaction behavior data; the training data of the interference detection large model includes historical surgical videos with surgical stage labels, historical physiological monitoring data, historical standing posture data of extracorporeal circulation doctors, and corresponding surgical state labels; Output a postoperative report, where the postoperative report includes the interference detection result output by the interference detection large model.

9. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, A computer instruction is stored thereon, and when the computer instruction is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images