Operation report generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of medical data processing, and discloses an operation report generation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a video image collected in an operation process; for a first target frame image in the video image, performing surgical instrument identification by using an image identification neural network model; if the target surgical instrument is identified and the first target frame image is the first frame image of the identified target surgical instrument, determining a target surgical event corresponding to the target surgical instrument according to a corresponding relationship between the surgical instrument and the surgical event, a target moment corresponding to the first target frame image being a start moment of the target surgical event; adding an event label at a target position on the target time axis to form an operation video report; wherein the target time axis is a time axis corresponding to the video image, and the target position at least comprises a position corresponding to the target moment. The generation efficiency of the operation report can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a method, device, equipment and storage medium for generating an operation report. Background Art

[0002] In the related technology, medical staff need to record each step of the operation one by one, including the start time, end time and instruments used, etc., according to the actual progress of the operation, in order to form a surgical report. This method is not only time-consuming and labor-intensive, but also prone to recording errors or missing important information, resulting in low efficiency. Summary of the invention

[0003] In view of this, the present invention provides a method, device, equipment and storage medium for generating a surgical report to solve the problem of low efficiency in manually recording surgical information to form a surgical report.

[0004] In a first aspect, the present invention provides a method for generating a surgical report, the method comprising:

[0005] Acquire video images collected during surgery;

[0006] For the first target frame image in the video image, using an image recognition neural network model to identify the surgical instrument;

[0007] If a target surgical instrument is identified, and the first target frame image is the first frame image in which the target surgical instrument is identified, a target surgical event corresponding to the target surgical instrument is determined according to a correspondence between the surgical instrument and the surgical event, and a target time corresponding to the first target frame image is a start time of the target surgical event;

[0008] An event annotation is added to a target position on a target time axis to form a surgical video report; wherein the target time axis is a time axis corresponding to the video image, and the target position includes at least a position corresponding to the target moment.

[0009] In an optional embodiment, the event annotation includes an event tag, and different event tags correspond to different surgical events; and / or,

[0010] The surgical events correspond to different surgical steps during the surgical procedure.

[0011] In an optional implementation, after determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes:

[0012] Performing surgical instrument recognition on a second target frame image following the first target frame image in chronological order;

[0013] If the target surgical instrument is determined to have disappeared based on N consecutive frames of the second target frame images, the target surgical event corresponding to the target surgical instrument is determined to have ended, and the moment corresponding to the second target frame image when the target surgical instrument is not recognized for the first time is the end moment of the target surgical event; the end moment is used to determine the event label or update the event label.

[0014] In an optional implementation, the performing surgical instrument recognition on a second target frame image following the first target frame image in chronological order includes:

[0015] Using a preset sliding window, acquiring N frames of second target frame images after the first target frame image, where N is the width of the sliding window;

[0016] If among the N frames of the second target frame images, there are M frames of the second target frame images in which the target surgical instrument is not detected, it is determined that the target surgical instrument disappears; M is less than or equal to N and greater than a preset frame number threshold.

[0017] In an optional implementation, if it is determined based on N consecutive frames of the second target frame image that the target surgical instrument disappears, then determining that the target surgical event corresponding to the target surgical instrument ends includes:

[0018] If the target surgical instrument is determined to have disappeared based on N consecutive frames of the second target frame images, and the interval between the moment corresponding to the second target frame image when the target surgical instrument is not recognized for the first time and the start moment is greater than a first preset threshold, then it is determined that the target surgical event corresponding to the target surgical instrument has ended.

[0019] In an optional implementation, after determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes:

[0020] Performing surgical instrument recognition on a third target frame image following the first target frame image in chronological order;

[0021] If it is detected based on the third target frame image that the target surgical instrument disappears and then reappears, obtaining a first duration of each appearance of the target surgical instrument;

[0022] Calculating variances of a plurality of said first durations;

[0023] If the variance is less than a second preset threshold, the moment when the target surgical instrument disappears for the last time is determined as the end moment of the target surgical event; the end moment is used to determine the event label or update the event label.

[0024] In an optional implementation, after adding the event annotation at the target position on the target time axis, the method further includes:

[0025] Receiving a target operation marked for the event;

[0026] In response to the target operation, displaying an event description interface;

[0027] receiving a target input for the event description interface;

[0028] In response to the target input, displaying event description information input by the user;

[0029] A text report of the operation is generated based on the event description information.

[0030] In a second aspect, the present invention provides a surgical report generating device, the device comprising:

[0031] A video image acquisition module, used to acquire video images collected during the operation;

[0032] A first recognition module, used for identifying surgical instruments using an image recognition neural network model for a first target frame image in the video image;

[0033] a surgery time determination module, configured to determine a target surgery event corresponding to the target surgery instrument according to a correspondence between the surgery instrument and the surgery event if a target surgery instrument is identified and the first target frame image is the first frame image in which the target surgery instrument is identified, and the target time corresponding to the first target frame image is the start time of the target surgery event;

[0034] The first report generation module is used to add event annotations to the target position on the target time axis to form a surgical video report; wherein the target time axis is the time axis corresponding to the video image, and the target position at least includes the position corresponding to the target moment.

[0035] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the surgical report generation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the surgical report generating method of the first aspect or any corresponding embodiment thereof.

[0037] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the surgical report generating method of the first aspect or any corresponding embodiment thereof.

[0038] The surgical report generation method, device, equipment and storage medium provided by the embodiments of the present invention can automatically identify key surgical instruments based on the video images collected in real time during the operation to identify the surgical events corresponding to the key surgical instruments, thereby realizing automatic recognition of surgical events, and automatically adding event annotations of surgical events, thereby realizing automatic editing of surgical video reports. Compared with the method of manually recording surgical events or manually identifying surgical videos to add event annotations, the embodiments of the present invention can greatly improve the efficiency of surgical report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flow chart of a method for generating a surgical report according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of a display interface of a surgical video according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a text report according to an embodiment of the present invention;

[0043] Figure 4 is a structural block diagram of a surgical report generating device according to an embodiment of the present invention;

[0044] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] According to an embodiment of the present invention, an embodiment of a surgical report generating method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] In this embodiment, a method for generating a surgical report is provided, which can be used in various computer devices, such as medical devices. Figure 1 is a flow chart of a method for generating a surgical report according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0048] Step S101, obtaining video images collected during the operation.

[0049] Specifically, the video images collected during the operation are video images collected for the operation area. If it is a microsurgery, then the video images are images under the microscope field of view.

[0050] Step S102: For the first target frame image in the video image, surgical instrument recognition is performed using an image recognition neural network model.

[0051] Specifically, the image recognition neural network model is a neural network model based on an improved convolutional attention mechanism. For example, the neural network model can be YOLOv8, which is improved by adding a global attention mechanism (GAM, GlobalAttention Mechanism) and a convolutional block attention module (CBAM), and the performance of the deep neural network model is improved by reducing information redundancy and amplifying global interaction representation. The image recognition neural network model can identify a variety of surgical instruments required during the operation. In an embodiment of the present invention, the image recognition neural network model only needs to identify some key surgical devices during the operation. These key surgical instruments can be used to perform specific surgical operations. Therefore, the appearance of key surgical instruments in the surgical field often means a specific surgical event. The surgical event is the surgical step in the surgical process. Taking ophthalmic cataract surgery as an example, it includes 6 steps such as incision, capsule tearing, chopping nucleus, phacoemulsification, cortical suction, and lens implantation. These 6 steps are 6 surgical events. Each surgical event requires a specific surgical instrument to perform. The correspondence between surgical events and surgical instruments of different operations can be determined in advance by professionals.

[0052] To improve the accuracy of surgical instrument recognition, different types of surgeries can train corresponding image recognition neural network models. If two surgeries use the same key surgical instruments, they can be classified as the same type of surgeries and use the same image recognition neural network model.

[0053] Step S103: If the target surgical instrument is identified and the first target frame image is the first frame image in which the target surgical instrument is identified, the target surgical event corresponding to the target surgical instrument is determined based on the correspondence between the surgical instrument and the surgical event, and the target time corresponding to the first target frame image is the start time of the target surgical event.

[0054] The correspondence between surgical instruments and surgical events is determined in advance.

[0055] In the embodiment of the present invention, surgical instrument recognition can be performed in real time for each frame of video image collected during the operation. The moment when the target surgical instrument is first recognized is the start moment of the surgical event corresponding to the target surgical instrument.

[0056] Step S104, adding an event annotation at a target position on a target time axis to form a surgical video report, wherein the target time axis is a time axis corresponding to the video image, and the target position at least includes a position corresponding to the target moment.

[0057] If the end time of the target surgical event can be determined, the target position may include the start time, the end time and the position corresponding to the intermediate time period. The content and method of event annotation can be determined according to actual needs.

[0058] In addition, the above-mentioned surgical video report can also be used as a surgical teaching video or error correction video.

[0059] The surgical report generation method provided by the embodiment of the present invention can automatically identify key surgical instruments based on the video images collected in real time during the operation to identify the surgical events corresponding to the key surgical instruments, thereby realizing automatic recognition of surgical events, and automatically adding event annotations of surgical events, thereby realizing automatic editing of surgical video reports. Compared with the method of manually recording surgical events or manually identifying surgical videos to add event annotations, the embodiment of the present invention can greatly improve the efficiency of generating surgical reports.

[0060] In order to facilitate medical staff to view surgical videos, for example, to check whether the shooting angle is appropriate during surgery, the embodiments of the present invention can display the captured video images on a display screen. Figure 2The display interface shown can display the captured video image in the first display area 101. In addition, the target timeline can be displayed in the second display area 102, and the added event annotations can be displayed in the third display area 103, so that the medical staff can check whether the added event annotations and the added positions are correct. In addition, the event screenshot can be displayed in the fourth display area 104; and the button for the medical staff or other users to manually add event annotations can be displayed in the fifth display area 105.

[0061] In some specific implementations, the event annotation includes an event tag, and different event tags correspond to different surgical events.

[0062] The event mark may include one or more of a color block, a graphic, and text. The event mark in text form may be an event name or an event code, etc.

[0063] In some optional specific implementations, after determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes:

[0064] Performing surgical instrument recognition on a second target frame image following the first target frame image in chronological order;

[0065] If it is determined that the target surgical instrument disappears based on N consecutive frames of the second target frame image, the target surgical event corresponding to the target surgical instrument is determined to be over, and the moment corresponding to the first time the second target frame image fails to identify the target surgical instrument is the end moment of the target surgical event; N is an integer greater than or equal to 1. The end moment is used to determine the event annotation or update the event annotation. For example, the duration of the target surgical event is determined based on the start moment and end moment of the target surgical event; the event annotation is determined or updated based on the duration. The event annotation may include the duration of the target surgical event.

[0066] In some optional specific implementations, the performing surgical instrument recognition on the second target frame image after the first target frame image in chronological order includes:

[0067] Using a preset sliding window, acquiring N frames of second target frame images after the first target frame image, where N is the width of the sliding window;

[0068] If the target surgical instrument is not detected in M ​​frames of the second target frame images among N frames of the second target frame images, it is determined that the target surgical instrument disappears; M is less than or equal to N and greater than a preset frame number threshold. The preset frame number threshold may be, for example, greater than N / 2 or 2N / 3.

[0069] In the embodiment of the present invention, surgical instrument recognition can be performed in real time for each frame of the collected video image during the operation. After the target surgical instrument is recognized for the first time, the surgical instrument is continuously recognized frame by frame in chronological order. If the target surgical instrument is not recognized in N consecutive frames (i.e., M is equal to N) of video images, the surgical event marked by the target surgical instrument is determined to be over. Alternatively, if the target surgical instrument is not recognized in M ​​(M is less than N) frames of video images among N consecutive frames of video images, the surgical event marked by the target surgical instrument is determined to be over.

[0070] In other implementations, the duration of the target surgical event may not be calculated, and the event annotation may be determined or updated directly based on the end time. For example, on the timeline corresponding to the video image, color blocks or other marks may be added to the start time, end time, and middle time period of the target surgical event.

[0071] In some other optional implementations, if it is determined based on the continuous N frames of the second target frame image that the target surgical instrument disappears, then it is determined that the target surgical event corresponding to the target surgical instrument ends, including:

[0072] If the target surgical instrument is determined to have disappeared based on N consecutive frames of the second target frame images, and the interval between the time corresponding to the second target frame image in which the target surgical instrument is not recognized for the first time and the start time is greater than a first preset threshold, then the target surgical event corresponding to the target surgical instrument is determined to have ended. The first preset threshold is determined based on an empirical value of the duration of the target surgical event, or is determined based on a standard value of the duration of the target surgical event.

[0073] In an embodiment of the present invention, in order to avoid the interruption of some surgical events during the operation or the missed recognition of the model leading to the incorrect recognition of the continuous process of the surgical event, after the target surgical event is ended based on image recognition, it is also necessary to determine whether the duration of the target surgical event is greater than a certain threshold. If this condition is not met, it means that the target surgical event should not have ended yet.

[0074] In some optional specific implementations, after determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes:

[0075] Performing surgical instrument recognition on a third target frame image following the first target frame image in chronological order;

[0076] If it is detected based on the third target frame image that the target surgical instrument disappears and then reappears, obtaining a first duration of each appearance of the target surgical instrument;

[0077] Calculating variances of a plurality of said first durations;

[0078] If the variance is less than a second preset threshold, the moment when the target surgical instrument disappears for the last time is determined as the end moment of the target surgical event; the end moment is used to determine the event label or update the event label.

[0079] The embodiment of the present invention provides another method for determining the end of a target surgical event. The variance between the first durations of each appearance of the target surgical instrument is small, indicating that these appearances are not misdetected surgical instruments or wrong surgical instruments, but normal surgical operation processes, and multiple appearances belong to the same surgical event corresponding to the surgical instrument.

[0080] In other embodiments, the variance of the time interval between each appearance of the target surgical instrument may also be counted. If the variance is less than a third preset threshold, the moment when the target surgical instrument disappears for the last time is determined as the end time of the target surgical event. If the variance is greater than or equal to the third preset threshold, the time interval between the last appearance and the previous appearance is removed, and the variance is calculated again. If the variance is less than the third preset threshold, the moment when the target surgical instrument disappears for the second to last time is determined as the end time of the target surgical event, and so on.

[0081] In some further optional specific implementations, after determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes:

[0082] Using a preset sliding window, acquiring M fourth target frame images after the first target frame image, where M is the width of the sliding window; M is an integer greater than or equal to 1;

[0083] If, among the M frames of the fourth target frame images, one frame of the fourth target frame image detects the target surgical instrument, it is determined that the target surgical event has not yet ended, and the M frames of the fourth target frame images in the next sliding window are obtained, and the target surgical instrument detection is performed until the target surgical instrument is not detected in the M frames of the fourth target frame images in a sliding window, and it is determined that the target surgical event is ended.

[0084] In the embodiment of the present invention, the disappearance of the target surgical instrument is detected by using a sliding window to determine the end of the target surgical event.

[0085] In some specific implementations, after adding the event annotation at the target position on the target time axis, the method further includes:

[0086] Receiving a target operation marked for the event;

[0087] In response to the target operation, displaying an event description interface;

[0088] receiving a target input for the event description interface;

[0089] In response to the target input, displaying event description information input by the user;

[0090] Generate a text report of the operation based on the event description information. The step of generating the text report can be performed in response to a text report generation operation of the user. Specifically, the event name and event description information are added to a text report template to generate the text report.

[0091] In the embodiment of the present invention, the user can add event description information to the event annotation, and the event description information can be used as part of the video report of the operation. The event description information can also be used to generate a text report of the operation (such as Figure 3 shown).

[0092] In addition, users can also review event annotations and then modify them, such as modifying the position of the annotation, modifying the marked event, etc.

[0093] In this embodiment, a surgical report generating device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0094] This embodiment provides a surgical report generating device, such as Figure 4 As shown, including:

[0095] The video image acquisition module 401 is used to acquire video images collected during the operation;

[0096] A first recognition module 402 is used to recognize surgical instruments using an image recognition neural network model for a first target frame image in the video image;

[0097] The operation time determination module 403 is used to determine the target operation event corresponding to the target operation instrument according to the correspondence between the operation instrument and the operation event if the target operation instrument is identified and the first target frame image is the first frame image of the target operation instrument identified, and the target time corresponding to the first target frame image is the start time of the target operation event;

[0098] The first report generating module 404 is used to add event annotations at the target position on the target time axis to form a surgical video report; wherein the target time axis is the time axis corresponding to the video image, and the target position at least includes the position corresponding to the target moment.

[0099] In some optional embodiments, the event annotation includes an event marker, and different event markers correspond to different surgical events; and / or,

[0100] The surgical events correspond to different surgical steps during the surgical procedure.

[0101] In some optional embodiments, the surgical report generating device further includes:

[0102] A second recognition module, configured to perform surgical instrument recognition on a second target frame image following the first target frame image in chronological order;

[0103] The first determination module is used to determine that the target surgical instrument has disappeared based on N consecutive frames of the second target frame image, and then determine that the target surgical event corresponding to the target surgical instrument has ended, and the moment corresponding to the second target frame image when the target surgical instrument is not recognized for the first time is the end moment of the target surgical event; the end moment is used to determine the event label or update the event label.

[0104] In some optional embodiments, the second recognition module is specifically used to use a preset sliding window to obtain N frames of second target frame images after the first target frame image, where N is the width of the sliding window; if among the N frames of the second target frame images, there are M frames of the second target frame images in which the target surgical instrument is not detected, it is determined that the target surgical instrument disappears; M is less than or equal to N and greater than a preset frame number threshold.

[0105] In other optional embodiments, the first determination module is used to determine that the target surgical event corresponding to the target surgical instrument has ended if it is determined that the target surgical instrument has disappeared based on N consecutive frames of the second target frame images, and the interval between the moment corresponding to the second target frame image at which the target surgical instrument is not recognized for the first time and the start moment is greater than a first preset threshold.

[0106] In some optional embodiments, the surgical report generating device further includes:

[0107] A third recognition module, used for performing surgical instrument recognition on a third target frame image following the first target frame image in chronological order;

[0108] a duration acquisition module, configured to acquire a first duration of each appearance of the target surgical instrument if it is detected based on the third target frame image that the target surgical instrument disappears and then reappears;

[0109] A variance calculation module, used for calculating the variances of the first time lengths;

[0110] The second determination module is used to determine the moment when the target surgical instrument disappears for the last time as the end moment of the target surgical event if the variance is less than a second preset threshold; the end moment is used to determine the event label or update the event label.

[0111] In some optional embodiments, the surgical report generating device further includes:

[0112] A frame image selection module, used to obtain M fourth target frame images after the first target frame image by using a preset sliding window, where M is the width of the sliding window;

[0113] An event end determination module is used to determine that the target surgical event has not yet ended if one of the fourth target frame images of M frames detects the target surgical instrument, obtain the M frames of the fourth target frame images in the next sliding window, and perform the target surgical instrument detection until the target surgical instrument is not detected in any of the M frames of the fourth target frame images in a sliding window, and determine that the target surgical event is ended.

[0114] In some optional embodiments, the surgical report generating device further includes:

[0115] A first operation receiving module, used for receiving a target operation marked for the event;

[0116] A first response module, configured to display an event description interface in response to the target operation;

[0117] A second operation receiving module, used for receiving a target input for the event description interface;

[0118] A second response module, configured to display event description information input by a user in response to the target input;

[0119] The second report generating module is used to generate a text report of the operation based on the event description information.

[0120] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0121] The surgical report generating device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0122] The embodiment of the present invention also provides a computer device having the above Figure 4 The surgical report generating device shown.

[0123] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0124] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0125] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0126] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0128] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0129] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0130] The computer device also includes a communication interface for the computer device to communicate with other devices or a communication network.

[0131] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0132] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0133] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for generating a surgical report, characterized in that: The method comprises: Acquire video images collected during surgery; For the first target frame image in the video image, using an image recognition neural network model to identify the surgical instrument; If a target surgical instrument is identified, and the first target frame image is the first frame image in which the target surgical instrument is identified, a target surgical event corresponding to the target surgical instrument is determined according to a correspondence between the surgical instrument and the surgical event, and a target time corresponding to the first target frame image is a start time of the target surgical event; An event annotation is added to a target position on a target time axis to form a surgical video report; wherein the target time axis is a time axis corresponding to the video image, and the target position includes at least a position corresponding to the target moment.

2. The method according to claim 1, characterized in that The event annotation includes an event tag, and different event tags correspond to different surgical events; and / or, The surgical events correspond to different surgical steps during the surgical procedure.

3. The method according to claim 1, characterized in that After determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes: Performing surgical instrument recognition on a second target frame image following the first target frame image in chronological order; If the target surgical instrument is determined to have disappeared based on N consecutive frames of the second target frame images, the target surgical event corresponding to the target surgical instrument is determined to have ended, and the moment corresponding to the second target frame image when the target surgical instrument is not recognized for the first time is the end moment of the target surgical event; the end moment is used to update the event label.

4. The method according to claim 3, characterized in that The performing surgical instrument recognition on the second target frame image after the first target frame image in chronological order comprises: Using a preset sliding window, acquiring N frames of second target frame images after the first target frame image, where N is the width of the sliding window; If among the N frames of the second target frame images, there are M frames of the second target frame images in which the target surgical instrument is not detected, it is determined that the target surgical instrument disappears; M is less than or equal to N and greater than a preset frame number threshold.

5. The method according to claim 3, characterized in that: If it is determined based on the continuous N frames of the second target frame images that the target surgical instrument disappears, then determining that the target surgical event corresponding to the target surgical instrument is finished includes: If the target surgical instrument is determined to have disappeared based on N consecutive frames of the second target frame images, and the interval between the moment corresponding to the second target frame image when the target surgical instrument is not recognized for the first time and the start moment is greater than a first preset threshold, then it is determined that the target surgical event corresponding to the target surgical instrument has ended.

6. The method according to claim 1, characterized in that After determining the target surgical event corresponding to the target surgical instrument according to the correspondence between the surgical instrument and the surgical event, the method further includes: Performing surgical instrument recognition on a third target frame image following the first target frame image in chronological order; If it is detected based on the third target frame image that the target surgical instrument disappears and then reappears, obtaining a first duration of each appearance of the target surgical instrument; Calculating variances of a plurality of said first durations; If the variance is less than a second preset threshold, the moment when the target surgical instrument disappears for the last time is determined as the end moment of the target surgical event; the end moment is used to determine the event label or update the event label.

7. The method according to claim 1, characterized in that After adding the event annotation to the target position on the target time axis, the method further includes: Receiving a target operation marked for the event; In response to the target operation, displaying an event description interface; receiving a target input for the event description interface; In response to the target input, displaying event description information input by the user; A text report of the operation is generated based on the event description information.

8. A surgical report generating device, characterized in that: The device comprises: A video image acquisition module, used to acquire video images collected during the operation; A first recognition module, used for identifying surgical instruments using an image recognition neural network model for a first target frame image in the video image; a surgery time determination module, configured to determine a target surgery event corresponding to the target surgery instrument according to a correspondence between the surgery instrument and the surgery event if a target surgery instrument is identified and the first target frame image is the first frame image in which the target surgery instrument is identified, and the target time corresponding to the first target frame image is the start time of the target surgery event; The first report generation module is used to add event annotations to the target position on the target time axis to form a surgical video report; wherein the target time axis is the time axis corresponding to the video image, and the target position at least includes the position corresponding to the target moment.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the surgical report generation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the surgical report generation method according to any one of claims 1 to 7.

Images