Surgical instrument identification method and device

Abstract

The embodiment of the invention provides an identification method and device of a surgical instrument, and is used for solving the problem that the surgical instrument cannot be positioned in an operation in the prior art. The method comprises the following steps: acquiring a video stream collected in an operation execution process, wherein the video stream comprises a plurality of video frames; performing feature extraction on each video frame in the plurality of video frames to obtain a feature map of each video frame; performing global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame; according to the first feature vectors of at least two continuous video frames in the plurality of video frames, extracting context features of each of the at least two video frames in the at least two video frames to obtain a second feature vector of each of the at least two video frames; and determining the position of at least one surgical instrument included in the video stream in the video frame based on the second feature vector corresponding to each video frame.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for identifying surgical instruments. Background Art

[0002] Most of the surgical instrument identification systems in the prior art are based on additional equipment such as physical RFID to track instruments, but they cannot automatically record the use of instruments and surgical events according to the surgical scene. In addition, the manual recording method of medical staff not only wastes precious surgical time, but also affects the sterility of the surgery. There is a time difference between manual recording and actual use time, which is very unfriendly. RFID and other methods cannot achieve intraoperative marking, and medical staff are required to identify the location of the equipment. Automatic recording of intraoperative use cannot be achieved. RFID will cause additional consumables and have requirements for the type of surgical instruments. Summary of the invention

[0003] The embodiments of the present application provide a method and device for identifying a surgical instrument, so as to solve the problem in the prior art that the surgical instrument cannot be positioned during surgery.

[0004] In a first aspect, the present application provides an embodiment of a method for identifying a surgical instrument, comprising:

[0005] Acquire a video stream collected during a surgical procedure, wherein the video stream includes multiple video frames; perform feature extraction on each of the multiple video frames to obtain a feature map of each video frame; perform global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame; extract context features of each of the at least two video frames in the at least two video frames based on the first feature vectors of at least two consecutive video frames in the multiple video frames to obtain a second feature vector of each of the at least two video frames; determine the position of at least one surgical instrument included in the video stream in the video frame based on the second feature vector corresponding to each video frame, wherein the at least one surgical instrument is an instrument used during the surgical procedure.

[0006] Based on the above solution, no additional consumables are required, and the position of surgical instruments can be automatically identified based on the video stream alone. This method has no restrictions on the types of surgical instruments, and there is no need for personnel to operate the information system to record surgical instruments.

[0007] In a possible implementation, the method further includes: determining, based on a second feature vector of each video frame in the plurality of video frames, an instrument category to which at least one surgical instrument included in the video stream belongs.

[0008] Based on the above scheme, the types of surgical instruments used during the operation can be identified, so that personnel can count the types of instruments during the event, which is beneficial for subsequent review.

[0009] In one possible implementation, the determining of the position of at least one of the instruments included in the video stream in the video frame based on the second feature vector corresponding to each video frame includes: inputting the second feature vector of each video frame into a fully connected layer to determine the heat maps corresponding to different instrument categories in each video frame; comparing the thermal values ​​in the heat maps corresponding to the different instrument categories with a set threshold, and taking the area where the thermal value is greater than the set threshold as the position of the at least one surgical instrument in each video frame.

[0010] In one possible implementation, determining the instrument category to which at least one surgical instrument included in the video stream belongs based on the second feature vector of each video frame in the multiple video frames includes: inputting the second feature vector of each video frame into a fully connected layer to determine the heat maps corresponding to different instrument categories in each video frame; and determining the instrument category to which at least one surgical instrument included in each video frame belongs based on the thermal values ​​in the heat maps corresponding to the different instrument categories.

[0011] In one possible implementation, the method also includes: determining a first duration of continuous appearance of the first surgical instrument in the video stream based on the position of the first surgical instrument in the video stream; determining, based on configuration information, a duration of a first sub-event corresponding to the first surgical instrument as the first duration; the configuration information is used to describe an association between multiple sub-events included in the surgery and at least one required instrument category; the first sub-event is any sub-event among the multiple sub-events, the first instrument is any surgical instrument of at least one surgical instrument included in the video stream, and there is a corresponding relationship between the first surgical instrument and the first sub-event in the configuration information.

[0012] Based on the above scheme, during the event process, the instrument can be intelligently identified, and the duration of the instrument's appearance can be counted. Through the correspondence between surgical instruments and sub-events, the duration of sub-events during the operation can be determined, assisting doctors in automatically recording intraoperative events.

[0013] In a second aspect, an embodiment of the present application provides an electronic device, including a memory and a processor:

[0014] A memory for storing program instructions;

[0015] The processor is used to obtain a video stream collected during the operation, wherein the video stream includes a plurality of video frames;

[0016] Extracting features from each of the multiple video frames to obtain a feature map of each video frame;

[0017] Performing global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame;

[0018] Extracting context features of each of the at least two video frames in the at least two video frames according to first feature vectors of at least two consecutive video frames in the plurality of video frames to obtain a second feature vector of each of the at least two video frames;

[0019] The determination module is used to determine the position of at least one surgical instrument included in the video stream in the video frame based on the second feature vector corresponding to each video frame, and the at least one surgical instrument is an instrument used in performing a surgical procedure.

[0020] In a possible implementation, the processor is further used to determine, based on a second feature vector of each video frame in the multiple video frames, an instrument category to which at least one surgical instrument included in the video stream belongs.

[0021] In one possible implementation, the processor, when determining the position of at least one surgical instrument included in the video stream in the video frame based on the second feature vector corresponding to each video frame, is specifically used to: input the second feature vector of each video frame into the fully connected layer to determine the heat maps corresponding to different instrument categories in each video frame; compare the thermal values ​​in the heat maps corresponding to the different instrument categories with a set threshold, and take the area where the thermal value is greater than the set threshold as the position of the at least one surgical instrument in each video frame.

[0022] In a possible implementation, the processor, when determining, according to the second feature vector of each of the multiple video frames, the instrument category to which at least one surgical instrument included in the video stream belongs, is specifically configured to:

[0023] Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame;

[0024] The instrument category to which at least one surgical instrument included in each video frame belongs is determined according to the thermal values ​​in the thermal maps corresponding to the different instrument categories.

[0025] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect and different implementations of the first aspect.

[0026] In addition, the technical effects brought about by any implementation method of the second to third aspects can refer to the technical effects brought about by the first aspect and different implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of the system architecture provided for an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0030] Figure 3 A flowchart of a surgical instrument identification method provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of configuration information provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of an interface provided in an embodiment of the present application;

[0033] Figure 6 A structural diagram of an Inception A module provided in an embodiment of the present application;

[0034] Figure 7 A structural diagram of an Inception B module provided in an embodiment of the present application;

[0035] Figure 8 A structural diagram of an Inception C module provided in an embodiment of the present application;

[0036] Fig. 9 A structural diagram of an Inception D module provided in an embodiment of the present application;

[0037] Fig.10 A structural diagram of an Inception E module provided in an embodiment of the present application;

[0038] Fig.11 A schematic diagram of a thermal diagram of a surgical instrument provided in an embodiment of the present application;

[0039] Fig.12 A schematic diagram of the position of a surgical instrument in a video frame provided in an embodiment of the present application;

[0040] Fig.13 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0042] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0043] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0044] At present, the information-based operating room applies modern information technology to the working environment of the operating room, and uses some software and hardware facilities to establish a complete operating room in a clean operating room. Intraoperative images will collect real-time intraoperative conditions in detail, but the use of instruments and intraoperative events during the operation still need to be manually entered into the surgical anesthesia system or intraoperative recording system by medical staff. Most of the surgical instrument identification systems on the market are based on additional equipment such as physical RFID to track instruments, but they cannot automatically record the use of instruments and surgical events according to the surgical scene, and there is no product that can enable doctors to review the operation after surgery and record the use of surgical instruments in detail. The manual recording method of medical staff not only wastes precious surgical time, but also requires on-site personnel to operate the information system in real time, which also affects the sterility of the operation. There is a time difference between manual recording and actual use time, which is very unfriendly. However, methods such as RFID cannot achieve intraoperative marking, and medical staff are required to identify the location of the equipment, and automatic recording of intraoperative use cannot be achieved. In addition, RFID will cause additional consumables and have requirements for the type of surgical instruments.

[0045] In response to the above problems, the embodiments of the present application provide a method and device for identifying instruments, which uses a class activation graph and an attention mechanism to identify surgical instruments and instrument positions for each video frame in a video stream, and determines the record of intraoperative events through the correspondence between instrument types and intraoperative events. Furthermore, intraoperative surgical instrument records and type counting can be achieved, assisting doctors in recording surgical events and subsequent review by doctors.

[0046] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0047] The surgical instrument recognition method provided in the embodiment of the present application can be implemented by an electronic device. In some embodiments, the electronic device can be a terminal device. The terminal device can be a display device with a display function. The display device can include: a smart TV, a mobile phone, a tablet computer, etc. In other embodiments, the electronic device can be implemented by one or more servers, and the server can be a local server or a cloud server. In this case, the electronic device can also execute the training method of the surgical instrument recognition network in the embodiment of the present application.

[0048] The following describes the structure and application scenarios of the electronic device by taking the electronic device implemented by one or more servers as an example. Figure 1As shown, the server 100 may be implemented by a physical server or a virtual server. The server may be implemented by a single server or a server cluster composed of multiple servers. The training method of the surgical instrument recognition network or the surgical instrument recognition method provided in the present application may be implemented by a single server or a server cluster. Figure 1 As shown, the server 100 is connected to the display device 200 as an example. The server 100 can execute the training method of the surgical instrument recognition network. In some scenarios, the server 100 can receive the recognition task of the surgical instrument sent by the display device 200, and recognize the surgical instrument according to the recognition task of the surgical instrument. In one example, the server 100 can send the recognition result of the surgical instrument to the display device 200, so that the display device 200 displays the received recognition result of the surgical instrument.

[0049] In the present application, the electronic device may also be a personal computer, a handheld or laptop device, a mobile device (such as a mobile phone, a tablet computer, a personal digital assistant, etc.

[0050] As an example, see Figure 2 As shown, the server 100 may include a processor 110 and a communication interface 120. The electronic device may also include a memory 130. Of course, the server may also include other components, Figure 2 Not shown.

[0051] The communication interface 120 is used to communicate with the display device, to receive a surgical instrument identification task sent by the display device, or to send an identification result of the surgical instrument to the display device.

[0052] In the embodiment of the present application, the processor 110 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0053] The processor 110 is the control center of the electronic device, which connects various parts of the electronic device using various interfaces and routes, executes various functions of the electronic device and processes data by running or executing software programs / or modules stored in the memory 130, and calling data stored in the memory 130. Optionally, the processor 510 may include one or more processing units. The processor 110 may be, for example, a control component such as a processor, a microprocessor, a controller, and may be, for example, a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0054] The memory 130 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to business processing, etc. The memory 130, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 130 may include at least one type of storage medium, for example, a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 130 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 130 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0055] It should be noted that the above Figure 1-2 The structure shown is only an example and is not limited to this embodiment of the present invention.

[0056] The recognition of surgical instruments provided in the embodiments of the present application can be achieved through a surgical instrument recognition network, which may include a spatial feature extraction module, a CAP (context-aware attentional pooling) context perception module, a multi-head attention module and a fully connected layer.

[0057] The present application embodiment provides a method for identifying a surgical instrument. Figure 3 The process of the surgical instrument identification method is exemplarily shown, and the process can be executed by an electronic device, which can be Figure 2 The electronic device shown can specifically perform surgical instrument identification through the processor 210 in the electronic device. The specific process is as follows:

[0058] 301, obtaining a video stream collected during the operation.

[0059] In some embodiments, the video stream includes multiple video frames. The video frames are video frames captured by the capture device. Exemplarily, after the capture device captures the video frames, the server obtains the video frames from the capture device. In some scenarios, after the capture device captures a frame of video, the server obtains the frame of video from the capture device, or after the capture device captures multiple video frames, the server can obtain multiple video frames from the capture device.

[0060] In some embodiments, a video file sent by an acquisition device may be received, the video file including video frames. The video file may be an encoded file of a video, so that the received video file may be decoded to obtain a video stream. Encoding the video may effectively reduce the file size of the video file for easy transmission. Thus, the transmission speed of the video may be increased to improve the efficiency of subsequent device recognition of the video. The method for obtaining the encoded bitstream data may be any applicable method, including but not limited to: Real Time Streaming Protocol (RTSP), Open Network Video Interface Forum (ONVIF) standard or private protocol method, etc.

[0061] 302 , extract features from each of the multiple video frames to obtain a feature map of each video frame.

[0062] In some embodiments, after the video stream is acquired, the video stream may be preprocessed. Taking the video stream as a laparoscope real-time video image as an example, the preprocessing steps specifically include:

[0063] First, useless edges are cropped. In the real-time video images of the laparoscope, there will be invalid black edges on the periphery. The grayscale value can be used to crop the invalid area of ​​the image to retain the valid information in the image.

[0064] Next, the image size is adjusted to uniformly adjust the size of each video frame in the video stream to 299*299*3 for subsequent recognition. Furthermore, the contrast and grayscale value can be adjusted to highlight the video feature information.

[0065] In some embodiments, after preprocessing each video frame in the video stream, spatial feature extraction can be performed on each video frame to obtain a high-dimensional feature matrix corresponding to each video frame. As an example, after feature extraction is performed on a video frame with a resolution of 299*299*3, a high-dimensional feature matrix of 8*8*2048 dimensions corresponding to each video frame is obtained.

[0066] 303 , perform global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame.

[0067] As an example, a high-dimensional feature matrix of 8*8*2048 dimensions is globally averaged and pooled, and the first feature vector of each video frame can be expressed as 1*1*2048.

[0068] 304 , extracting context features of each of the at least two video frames in the at least two video frames according to the first feature vectors of at least two consecutive video frames in the plurality of video frames, to obtain a second feature vector of each of the at least two video frames.

[0069] In some embodiments, the first feature vectors of at least two videos can be spliced, and the spliced ​​first feature vector can be input into the multi-head attention module for multi-head attention adjustment to obtain the spliced ​​first feature vector after the multi-head attention adjustment, and the spliced ​​first feature vector after the multi-head attention adjustment can be split to obtain the second feature vector specifically corresponding to each video frame.

[0070] As an example, the first feature vector features corresponding to L video frames can be concatenated to obtain a feature matrix of L*2048, and the feature matrix of L*2048 is adjusted by head multi-attention to obtain an adjusted feature matrix of L*2048. The adjusted feature matrix of L*2048 is split into L feature vectors of 1*2048, and each feature vector of 1*2048 is used as the second feature vector of each video frame.

[0071] 305 : Determine the position of at least one surgical instrument included in the video stream in each video frame based on the second feature vector corresponding to each video frame.

[0072] In some embodiments, the second eigenvector of each video frame can be input into the fully connected layer to determine the heat maps corresponding to different instrument categories in each video frame. Furthermore, the thermal values ​​in the heat maps corresponding to different instrument categories can be compared with the set threshold value, and the area where the thermal value is greater than the set threshold value is used as the position of at least one surgical instrument in each video frame. As an example, 5 instrument categories are preset. When the second eigenvector of the video frame is input into the fully connected layer, the heat maps corresponding to the 5 instrument categories in the video frame can be determined. Furthermore, the thermal values ​​of the heat maps corresponding to each instrument category can be compared with the set threshold value, and the area where the thermal value is greater than the set threshold value is used as the position of the surgical instrument corresponding to the instrument category in the video.

[0073] In some embodiments, the instrument category to which at least one surgical instrument included in the video stream belongs can be determined based on the second feature vector of each video frame in the plurality of video frames. The at least one surgical instrument is an instrument used during the surgical procedure. Specifically, the second feature vector of each video frame can be input into a fully connected layer to determine the heat map corresponding to different instrument categories in each video frame. The instrument category to which at least one instrument included in each video frame belongs can be determined based on the thermal values ​​in the heat map corresponding to different instrument categories.

[0074] In some embodiments, the first duration of the continuous appearance of the first surgical instrument in the video stream can be determined based on the position of the first surgical instrument in the video stream. Furthermore, the duration of the first sub-event corresponding to the first surgical instrument can be determined as the first duration based on the configuration information. The configuration information is used to describe the association between multiple sub-events included in the surgery and at least one required instrument category, the first sub-event is any sub-event among the multiple sub-events, the first surgical instrument is any instrument of at least one surgical instrument included in the video stream, and there is a corresponding relationship between the first surgical instrument and the first sub-event in the configuration information.

[0075] As an example, the configuration information is used to describe the association between multiple sub-events included in the surgery and at least one required instrument category. As an example, the corresponding relationship between the instrument category and the first sub-event is as follows: Figure 4 When a flusher appears in the video stream, it can be determined that the current sub-event is cleaning and stopping bleeding; when a specimen bag appears in the video stream, it can be determined that the current sub-event is specimen packaging.

[0076] Furthermore, the duration of the sub-event can be determined based on the duration of the continuous appearance of the surgical instrument. As an example, surgical instruments are identified based on the video stream, and a total of flusher, specimen bag, hemostatic forceps, scissors, skin preparation device and bipolar electrocoagulation are identified. The corresponding sub-events in the configuration information are cleaning and hemostasis, specimen packaging, hemostatic forceps hemostasis, shearing, clamping blood vessels and coagulation. Furthermore, the duration of the sub-event can be determined based on the duration of the different identified surgical instruments.

[0077] In some embodiments, the video stream can be segmented according to the duration of each sub-event. The sub-events and durations can be displayed on the interface. Figure 5 As shown, the surgical process includes six sub-events: cleaning and hemostasis, flushing, specimen packaging, hemostasis with hemostatic forceps, clamping blood vessels, and coagulation, and each sub-event corresponds to a different duration. In response to the user clicking the control corresponding to the sub-event, the video corresponding to the sub-event is played. As an example, when the user clicks the view control corresponding to cleaning and hemostasis, the video stream corresponding to cleaning and hemostasis is played.

[0078] In some embodiments, the types of surgical instruments in the video stream can be counted to assist medical staff in performing the counting.

[0079] In some embodiments, in view of the characteristics of surgical events, it is not the processing of a video frame, and the use of surgical instruments is continuous. Therefore, the following post-processing can be performed for surgical instrument identification:

[0080] First, abnormal identification, elimination and correction can be performed: because the use and operation of surgical instruments must be continuous, if a situation that is inconsistent with other continuous video frames suddenly appears in a continuous video frame, it will be eliminated and corrected. As an example, the surgical instrument recognition results of video frames 25-37 and 39-46 are all scissors, and the surgical instrument recognition result of video frame 38 is hemostatic forceps. Since the surgical instrument recognition results of the continuous video frames before and after the 38th video frame are all scissors, it can be determined that the surgical instrument recognition result of the 38th video frame is abnormal, then the abnormal recognition result is deleted, and the surgical instrument recognition result of the 38th video frame is corrected to scissors.

[0081] Secondly, non-intracavitary device marks can be deleted: the video stream can be gray-processed, and then each video frame can be determined whether it is an intracavitary video frame or an extracavitary video frame according to the pixel gray value, and then useless marks can be deleted. Specifically, a gray value threshold and a set threshold can be set. When the number of pixels in a certain video frame whose pixel gray value is less than the gray value threshold accounts for a proportion of the video frame that is less than the set threshold, the video frame is determined to be an extracavitary video frame; otherwise, it is an intracavitary video frame.

[0082] In some embodiments, the spatial feature extraction module may be composed of multiple different convolution modules. In one possible implementation, the spatial feature extraction module may be composed of 3 Inception A modules, 1 Inception B module, 4 Inception C modules, 1 Inception D module, and 1 Inception E module. Figure 6-10 As shown, this is a schematic diagram of the Inception AE module.

[0083] As an example, after obtaining the video frame, the current video frame can be preprocessed, and the size of the processed video frame is 299*299*3. Further, the 299*299*3 video frame can be input into the spatial feature extraction module, first through three 3*3 convolutions with step sizes of 2, 1, and 1, and the output is 147*147*64. Further, maximum pooling can be performed, where kernel_size=3, stride=2, and the output size is 73*73*64. Then, after a 1*1 convolution with a step size of 1, the output is 73*73*80, and then after a 3*3 convolution with a step size of 1, the output is 71*71*192, and then maximum pooling is performed, where kernel_size=3, stride=2, and the output size is 35*35*192. Further, the input is taken as follows Figure 6 The Inception A module shown in the figure has channel m=32, and the obtained feature map is 35*35*(224+32)=35*35*256. After that, it passes through two Inception A modules, and the channel m=64, and the obtained feature map is 35*35*(224+64)=35*35*288. Further, the output of the Inception A module is as follows Figure 7 The input of the Inception B module shown in Figure 1 yields a feature map of 17*17*768. Figure 8 The Inception C module shown in the figure is used for feature extraction, where the number of channels c = 128, and the obtained feature map is 17*17*768. Further, after passing through two Inception C modules, where the number of channels c = 160, the obtained feature map is 17*17*768. After that, the 17*17*768 feature map can be used as Fig. 9 The input of the Inception D module shown in the figure is used for feature extraction, and the obtained feature map is 8*8*1280, and the 8*8*1280 feature map is used as Fig.10 The input of the Inception E module shown in the figure is used for feature extraction, and the resulting feature map is 8*8*2048.

[0084] After the spatial feature extraction module performs feature extraction to obtain an 8*8*2048 feature map of the video frame, global average pooling can be performed through the CAP context perception module to obtain a first feature vector of the video frame of 1*1*2048.

[0085] In some scenarios, the first feature vector of 1*1*2048 output by the CAP module can be used as the input of the Transformer. In some embodiments, the feature vectors of L video frames, i.e., L*1*1*2048 with a sequence length of L, can be transformed into L*2048 as the input of the Transformer. Since the surgical video is continuous, the operation of the surgical instrument must be continuous, so adding the Transformer and the multi-head attention mechanism can use the first feature vectors of the L video frames as the sequence input for multi-head attention adjustment. The L*2048 sequence will be used as the QKV value in the multi-head attention sequence for multi-head attention adjustment to obtain a feature matrix of L*2048, and each 2048 feature, i.e., L 1*1*2048 feature vectors, is used as the second feature vector corresponding to the L video frames. As an example, taking L=30 as an example, the first feature vector of 1*1*2048 of 30 video frames can be transformed into 30*2048 and input into the Transformer and multi-head attention adjustment is performed to obtain a feature matrix of 30*2048. Furthermore, the 30*2048 feature matrix may be split into 30 1*1*2048 feature vectors as the second feature vectors corresponding to each of the 30 video frames.

[0086] Furthermore, the second feature vector corresponding to each video frame can be input into the fully connected layer to determine the type of at least one surgical instrument included in each video frame. As an example, the weight of the fully connected layer is 2048*c, and after the video frame passes through the fully connected layer, a classification vector of size 1*c is obtained, where c is the number of instrument types.

[0087] At the same time, the feature map of each channel in the second feature vector corresponding to each video frame can be weighted and summed to obtain an attention heat map, and the position of the surgical instrument in the video frame can be identified based on the heat map distribution. As an example, the weight of the fully connected layer can be used as the feature fusion weight, and the second feature vector can be linearly fused to obtain a heat map, and then the position of the surgical instrument can be labeled by setting a threshold. As an example, the heat map of the surgical instrument is as follows: Fig.11 As shown, the position of the surgical instrument is marked as Fig.12 shown.

[0088] Furthermore, the duration of use of the surgical instrument may be marked, and the current sub-event may be matched and marked according to the surgical instrument identification result and configuration information to determine the current sub-event and the duration of the sub-event.

[0089] Based on the same technical concept, see Fig.13 As shown, the embodiment of the present application provides an electronic device 1300 , which can perform any step in the above surgical instrument identification method, and will not be described here to avoid repetition. The device 1300 includes a memory 1301 and a processor 1302 .

[0090] The memory 1301 is used to store program instructions;

[0091] The processor 1302 is used to obtain a video stream collected during the operation, where the video stream includes multiple video frames;

[0092] Extracting features from each of the multiple video frames to obtain a feature map of each video frame;

[0093] Performing global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame;

[0094] Extracting context features of each of the at least two video frames in the at least two video frames according to first feature vectors of at least two consecutive video frames in the plurality of video frames to obtain a second feature vector of each of the at least two video frames;

[0095] The position of at least one surgical instrument included in the video stream in each video frame is determined based on the second feature vector corresponding to each video frame, and the at least one surgical instrument is an instrument used during a surgical procedure.

[0096] In some embodiments, the processor 1302 is further configured to determine, based on a second feature vector of each video frame in the plurality of video frames, an instrument category to which at least one surgical instrument included in the video stream belongs.

[0097] In some embodiments, the processor 1302, when determining the position of at least one surgical instrument included in the video stream in the video frame based on the second feature vector corresponding to each video frame, is specifically configured to:

[0098] Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame;

[0099] The thermal values ​​in the thermal maps corresponding to the different instrument categories are compared with the set thresholds, and the areas where the thermal values ​​are greater than the set thresholds are used as the positions of the at least one surgical instrument in each video frame.

[0100] In some embodiments, the processor 1302, when determining the instrument category to which at least one surgical instrument included in the video stream belongs according to the second feature vector of each video frame in the plurality of video frames, is specifically configured to:

[0101] Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame;

[0102] The instrument category to which at least one surgical instrument included in each video frame belongs is determined according to the thermal values ​​in the thermal maps corresponding to the different instrument categories.

[0103] In some embodiments, the processor 1302 is further configured to determine a first duration of continuous appearance of the first surgical instrument in the video stream based on a position of the first surgical instrument in the video stream;

[0104] Based on the configuration information, determine that the duration of the first sub-event corresponding to the first surgical instrument is a first duration;

[0105] The configuration information is used to describe the association between multiple sub-events included in the operation and at least one required instrument category; the first sub-event is any sub-event among the multiple sub-events, the first surgical instrument is any instrument of at least one surgical instrument included in the video stream, and there is a corresponding relationship between the first surgical instrument and the first sub-event in the configuration information.

[0106] In some embodiments, the processor 1302, when determining the instrument category to which at least one surgical instrument included in the video stream belongs according to the second feature vector of each video frame in the plurality of video frames, is specifically configured to:

[0107] Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame;

[0108] The instrument category to which at least one surgical instrument included in each video frame belongs is determined according to the thermal values ​​in the thermal maps corresponding to the different instrument categories.

[0109] In some embodiments, the electronic device 1300 may further include a display 1303, and the display 1303 is used to display the recognition result of the surgical instrument.

[0110] Based on the same technical concept, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes any step in the above-mentioned surgical instrument identification method.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for identifying a surgical instrument, characterized in that: include: Acquire a video stream collected during a surgical procedure, wherein the video stream includes a plurality of video frames; Extracting features from each of the multiple video frames to obtain a feature map of each video frame; Performing global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame; Extracting context features of each of the at least two video frames in the at least two video frames according to first feature vectors of at least two consecutive video frames in the plurality of video frames to obtain a second feature vector of each of the at least two video frames; The position of at least one surgical instrument included in the video stream in each video frame is determined based on the second feature vector corresponding to each video frame, and the at least one surgical instrument is an instrument used during a surgical procedure.

2. The method according to claim 1, characterized in that The method further comprises: According to the second feature vector of each video frame in the multiple video frames, the instrument category to which at least one surgical instrument included in the video stream belongs is determined.

3. The method according to claim 1 or 2, characterized in that The determining the position of at least one device included in the video stream in the video frame based on the second feature vector corresponding to each video frame includes: Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame; The thermal values ​​in the thermal maps corresponding to the different instrument categories are compared with the set thresholds, and the areas where the thermal values ​​are greater than the set thresholds are used as the positions of the at least one surgical instrument in each video frame.

4. The method according to claim 2, characterized in that The step of determining, based on the second feature vector of each video frame in the plurality of video frames, the instrument category to which at least one surgical instrument included in the video stream belongs, comprises: Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame; The instrument category to which at least one surgical instrument included in each video frame belongs is determined according to the thermal values ​​in the thermal maps corresponding to the different instrument categories.

5. The method according to claim 2 or 4, characterized in that The method further comprises: determining a first duration for which the first surgical instrument continuously appears in the video stream based on a position of the first surgical instrument in the video stream; Based on the configuration information, determine that the duration of the first sub-event corresponding to the first surgical instrument is a first duration; The configuration information is used to describe the association between multiple sub-events included in the operation and at least one required instrument category; the first sub-event is any sub-event among the multiple sub-events, the first surgical instrument is any surgical instrument of at least one surgical instrument included in the video stream, and there is a corresponding relationship between the first surgical instrument and the first sub-event in the configuration information.

6. An electronic device, characterized in that: including memory and processor; A memory for storing program instructions; The processor is used for collecting a video stream during the operation, wherein the video stream includes a plurality of video frames; Extracting features from each of the multiple video frames to obtain a feature map of each video frame; Performing global average pooling on the feature map of each video frame to obtain a first feature vector of each video frame; Extracting context features of each of the at least two video frames in the at least two video frames according to first feature vectors of at least two consecutive video frames in the plurality of video frames to obtain a second feature vector of each of the at least two video frames; The position of at least one surgical instrument included in the video stream in each video frame is determined based on the second feature vector corresponding to each video frame, and the at least one surgical instrument is an instrument used during a surgical procedure.

7. The electronic device according to claim 6, characterized in that: The processor is further configured to: According to the second feature vector of each video frame in the multiple video frames, the instrument category to which at least one surgical instrument included in the video stream belongs is determined.

8. The electronic device according to claim 6 or 7, characterized in that: The processor, when determining the position of at least one surgical instrument included in the video stream in the video frame based on the second feature vector corresponding to each video frame, is specifically configured to: Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame; The thermal values ​​in the thermal maps corresponding to the different instrument categories are compared with the set thresholds, and the areas where the thermal values ​​are greater than the set thresholds are used as the positions of the at least one surgical instrument in each video frame.

9. The electronic device according to claim 7, characterized in that: The processor, when determining the instrument category to which at least one surgical instrument included in the video stream belongs according to the second feature vector of each video frame in the plurality of video frames, is specifically configured to: Inputting the second feature vector of each video frame into a fully connected layer to determine heat maps corresponding to different equipment categories in each video frame; The instrument category to which at least one surgical instrument included in each video frame belongs is determined according to the thermal values ​​in the thermal maps corresponding to the different instrument categories.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 5.

Images