Method and device for determining key video frame in operation event

Abstract

The embodiment of the invention provides a method and a device for determining a key video frame in an operation event. The method and the device are used for solving the problem that a medical team cannot automatically store an image under the condition that the operation is not interfered. The method comprises the following steps: acquiring a video stream collected when a target operation event is executed, wherein the video stream comprises a plurality of video frames to be detected; performing target detection on each to-be-detected video frame, and determining a target object included in each to-be-detected video frame; determining a target operation stage in the target operation event corresponding to each to-be-detected video frame according to a feature map obtained by object features in the to-be-detected video frames; determining a standard object corresponding to the target operation stage based on a preset association relationship between the operation stage and the standard object; and when it is determined that the target object is not included in the standard object, determining that the to-be-detected video frame is a key video frame when a target operation event is executed.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for determining key video frames in a surgical event. Background Art

[0002] At present, during the operation, various types of equipment such as panoramic cameras, surgical field cameras, and laparoscopic cameras are used in information-based operating rooms to capture a large amount of image data so that doctors and other medical staff can better understand the operation and conduct subsequent evaluation, treatment, and research. The snapshot function can capture the surgical screen and provide support for subsequent analysis and research. However, the current snapshot function requires manual clicking to take pictures or additional factors such as voice and gestures to prompt the system to take pictures and save them. During the operation, the medical team cannot automatically save without interfering with the operation. Summary of the invention

[0003] The embodiments of the present application provide a method and device for determining key video frames in a surgical event, so as to solve the problem that the medical team cannot automatically save images without interfering with the operation.

[0004] In a first aspect, an embodiment of the present application provides a method for determining a key video frame in a surgical event, comprising:

[0005] A video stream collected when a target surgical event is performed is obtained, wherein the video stream includes a plurality of video frames to be detected; target detection is performed on each of the video frames to be detected to determine a target object included in each of the video frames to be detected; a target surgical stage in the target surgical event corresponding to each of the video frames to be detected is determined based on a feature map obtained from object features in the video frames to be detected; a standard object corresponding to the target surgical stage is determined based on an association relationship between a preset surgical stage and a standard object; and when it is determined that the target object is not included in the standard object, the video frame to be detected is determined to be a key video frame when the target surgical event is performed.

[0006] Based on the above scheme, by performing target recognition on the video frame to be detected, multiple target objects included in the video frame to be detected are determined, and the target surgical stage corresponding to the video frame to be detected is determined, and then the key video frame when performing the target surgical event can be determined and saved through the association relationship between the preset surgical stage and the standard object. This method can automatically save surgical images with teaching significance for subsequent analysis and research without interfering with the medical team.

[0007] In one possible implementation, target detection is performed on each of the video frames to be detected to determine the target object included in each of the video frames to be detected, including: performing target enhancement processing on the video frames to be detected to obtain a target enhancement feature map; and inputting the target enhancement feature map into a target detection model to determine the target object included in the video frames to be detected.

[0008] In a possible implementation, the target enhancement processing is performed on the video frame to be detected to obtain a target enhancement feature map, including: feature extraction is performed on the video frame to be detected to obtain a feature map of the video frame to be detected; background enhancement and target enhancement processing are performed on the feature map of each video frame to obtain a background feature map and a target feature map; and weighted fusion is performed on the background feature map and the target feature map to obtain a target enhancement feature map.

[0009] In a possible implementation, the inputting the target enhancement feature map into a target detection model to determine the target object included in the video frame to be detected includes: performing feature extraction on the target enhancement feature map through a convolutional layer in the target detection model to obtain a target feature map; inputting the target feature map into a region proposal module in the target detection model to determine a candidate region containing the target object; and inputting region features in the target feature map corresponding to the candidate region into a classifier in the target detection model to determine the category of the target object included in the candidate region.

[0010] In a possible implementation, the inputting the target feature map into the region proposal module in the target detection model to determine a candidate region including the target object includes: setting multiple anchor frames at the position where each feature value in the target feature map is located; determining the region category of the region where each anchor frame in the target feature map is located according to the feature value included in each anchor frame in the target feature map among the multiple anchor frames, the region category including the target region and the background region; when it is determined that the region where the anchor frame in the target feature map is located is the target region, using the region where the anchor frame in the target feature map is located as a candidate region containing the target object.

[0011] In a second aspect, an embodiment of the present application provides a device for determining key video frames in a surgical event, comprising: an acquisition module, used to acquire a video stream collected when performing a target surgical event, the video stream comprising multiple video frames to be detected; a detection module, used to perform target detection on each of the video frames to be detected, and determine the target object included in each of the video frames to be detected; a determination module, used to determine the target surgical stage in the target surgical event corresponding to each of the video frames to be detected based on a feature map obtained from object features in the video frames to be detected; based on an association relationship between a preset surgical stage and a standard object, determine the standard object corresponding to the target surgical stage; when it is determined that the target object is not included in the standard object, determine that the video frame to be detected is a key video frame when performing the target surgical event.

[0012] In one possible implementation, the detection module, when performing target detection on each of the video frames to be detected and determining the target object included in each of the video frames to be detected, is specifically used to: perform target enhancement processing on the video frames to be detected to obtain a target enhancement feature map; and input the target enhancement feature map into a target detection model to determine the target object included in the video frames to be detected.

[0013] In a possible implementation, the detection module, when performing target enhancement processing on the video frame to be detected to obtain a target enhancement feature map, is specifically used to: perform feature extraction on the video frame to be detected to obtain a feature map of the video frame to be detected; perform background enhancement and target enhancement processing on the feature map of each video frame to obtain a background feature map and a target feature map; and perform weighted fusion of the background feature map and the target feature map to obtain a target enhancement feature map.

[0014] In a third aspect, an embodiment of the present application provides an execution device, including:

[0015] A memory for storing program instructions;

[0016] The processor is used to call the program instructions stored in the memory, and execute the method described in the first aspect and different implementation methods of the first aspect according to the obtained program instructions.

[0017] In a fourth aspect, the present application is an embodiment that provides a computer-readable storage medium, wherein the computer-readable storage medium includes computer instructions, and when the computer program instructions are executed by a computer, the computer executes the method described in the first aspect and different implementation methods of the first aspect.

[0018] In addition, the technical effects brought about by any implementation method of the second to fourth aspects can refer to the technical effects brought about by the first aspect and different implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1A A schematic diagram of the architecture of a business system provided in an embodiment of the present application;

[0021] Figure 1B A schematic diagram of the architecture of another business system provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0023] Figure 3 A flowchart of a method for determining key video frames in a surgical event provided in an embodiment of the present application;

[0024] Figure 4 A ConvNeXt Block structure diagram provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a Downsample structure provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of a target detection model provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a conv layers layer provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of an RPN module provided in an embodiment of the present application;

[0029] Fig. 9 A dictionary schematic diagram provided for an embodiment of the present application;

[0030] Fig.10 A flowchart for determining a key video frame provided in an embodiment of the present application;

[0031] Fig.11 A structural diagram of a feature extraction module provided in an embodiment of the present application;

[0032] Fig.12 A schematic diagram of the structure of an attention enhancement module provided in an embodiment of the present application;

[0033] Fig.13 A schematic diagram of the structure of a Faster RCNN module provided in an embodiment of the present application;

[0034] Fig.14 A training flow chart of a Faster RCNN module provided in an embodiment of the present application;

[0035] Fig.15 A schematic diagram of a device for determining key video frames in a surgical event provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0038] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0039] At present, the information operating room applies modern information technology to the working environment of the operating room, and uses some software and hardware facilities to establish a complete operating room in the clean operating room. During the operation, various devices such as panoramic cameras, surgical field cameras, and laparoscopic cameras will be used to capture a large amount of image data so that doctors and other medical staff can better understand the operation and conduct subsequent evaluation, treatment and research. However, the current snapshot function requires manual clicking to take pictures or additional factors such as voice and gestures to prompt the system to take pictures. During the operation, the medical team cannot automatically take pictures without interfering with the operation.

[0040] In response to the above problems, an embodiment of the present application provides a method and device for determining key video frames in a surgical event, by performing target detection on the video frame to be detected, determining the target object included in the video frame to be detected, and determining the target surgical stage corresponding to the video frame to be detected. Through the association relationship between the pre-established surgical stage and the standard object, it is determined whether the video frame to be detected is a key video frame when performing the target surgical event. When it is determined to be a key video frame, the video frame to be detected can be saved. This method can automatically determine and save surgical images with teaching significance without interfering with the medical team for subsequent analysis and research.

[0041] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0042] The method for determining key video frames in surgical events provided in the embodiments of the present application can be implemented by an execution device. In some embodiments, the execution device can be a terminal device. The terminal device can be a display device with a display function. The display device can include: a smart TV, a mobile phone, a tablet computer, etc. In other embodiments, the execution device can be an electronic device, and the electronic device can be implemented by one or more servers, and the server can be a local server or a cloud server. In some scenarios, the electronic device can also execute the training method of the target detection model in the embodiments of the present application.

[0043] The following takes the execution device as an electronic device as an example to describe the structure and application scenario of the execution device. The electronic device can be implemented by one or more servers. Figure 1A As shown, the server 100 can be implemented by a physical server or a virtual server. The server can be implemented by a single server or a server cluster composed of multiple servers. The training method of the target detection model provided in the present application or the method for determining key video frames in a surgical event can be implemented by a single server or a server cluster. Figure 1A In the example, the server 100 is connected to the terminal device 300 and the display device 200. The server 100 can execute the training method of the target detection model. In some scenarios, the server 100 can also receive the task of determining the key video frame in the surgical event sent by the terminal device 300, or send the determination result of the key video frame in the surgical event to the terminal device 300. In other scenarios, the server 100 can also receive the task of determining the key video frame in the surgical event sent by the display device 200 and perform the recognition of the key video frame, or display the determination result of the key video frame in the surgical event through the display device 200. Figure 1B As shown, the server 100 is connected to the display device 200 as an example. The server 100 can execute the training method of the target detection model. In some scenarios, the server 100 can receive the task of determining the key video frame in the surgical event sent by the display device, and determine the key video frame in the surgical event according to the task of determining the key video frame in the surgical event. In one example, the server 100 can send the determination result of the key video frame in the surgical event to the display device 200, so that the display device 200 displays the received determination result of the key video frame in the surgical event.

[0044] The electronic device may also be a personal computer, a handheld or laptop device, a mobile device such as a mobile phone, a tablet computer, a personal digital assistant, etc.

[0045] As an example, see Figure 2 As shown, the electronic device may include a processor 110 and a communication interface 120. The electronic device may also include a memory 130. Of course, the electronic device may also include other components. Figure 2 Not shown.

[0046] The communication interface 120 is used to communicate with the display device, to receive a task for determining a key video frame in a surgical event sent by the display device, or to send a determination result of a key video frame in a surgical event to an electronic device.

[0047] In the embodiment of the present application, the processor 110 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0048] The processor 110 is the control center of the electronic device, which connects various parts of the electronic device using various interfaces and routes, executes various functions of the electronic device and processes data by running or executing software programs / modules stored in the memory 130, and calling data stored in the memory 130. Optionally, the processor 110 may include one or more processing units. The processor 110 may be, for example, a control component such as a processor, a microprocessor, a controller, and may be, for example, a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0049] The memory 130 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to business processing, etc. The memory 130, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 130 may include at least one type of storage medium, for example, a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 130 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 130 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0050] It should be noted that the above Figure 1A , Figure 1B , Figure 2 The structure shown is only an example and is not limited to this embodiment of the present invention.

[0051] The present application embodiment provides a method for determining key video frames in a surgical event. Figure 3 The process of the method for determining the key video frame in the surgical event is exemplarily shown, and the process can be executed by an execution device, which can be Figure 2 The electronic device shown can specifically determine the key video frames in the surgical event through the processor 110 in the electronic device. The specific process is as follows:

[0052] 301, obtaining a video stream collected when performing a target surgical event.

[0053] In some embodiments, the video stream includes multiple video frames to be detected. In some scenarios, when performing a target surgical event, an acquisition device may be used to capture images of the process of performing the target surgical event, thereby obtaining a video stream including multiple video frames to be detected.

[0054] As an example, during a cholecystectomy operation, the operation process can be recorded by an image acquisition device, thereby obtaining a video stream of the cholecystectomy operation.

[0055] 302 , performing target detection on each to-be-detected video frame to determine a target object included in each to-be-detected video frame.

[0056] In some embodiments, target detection is performed on each video frame to be detected, and the target object included in each video frame to be detected is determined. This can be achieved in the following way: target enhancement processing is performed on the video frame to be detected to obtain a target enhancement feature map, and the target enhancement feature map is input into a target detection model to determine the target object included in the video frame to be detected.

[0057] Exemplarily, when performing target enhancement processing on the video frame to be detected to obtain the target enhancement feature map, the video frame to be detected can be firstly subjected to feature extraction to obtain the feature map of the video frame to be detected. Further, background enhancement and target enhancement processing can be performed on the feature map of each video frame to obtain a background feature map and a target feature map. The background feature map and the target feature map are then weightedly fused to obtain the target enhancement feature map.

[0058] In some scenarios, when extracting features from the video frames to be detected, the ConvNeXt network, which has achieved internationally leading levels on the Image-Net dataset, can be used as a feature extractor to further extract features from the video frames to be detected.

[0059] Specifically, after obtaining the video frame to be detected, the size of the video frame to be detected can be resized to a fixed size. For example, the video to be detected can be resized to 224×224×3. Furthermore, each video frame to be detected is passed through a 2D convolution layer with a convolution kernel size of 4x4, a stride size of 2, and a channel number of 96, so that the feature map size becomes 112×112×96. After three times of repeated stacking of ConvNeXt Block and Downsample structure, the feature map size of the video frame to be detected becomes 14×14×384. Among them, the ConvNeXt Block structure is as follows: Figure 4 As shown, the Downsample structure is as follows Figure 5 shown.

[0060] Furthermore, in order to improve the accuracy of target recognition, the attention mechanism can be used to make the target in each frame of the video more focused and remove background noise. For example, the feature map of the video frame can be input into the attention enhancement module to perform background enhancement and target enhancement processing on the feature map of the video frame to obtain a background feature map and a target feature map. Then, the background feature map and the target feature map are weighted fused to obtain a target enhancement feature map.

[0061] Specifically, the feature image of the video frame to be detected generated by the feature extraction module can be input into two ConvNeXtBlock structures. Among them, one ConvNeXt Block structure is used to extract the target in the feature map, and one ConvNeXtBlock structure is used to extract the background in the feature map. Through the two ConvNeXt Block structures, a target feature map and a background feature map of size 7×7×768 can be obtained. Furthermore, different weights can be assigned to them and fused into a target enhanced feature map through dot multiplication. Among them, the target features in the target enhanced feature map are more obvious and the background is weaker. In some embodiments, the size of the target enhanced feature map output by the attention enhancement module is 7×7×768. In some scenarios, the target enhanced feature map satisfies the conditions shown in the following formula:

[0062]

[0063] Among them, z represents the target enhanced feature map, are the target feature map and the background feature map, w 1 and w 2 is the weight.

[0064] In some embodiments, the object detection model includes a convolutional layer, a region proposal module, and a classifier, such as Figure 6As shown. When the target enhancement feature map is input into the target detection model to determine the target object included in the video frame to be detected, the target enhancement feature map can be first subjected to feature extraction through the convolution layer in the target detection model to obtain the target feature map. As an example, the target detection model can be a Faster RCNN network, and the convolution layer in the target detection model can be implemented by a convlayers layer. Exemplarily, the conv layers layer can be composed of 13 conv layers, 13 relu layers and 4 pooling layers. The structure of the conv layers layer is as follows Figure 7 As shown in the figure, the convolution kernel of the convolution layer has kernel_size=3, padding=1, and stride=1. In the pooling layer, kernel_size=2, padding=0, and stride=2. After the target enhanced feature map is input into the conv layers layer for feature extraction, the target feature map can be obtained.

[0065] Furthermore, the target feature map can be input into the region proposal module in the target detection model to determine the candidate region containing the target object.

[0066] For example, the region proposal module can be implemented by a region proposal network (RPN). As an example, the structure of the RPN module is as follows: Figure 8 shown.

[0067] When the target feature map is input into the region proposal module in the target detection model to determine the candidate region including the target object, it can be implemented in the following manner: multiple anchor frames can be set at the position where each feature value in the target feature map is located, and the region category of the region where each anchor frame in the target feature map is located is determined according to the feature value included in each anchor frame in the target feature map by the multiple anchor frames. The region category includes the target region and the background region. Further, when it is determined that the region where the anchor frame in the target feature map is located is the target region, the region where the anchor frame in the target feature map is located is used as the candidate region containing the target object.

[0068] As an example, nine anchor frames with three aspect ratios of [1:1, 1:2, 2:1] can be set at each position on the target feature map output by the convolution layer to cover targets of various positions and sizes on the video frame to be detected. Further, the region category of the region where each anchor is located in the target feature map can be determined according to the feature value included in the target feature map by each of the multiple anchors. Specifically, the positive and negative classifications can be obtained by softmax classification of anchors, and the probability of the target and background of each anchor can be calculated using softmax. Among them, positive indicates that the region category is the target region, and negative indicates that the region category is the background region. When it is determined that the region where the anchor frame is located in the target feature map is the target region, the region where the anchor frame is located in the target feature map can be used as a candidate region containing the target object. For example, the target anchor frame can be determined by the probability value of the positive of the target region, and then the region where the target anchor frame is located in the target feature map can be used as a candidate region including the target object. In some embodiments, the anchor frame can be represented by the coordinates of the target object bounding box, and the feature values ​​of the corresponding position can be extracted from the target feature map according to the coordinates of the target object bounding box to obtain the regional features of the candidate region.

[0069] Furthermore, after determining the candidate region containing the target object, the regional features at the position corresponding to the candidate region in the target feature map may be input into the classifier in the target detection model to determine the category of the target object included in the candidate region.

[0070] In some scenarios, target objects may include various organs and surgical instruments.

[0071] 303 , determining a target surgical stage in a target surgical event corresponding to each of the video frames to be detected according to a feature map obtained from the object features in the video frames to be detected.

[0072] In some embodiments, the video frame to be detected can be input into a classification model, and the classification model can be used to extract features from the video frame to be detected to obtain a feature map. Further, the target surgical stage in the target surgical event corresponding to the video frame to be detected can be determined according to the classification model. The target surgical stage is any surgical stage among multiple surgical stages included in performing the target surgical event.

[0073] In some embodiments, when determining the target surgical stage corresponding to the video frame to be detected, an auxiliary time feature dependency module and auxiliary timing feature information can be added, and a smoothing module can be added to correct abnormal information such as burrs to further improve the accuracy of target surgical stage recognition. For example, the target surgical stage corresponding to the M video frames before the video frame to be detected is surgical stage 1, and the target surgical stage corresponding to the N video frames after the video frame to be detected is surgical stage 1. When the classification model determines that the target surgical stage corresponding to the video frame to be detected is surgical stage 2, the recognition result of the video frame to be detected is corrected according to the recognition results of the video frames before and after the video frame to be detected, and the target surgical stage corresponding to the video frame to be detected is corrected to surgical stage 1.

[0074] 304 , based on the association between the preset surgical stage and the standard object, determine the standard object corresponding to the target surgical stage.

[0075] In some scenarios, there is a preset association between the surgical stage and the standard object, and the association can be expressed in the form of a dictionary. For example, the dictionary includes the surgical type, number, surgical stage, surgical stage name, and target object of the surgery, such as Fig. 9 shown.

[0076] In some scenarios, when the standard objects corresponding to surgical stage 1 include organ 1, instrument 1, instrument 2, and instrument 3, when the target surgical stage is determined to be surgical stage 1, according to the association between the surgical stage and the standard objects, the standard objects corresponding to the target surgical stage can be determined to be organ 1, instrument 1, instrument 2, and instrument 3. For example, when the target surgical stage is determined to be incision, the standard objects are abdomen, incision, and laparoscope.

[0077] 305 , when it is determined that the target object is not included in the standard objects, determine that the video frame to be detected is a key video frame when the target surgical event is performed.

[0078] In some embodiments, when it is determined that the target objects included in the video frame to be detected are organ 1, organ 2, instrument 1, instrument 2, instrument 3 and instrument 4, and the target surgical stage corresponding to the video frame to be detected is surgical stage 1, it is determined that the standard objects corresponding to surgical stage 1 do not include organ 2 and instrument 4, then the video frame to be detected can be determined to be a key video frame when performing the target surgical event, and the key video frame is then saved.

[0079] In some embodiments, when it is determined that the video frame to be detected is a key video frame, a photo of the video frame to be detected may be automatically taken, and the photo obtained by taking the photo may be saved.

[0080] In some scenarios, the saved photos can be named "Surgery Name - Surgical Stage - Organ or Instrument Not Appearing in the Dictionary" to facilitate subsequent reference by users.

[0081] As an example, after obtaining the video frame to be detected, the target surgical stage and the target object can be identified in the video frame to be detected respectively. Among them, the identification of the target surgical stage can be achieved through the classification model, and the identification of the target object can be achieved through the feature extraction module, the attention enhancement module and the Faster RCNN module. Furthermore, the target surgical stage output by the classification model and the target object output by the Faster RCNN module are consulted in the dictionary, and the association relationship between the surgical stage and the standard object is stored in the dictionary. When it is determined that the target object is not included in the standard object, a photo is automatically taken, such as Fig.10 shown.

[0082] Specifically, before the video frame to be detected is input into the feature extraction module, the video frame to be detected can be reset to 224×224×3, and the reset video frame is input into the feature extraction module, wherein the structure of the feature extraction module is as follows: Fig.11 As shown in Figure 2, the feature map size output by the feature extraction module is 14×14×384.

[0083] Furthermore, the feature map output by the feature extraction module can be input into the attention enhancement module. The attention enhancement module includes two ConvNeXt Block structures, such as Fig.12 As shown in the figure. After the feature maps are input into two ConvNeXtBlock structures, a 7×7×768 target feature map and a 7×7×768 background feature map are output. Then, a 7×7×768 target enhanced feature map is obtained by weighted fusion. Among them, * indicates weighted calculation.

[0084] Furthermore, the target enhanced feature map can be input into the Faster RCNN module for target detection. The structure of the Faster RCNN module is as follows: Fig.13 As shown in the figure, the category of the target object and the target object bounding box included in the video frame to be detected can be obtained through the Faster RCNN module.

[0085] In some embodiments, the identification of the target surgical stage and the identification of the target object can be performed in parallel, or the target surgical stage corresponding to the video frame to be detected can be determined first, and then the target object included in the video frame to be detected can be determined; or the target object included in the video frame to be detected can be determined first, and then the target surgical stage corresponding to the video frame to be detected can be determined. This application does not impose specific restrictions on this.

[0086] After determining the target surgical stage and the target object, based on the preset association between the surgical stage and the standard object, when it is determined that the standard object corresponding to the target surgical stage of the video frame to be detected does not include the target object identified based on the video frame to be detected, the video frame to be detected is determined to be the key video frame when performing the target surgical event and is saved.

[0087] In the embodiment of the present application, when determining the target object included in the video frame to be detected, it can be implemented by the Faster RCNN module. The training process of the Faster RCNN module is described below. Fig.14 As shown in the figure, the training process of the Faster RCNN module is as follows:

[0088] 1401, obtain a training sample set.

[0089] The training sample set includes multiple samples, each of which includes a first image, a bounding box containing a target object in the first image, and a category label.

[0090] 1402. Perform feature extraction on a first image in a first sample to obtain a feature map.

[0091] The first sample is any sample among multiple samples included in the training sample set.

[0092] Specifically, a pre-trained convolutional neural network (such as ResNet, VGG, etc.) can be used to extract features of the image to obtain a feature map corresponding to the first image.

[0093] 1403, input the feature map into a region proposal module to determine a predicted bounding box of the target object in the first image.

[0094] As an example, the RPN module can be used to find the bounding box that may include the target object. The specific process is as follows:

[0095] Step 1: Set 9 anchors with three aspect ratios of [1:1, 1:2, 2:1] at each position on the target feature map to cover targets of various positions and sizes on each target feature map. Determine the region category of each anchor in the target feature map based on the feature values ​​included in the target feature map by each of the multiple anchors. Specifically, the anchors can be classified into positive and negative categories by softmax, and the probability of the target and background of each anchor can be calculated using softmax. Among them, positive indicates that the region category is the target region, and negative indicates that the region category is the background region.

[0096] Step 2: Calculate the bounding box regression offset of each anchor. The coordinate offset of each anchor is: [tx, ty, tw, th], and the calculation formula is as follows:

[0097]

[0098]

[0099]

[0100]

[0101] Among them, [x a ,y a ,w a ,h a ] is the center point coordinates and width and height of the anchor, and [x, y, w, h] represents the corrected anchor coordinates. The calculation is as follows:

[0102]

[0103] Among them, [px, py, pw, ph] represents the coordinates of the original anchor, [dx, dy, dw, dh] represents the coordinate offset predicted by the RPN network, and [gx, gy, gw, gh] represents the corrected anchor coordinates.

[0104] Step 3: The proposal layer arranges all anchors from the largest to the smallest positive scores in step 1, takes the first 6000 anchors, removes the positive anchors with very small sizes, and gathers the positive anchors that exceed the boundary of the video frame to be detected to the boundary of the video frame to be detected, and corrects the coordinates of the anchors in step 2. Finally, the coordinates of the predicted bounding box of the target object are obtained.

[0105] 1404 , determining a predicted category of the target object according to the predicted bounding box of the target object, and performing predicted bounding box regression.

[0106] In some embodiments, the Rol Pooling layer collects the coordinates of the predicted bounding box of the target object generated by the RPN module, extracts it from the target feature map, and classifies all target objects into specific categories through the fully connected layer and softmax. At the same time, the bounding box regression is performed on the target object to obtain a more accurate predicted bounding box of the target object.

[0107] 1405 , calculating a first loss value according to the predicted category of the target object and the category label of the target object in the first sample, and calculating a second loss value according to the predicted bounding box of the target object and the bounding box of the target object in the first image.

[0108] 1406. Adjust the network model parameters according to the first loss value and the second loss value, and save the network parameters when the preset conditions are met.

[0109] In some embodiments, cross entropy can be used as the loss function for target classification to obtain a first loss value, and smooth L1 loss or Huber loss can be used as the loss function for bounding box regression to obtain a second loss value. Then, the model parameters are updated according to the first loss value and the second loss value through a back propagation algorithm and an optimizer (such as stochastic gradient descent) to minimize the loss function.

[0110] In some embodiments, when the model converges or reaches a preset number of training rounds, the network model parameters are saved.

[0111] Based on the same technical concept, see Fig.15 As shown, the embodiment of the present application provides a device 1500 for determining key video frames in a surgical event, and the device 1500 can perform any step in the method for determining key video frames in a surgical event, and will not be described here to avoid repetition. The device 1500 includes an acquisition module 1501, a detection module 1502, and a determination module 1503.

[0112] An acquisition module 1501 is used to acquire a video stream collected when a target surgical event is performed, wherein the video stream includes a plurality of video frames to be detected;

[0113] The detection module 1502 is used to perform target detection on each of the video frames to be detected, and determine the target object included in each of the video frames to be detected;

[0114] A determination module 1503 is used to determine a target surgical stage in the target surgical event corresponding to each of the video frames to be detected according to a feature map obtained from the object features in the video frames to be detected;

[0115] Determining the standard object corresponding to the target surgical stage based on the association relationship between the preset surgical stage and the standard object;

[0116] When it is determined that the target object is not included in the standard objects, the to-be-detected video frame is determined to be a key video frame when the target surgical event is performed.

[0117] In some embodiments, the detection module 1502, when performing target detection on each of the video frames to be detected and determining the target object included in each of the video frames to be detected, is specifically used to: perform target enhancement processing on the video frames to be detected to obtain a target enhancement feature map; and input the target enhancement feature map into a target detection model to determine the target object included in the video frames to be detected.

[0118] In some embodiments, the detection module 1502, when performing target enhancement processing on the video frame to be detected to obtain a target enhancement feature map, is specifically used to: perform feature extraction on the video frame to be detected to obtain a feature map of the video frame to be detected; perform background enhancement and target enhancement processing on the feature map of each video frame to obtain a background feature map and a target feature map; and perform weighted fusion of the background feature map and the target feature map to obtain a target enhancement feature map.

[0119] In some embodiments, when the detection module 1502 inputs the target enhancement feature map into the target detection model to determine the target object included in the video frame to be detected, it is specifically used to: perform feature extraction on the target enhancement feature map through the convolution layer in the target detection model to obtain a target feature map; input the target feature map into the region proposal module in the target detection model to determine the candidate region containing the target object; and input the region features in the target feature map corresponding to the candidate region into the classifier in the target detection model to determine the category of the target object included in the candidate region.

[0120] In some embodiments, the detection module 1502, when inputting the target feature map into the region proposal module in the target detection model to determine the candidate region including the target object, is specifically used to: set multiple anchor frames at the position of each feature value in the target feature map; determine the region category of the region where each anchor frame in the target feature map is located according to the feature value included in each anchor frame in the target feature map among the multiple anchor frames, the region category including the target region and the background region; when it is determined that the region where the anchor frame in the target feature map is located is the target region, use the region where the anchor frame in the target feature map is located as the candidate region including the target object.

[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0122] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0123] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0125] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for determining key video frames in a surgical event, characterized in that: include: Acquire a video stream collected when performing a target surgical event, wherein the video stream includes a plurality of video frames to be detected; Performing target detection on each of the to-be-detected video frames to determine the target object included in each of the to-be-detected video frames; Determining a target surgical stage in the target surgical event corresponding to each of the video frames to be detected according to a feature map obtained from the object features in the video frames to be detected; Determining the standard object corresponding to the target surgical stage based on the association relationship between the preset surgical stage and the standard object; When it is determined that the target object is not included in the standard objects, the to-be-detected video frame is determined to be a key video frame when the target surgical event is performed.

2. The method according to claim 1, characterized in that The performing target detection on each of the to-be-detected video frames to determine the target object included in each of the to-be-detected video frames includes: Performing target enhancement processing on the video frame to be detected to obtain a target enhancement feature map; The target enhancement feature map is input into a target detection model to determine the target object included in the video frame to be detected.

3. The method according to claim 2, characterized in that The performing target enhancement processing on the video frame to be detected to obtain a target enhancement feature map includes: Extracting features from the video frame to be detected to obtain a feature map of the video frame to be detected; Performing background enhancement and target enhancement processing on the feature map of each video frame to obtain a background feature map and a target feature map; The background feature map and the target feature map are weightedly fused to obtain a target enhanced feature map.

4. The method according to claim 2 or 3, characterized in that The step of inputting the target enhancement feature map into a target detection model to determine the target object included in the to-be-detected video frame comprises: The target enhanced feature map is subjected to feature extraction by a convolutional layer in the target detection model to obtain a target feature map; Inputting the target feature map into a region proposal module in the target detection model to determine a candidate region containing the target object; The regional features at the position corresponding to the candidate region in the target feature map are input into the classifier in the target detection model to determine the category of the target object included in the candidate region.

5. The method according to claim 4, characterized in that The step of inputting the target feature map into a region proposal module in the target detection model to determine a candidate region including the target object comprises: Setting a plurality of anchor frames at the position where each feature value is located in the target feature map; Determine a region category of a region where each anchor frame in the target feature map is located according to a feature value included in each anchor frame in the target feature map, wherein the region category includes a target region and a background region; When it is determined that the region where the anchor frame is located in the target feature map is the target region, the region where the anchor frame is located in the target feature map is used as a candidate region containing the target object.

6. A device for determining key video frames in a surgical event, characterized in that: include: An acquisition module, used to acquire a video stream collected when a target surgical event is performed, wherein the video stream includes a plurality of video frames to be detected; A detection module, used to perform target detection on each of the video frames to be detected, and determine the target object included in each of the video frames to be detected; A determination module, configured to determine a target surgical stage in the target surgical event corresponding to each of the video frames to be detected based on a feature map obtained from the object features in the video frames to be detected; Determining the standard object corresponding to the target surgical stage based on the association relationship between the preset surgical stage and the standard object; When it is determined that the target object is not included in the standard objects, the to-be-detected video frame is determined to be a key video frame when the target surgical event is performed.

7. The device according to claim 6, characterized in that The detection module, when performing target detection on each of the to-be-detected video frames to determine the target object included in each of the to-be-detected video frames, is specifically used to: Performing target enhancement processing on the video frame to be detected to obtain a target enhancement feature map; The target enhancement feature map is input into a target detection model to determine the target object included in the video frame to be detected.

8. The device according to claim 7, characterized in that The detection module, when performing target enhancement processing on the video frame to be detected to obtain a target enhancement feature map, is specifically used to: Extracting features from the video frame to be detected to obtain a feature map of the video frame to be detected; Performing background enhancement and target enhancement processing on the feature map of each video frame to obtain a background feature map and a target feature map; The background feature map and the target feature map are weightedly fused to obtain a target enhanced feature map.

9. An execution device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 5 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer program instructions, and when the computer program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 5.

Images