Operation stage and operation event identification method and device

Abstract

The embodiment of the invention provides a method and a device for identifying a surgical stage and a surgical event, which are used for solving the problem of inaccurate identification of the surgical event and the surgical stage in the prior art. The method comprises the following steps: aiming at each image frame of an image video stream in an operation process acquired by laparoscope equipment, respectively executing the following operations: carrying out target detection on the image frame, and identifying at least one target area; performing semantic segmentation on at least one target area in the image frame to obtain event description features; performing operation stage feature extraction on the image frame to obtain stage description features; performing feature fusion on the event description features and the stage description features to obtain fusion features; determining context features corresponding to the image frames according to the fusion features corresponding to the image frames and fusion features corresponding to N image frames adjacent to the image frames in the image video stream; and according to the context features, determining an operation stage and / or an operation event corresponding to the image frame.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for identifying surgical stages and surgical events. Background Technology

[0002] The medical process involves numerous surgical events and stages. Automatic identification of these events and stages facilitates precise location of surgical events and stages during and after surgery, as well as subsequent analysis. Current methods for identifying surgical stages and events primarily focus on feature classification within the current frame. However, due to unique factors during laparoscopic surgery, such as fogging, obstructed vision, bleeding, and instrument occlusion, feature extraction from the current frame may be inaccurate. This can lead to inaccurate identification of surgical events (such as bleeding, suturing, and cutting) and surgical stages (such as preparation and anesthesia) within the current frame. Summary of the Invention

[0003] This application provides a method and apparatus for identifying surgical stages and surgical events, in order to solve the problem of inaccurate identification of surgical events and surgical stages in the prior art.

[0004] In a first aspect, embodiments of this application provide an image processing method, including:

[0005] For each frame of the video stream captured by the laparoscopic equipment during the surgical procedure, perform the following operations:

[0006] Target detection is performed on the image frame to identify at least one target region in the image frame; semantic segmentation is performed on the at least one target region in the image frame to obtain event description features for describing the surgical event corresponding to the image frame; and surgical stage features are extracted for the image frame to obtain stage description features for the surgical stage corresponding to the image frame.

[0007] The event description features and the stage description features are fused to obtain fused features;

[0008] Based on the fusion features corresponding to the image frame and the fusion features corresponding to the N image frames adjacent to the image frame in the image video stream, the context features corresponding to the image frame are determined.

[0009] Based on the contextual features, the surgical stage and / or surgical event corresponding to the image frame are determined.

[0010] In one possible implementation, the method further includes:

[0011] For each image frame, anomaly frame identification is performed to determine the identification result for each image frame;

[0012] If the identification result indicates that the corresponding image frame is an abnormal frame, then the image frame is processed according to the abnormality type corresponding to the identification result of the image frame.

[0013] In one possible implementation, the semantic segmentation of the at least one target region in the image frame to obtain event description features for representing the surgical event corresponding to the image frame includes:

[0014] Semantic segmentation is performed on the at least one target region, and event feature regions in the image frame used to characterize surgical events are determined based on the semantic segmentation results;

[0015] The image information of the event feature regions is arranged according to the time order to obtain time-series image data that characterizes the distribution of event features based on the time order.

[0016] Based on the time-series image data, the event description features of the image frames are determined.

[0017] In one possible implementation, determining the surgical stage and surgical event corresponding to the image frame based on the context features includes:

[0018] Based on the contextual features, surgical stage identification is performed to determine the surgical stage corresponding to the image frame; and

[0019] Based on the context features, the surgical event features corresponding to the image frame are determined, and based on the surgical event features and the event description features, the surgical event corresponding to the image frame is determined.

[0020] In one possible implementation, determining the surgical event of the image frame based on the surgical event features and the event description features includes:

[0021] The event description features and the surgical event features are weighted to obtain weighted features;

[0022] Based on the weighted features, the surgical event corresponding to the image frame is determined.

[0023] In one possible implementation, the method further includes:

[0024] When it is determined that the video stream corresponds to multiple surgical stages, the target surgical stage that meets the stage adjustment conditions among the multiple surgical stages is adjusted.

[0025] The stage adjustment conditions include at least one of the following:

[0026] The time percentage of the target surgical stage is less than a set threshold, or the time of the surgical stage is less than a set duration;

[0027] The target surgical stage is the same for two adjacent surgical stages in time.

[0028] In one possible implementation, the method further includes performing the following operations for each image frame:

[0029] Invalid edge information is determined based on the image information of the video frame, and invalid edge cropping is performed on the video frame to obtain a cropped image of each video frame;

[0030] The contrast and grayscale of the cropped images of the image frames are adjusted to obtain the adjusted image frames.

[0031] Secondly, embodiments of this application provide an image processing apparatus, including:

[0032] The first processing module performs the following operations for each frame of the video stream captured by the laparoscopic equipment during the surgical procedure:

[0033] Target detection is performed on the image frame to identify at least one target region in the image frame; semantic segmentation is performed on the at least one target region in the image frame to obtain event description features for describing the surgical event corresponding to the image frame; and surgical stage features are extracted for the image frame to obtain stage description features for the surgical stage corresponding to the image frame.

[0034] The event description features and the stage description features are fused to obtain fused features;

[0035] Based on the fusion features corresponding to the image frame and the fusion features corresponding to the N image frames adjacent to the image frame in the image video stream, the context features corresponding to the image frame are determined.

[0036] Based on the contextual features, the surgical stage and / or surgical event corresponding to the image frame are determined.

[0037] In one possible implementation, the first processing module is further configured to:

[0038] For each image frame, anomaly frame identification is performed to determine the identification result for each image frame;

[0039] If the identification result indicates that the corresponding image frame is an abnormal frame, then the image frame is processed according to the abnormality type corresponding to the identification result of the image frame.

[0040] In one possible implementation, the first processing module, when performing semantic segmentation on at least one target region in the image frame to obtain event description features for representing the surgical event corresponding to the image frame, specifically performs the following:

[0041] Semantic segmentation is performed on the at least one target region, and event feature regions in the image frame used to characterize surgical events are determined based on the semantic segmentation results;

[0042] The image information of the event feature regions is arranged according to the time order to obtain time-series image data that characterizes the distribution of event features based on the time order.

[0043] Based on the time-series image data, the event description features of the image frames are determined.

[0044] In one possible implementation, the first processing module, when determining the surgical stage and surgical event corresponding to the image frame based on the context features, specifically performs the following:

[0045] Based on the contextual features, surgical stage identification is performed to determine the surgical stage corresponding to the image frame; and

[0046] Based on the context features, the surgical event features corresponding to the image frame are determined, and based on the surgical event features and the event description features, the surgical event corresponding to the image frame is determined.

[0047] In one possible implementation, the first processing module, when determining the surgical event of the image frame based on the surgical event features and the event description features, specifically performs the following:

[0048] The event description features and the surgical event features are weighted to obtain weighted features;

[0049] Based on the weighted features, the surgical event corresponding to the image frame is determined.

[0050] In one possible implementation, the apparatus further includes a second processing module, the second processing module being configured to:

[0051] When it is determined that the video stream corresponds to multiple surgical stages, the target surgical stage that meets the stage adjustment conditions among the multiple surgical stages is adjusted.

[0052] The stage adjustment conditions include at least one of the following:

[0053] The time percentage of the target surgical stage is less than a set threshold, or the time of the surgical stage is less than a set duration;

[0054] The target surgical stage is the same for two adjacent surgical stages in time.

[0055] In one possible implementation, the second processing module is further configured to: perform the following operations for each image frame:

[0056] Invalid edge information is determined based on the image information of the video frame, and invalid edge cropping is performed on the video frame to obtain a cropped image of each video frame;

[0057] The contrast and grayscale of the cropped image of the image frame are adjusted to obtain the adjusted image frame.

[0058] Thirdly, embodiments of this application provide an execution device, including:

[0059] Memory, used to store program instructions;

[0060] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect and different implementations of the first aspect according to the obtained program instructions.

[0061] Fourthly, embodiments of this application provide a computer-readable storage medium, the computer-readable storage medium including computer instructions, which, when executed by a computer, cause the computer to perform the methods described in the first aspect and different implementations of the first aspect.

[0062] The beneficial effects of this application are as follows:

[0063] In this application, when determining the surgical event and surgical stage corresponding to an image frame, the influence of the relationship between the surgical event and the surgical stage, as well as the temporal relationship, on the surgical stage and surgical event is fully considered. Contextual analysis is performed using the fused features of stage description features and event description features, and then the surgical event and surgical features are determined based on the contextual analysis results. This method can effectively improve the recognition accuracy of surgical events and surgical stages. Furthermore, in this application, abnormal frame identification is performed on the image frames before recognition. After identifying abnormal frames, abnormal frames are processed to effectively improve the recognition accuracy caused by fogging, unclear vision, or bleeding, further enhancing the recognition accuracy of surgical events and surgical stages. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0066] Figure 2A schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0067] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0068] Figure 4 A schematic diagram of a target detection result provided in an embodiment of this application;

[0069] Figure 5 A schematic diagram of a plurality of target regions provided in an embodiment of this application;

[0070] Figure 6 A schematic diagram of a region of interest provided in an embodiment of this application;

[0071] Figure 7 A schematic diagram illustrating a feature change provided in an embodiment of this application;

[0072] Figure 8 A schematic diagram illustrating feature fusion as provided in an embodiment of this application;

[0073] Figure 9 A schematic diagram of a dilated convolution provided for an embodiment of this application;

[0074] Figure 10 A schematic diagram of a fully connected layer provided in an embodiment of this application;

[0075] Figure 11 A network architecture diagram of Inception-Resnet-V2 provided for embodiments of this application;

[0076] Figure 12 A structural diagram of an Inception-resnet-A module provided in an embodiment of this application;

[0077] Figure 13 This is a schematic diagram of the structure of a ReductionA module provided in an embodiment of this application;

[0078] Figure 14 A structural diagram of an Inception-resnet-B module provided in an embodiment of this application;

[0079] Figure 15 A schematic diagram of the structure of a ReductionB module provided in an embodiment of this application;

[0080] Figure 16 A structural diagram of an Inception-resnet-C module provided in an embodiment of this application;

[0081] Figure 17A schematic diagram of a display interface provided in an embodiment of this application;

[0082] Figure 18 A schematic diagram of another display interface provided in an embodiment of this application;

[0083] Figure 19 A flowchart for identifying surgical stages and surgical events provided in this application embodiment;

[0084] Figure 20 A schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0085] Figure 21 This is a schematic diagram of an execution device provided in an embodiment of this application. Detailed Implementation

[0086] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0087] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0088] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] To facilitate understanding of the interactive medical sequence image segmentation method and device provided in the embodiments of this application, some terms used in the embodiments of this application will be explained below so that those skilled in the art can understand them.

[0090] (1) A convolutional neural network (CNN) is a type of feedforward neural network that includes convolutional computation and has a deep structure. CNNs are specifically designed to process data with a grid-like structure. A CNN consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. Convolutional layers extract features from the input image frames; pooling layers, also called pooling or subsampling layers, primarily perform feature selection, reducing the number of features and thus the number of parameters. Fully connected layers do not inherently extract features; instead, they cause the target feature map to lose its spatial topological structure and be unfolded into vectors. Their function is to non-linearly combine the extracted features to obtain the output result.

[0091] (2) Dilated Convolution, also known as dilated convolution, is a special convolutional operation used in convolutional neural networks. It increases the size of the receptive field by introducing spacing between convolutional kernels without increasing parameters or reducing resolution. In traditional convolution operations, the kernels are element-wise multiplied and summed with the input feature map to produce the output feature map. In dilated convolution, however, there is a fixed spacing between the elements of the kernels, which makes each element of the output feature map dependent only on a local region in the input feature map, while also expanding the receptive field of each kernel. The main advantage of dilated convolution is its ability to increase the network's receptive field without increasing parameters or computational cost, thereby improving the network's ability to grasp global information. Dilated convolution is very useful in many computer vision tasks, especially when dealing with large-scale objects or requiring broader contextual information.

[0092] (3) Multi-head attention mechanism is a technique used to enhance the ability of neural networks to model sequential data (such as natural language, time series, etc.). It extends the traditional attention mechanism, allowing the network to simultaneously focus on different subspaces of the input sequence to capture richer information. In traditional attention mechanisms, the network computes an attention weight vector, which is used to weight and sum the various parts of the input sequence to generate a context vector. In multi-head attention mechanism, the network learns multiple sets of attention weight vectors, each called an attention head. Each attention head focuses on different subspaces of the input sequence and generates a context vector. Finally, the outputs of multiple attention heads are concatenated or weighted and summed to obtain the final multi-head attention output. The advantage of multi-head attention mechanism is that it can learn multiple attention representations in parallel, thereby enhancing the network's ability to focus on different features. This helps improve the model's representational and generalization abilities, especially when dealing with complex sequential data.

[0093] (4) Surgical stages refer to different phases or staged behaviors during the surgical procedure, and are usually used to describe different steps or time periods of surgical operation. The division of surgical stages can be determined according to the type of surgery, surgical procedure, and the standards of the medical institution. For example, common surgical stages include the preparation stage, anesthesia stage, incision stage, surgical operation stage, and suturing stage.

[0094] (5) Surgical events refer to specific events or procedures during the surgical process, which may be a single step, action, or observation. Surgical events are usually key moments or behaviors in the surgical process that may affect the outcome of the surgery or the safety of the patient. For example, surgical events may include incision, hemostasis, organ repositioning, suturing, etc. Different stages of surgery may contain different types of surgical events, or they may contain the same type of surgical events.

[0095] The medical process involves numerous surgical events and stages. Automatic identification of these events and stages facilitates precise location of surgical events and stages during and after surgery, as well as subsequent analysis. Current methods for identifying surgical stages and events primarily focus on feature classification within the current frame. However, due to unique factors during laparoscopic surgery, such as fogging, obstructed vision, bleeding, and instrument occlusion, feature extraction from the current frame may be inaccurate. This can lead to inaccurate identification of surgical events (such as bleeding, suturing, and cutting) and surgical stages (such as preparation and anesthesia) within the current frame.

[0096] To address the aforementioned problems, this application provides an image recognition method and apparatus. It involves target detection of image frames in a video stream of surgical procedures acquired by a laparoscopic device to identify at least one target region; semantic segmentation of the at least one target region to obtain event description features representing surgical events; and semantic feature extraction of the image frames to obtain stage description features corresponding to the surgical stage. Further, the event description features and stage description features are fused to obtain fused features. Based on the fused features corresponding to the image frame and the fused features corresponding to N adjacent image frames, context features corresponding to the image frame are determined. Based on the context features, the surgical stage and surgical event corresponding to the image frame are determined. In this application, when identifying surgical events and surgical stages, the stage description features and event description features of the image frame are fused, and the context features are determined based on the fused features of adjacent image frames. The surgical stage and surgical event of the image frame are then determined using the context features. This application fully considers the interrelationship between surgical stages and surgical events, and takes into account the event continuity of surgical stages and events, thus effectively improving the accuracy of recognition. Furthermore, this application can effectively improve the processing speed by performing semantic segmentation based on at least one target region after target recognition.

[0097] The application scenarios of the diagnostic report generation method provided in the embodiments of this application are described below with reference to the accompanying drawings.

[0098] like Figure 1 As shown, an application scenario of an optional image processing method in this application includes a laparoscopic device 10 and a terminal device 11.

[0099] The laparoscopic device 10 is used to acquire video streams during laparoscopic surgery, the video streams include multiple image frames, and the video streams are sent to the terminal device 11.

[0100] Terminal device 11 is used to display the image video stream sent by laparoscopic device 10 on the display interface; in response to the user's operation of selecting the image video stream, it performs image processing on the image video stream to obtain the surgical events and surgical stages corresponding to each image frame in the selected image video stream; according to the surgical events and surgical stages corresponding to each image frame, it segments the image video stream to determine the image stream corresponding to each surgical stage and the image stream corresponding to each surgical event.

[0101] The terminal device 11 is also used to display the processing results of image processing on the display interface, so that the user can determine the time and operation of each surgical stage in the surgical process based on the image processing results for viewing and subsequent analysis; in response to the user's click operation to view a surgical event, the video stream corresponding to the surgical event is displayed on the display page.

[0102] The terminal device 11 in this application embodiment can be an electronic device including but not limited to smartphones, tablets, wearable electronic devices (such as smartwatches), laptops, etc.

[0103] In this application, the laparoscopic device 10 can acquire video streams during the surgical procedure and send the complete video streams to the terminal device 11, or it can send the acquired image frames to the terminal device 11 in real time. The terminal device 11 can perform image processing after receiving the complete video streams, or it can perform image processing on the image frames in real time; this application does not specifically limit this.

[0104] In the application scenarios of image processing methods, in addition to the laparoscopic device 10 and the terminal device 11, a server 12 may also be included, such as... Figure 2 As shown:

[0105] Terminal device 11 is used to receive the image and video stream sent by laparoscopic device 10; in response to the image processing operation initiated by the user for the image and video stream, it sends the image processing task and the image and video stream to server 12.

[0106] Server 12 performs image processing on the video stream according to the image processing task and sends the image processing results to terminal device 11. The image processing results include the surgical stage and surgical event corresponding to each image frame.

[0107] Terminal device 11 segments the image video stream based on the image processing results received from server 12. Specifically, it segments the image video stream according to the type of each surgical stage to obtain video streams corresponding to each surgical stage; it also segments the image video stream according to the type of each surgical event to obtain video streams corresponding to each surgical event, and displays them on the display interface; in response to the user's viewing operation for a surgical event (surgical stage), it plays the video stream corresponding to the surgical event (surgical stage) on the surgical interface.

[0108] In this embodiment, server 12 can be a single server, a server cluster consisting of several servers, or a cloud computing center. Server 12 can be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0109] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1-2The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application are not limited thereto.

[0110] See Figure 3 The diagram shown is a flowchart of an image processing method provided in an embodiment of this application. The specific steps are as follows:

[0111] In this application, for each image frame of the video stream during the surgical procedure acquired by the laparoscopic device, the following identical operations can be performed, as follows:

[0112] Step S301: Target detection is performed on the image frame to identify at least one target region in the image frame; semantic segmentation is performed on at least one target region in the image frame to obtain event description features for describing the surgical event corresponding to the image frame; and surgical stage features are extracted for the image frame to obtain stage description features for the surgical stage corresponding to the image frame.

[0113] In some scenarios, after receiving a video stream, rules can be set to process it. One possible implementation is to process the received video stream based on its frame rate. For example, if the received video stream has a frame rate of 25fps, the first frame can be extracted, and then a frame can be extracted every 24 frames to obtain the video frames to be processed. This processing effectively reduces the amount of data processed, thereby improving image processing speed.

[0114] In another possible implementation, after receiving the video stream, it can be processed according to time intervals. For example, taking a time interval of 1 second, one image frame can be extracted in the first second, and another image frame can be extracted in the second second. The extracted image frame can be the first image frame within that second, or any image frame within that second; this application does not impose a specific limitation on this. Furthermore, the time interval can also be shorter or longer than 1 second; this application does not impose a specific limitation on this either.

[0115] In some embodiments, target detection can be performed on the image frame to identify at least one target region within the image frame. Specifically, YOLO can be used for target detection of instruments related to surgical events. Unlike traditional instrument detection methods, this application does not detect the entire instrument, but rather the interface between the instrument and the organ. For example, when the image frame is using an electrosurgical unit for hemostasis, the target region detected is the interface between the electrosurgical unit and the organ, such as... Figure 4 As shown.

[0116] It is understandable that when an image frame includes multiple instruments related to a surgical event, the target detection result of the image frame may include multiple target regions.

[0117] As an example, such as Figure 5 As shown, if the image frame contains three instruments, then the target detection result of the image frame includes three target regions.

[0118] Furthermore, when performing semantic segmentation on at least one target region in an image frame to obtain event description features that represent the surgical event corresponding to the image frame, this can be achieved in the following way:

[0119] At least one target region is semantically segmented, and event feature regions for characterizing surgical events are determined based on the semantic segmentation results. Further, the image information of the event feature regions can be arranged based on a time series to obtain temporal image data for characterizing the distribution of time-series-based event features. Subsequently, event description features of image frames can be determined based on the temporal image data.

[0120] For example, a region of interest (ROI) can be determined based on the range of at least one target region, and then features can be extracted from the ROI to obtain an event feature region. As an example, when an image frame includes one target region, the target region can be directly used as the ROI, or a region with a radius of Y centered on the target region can be used as the ROI, and then semantic segmentation can be performed on the ROI to obtain the event feature region. In one embodiment, when multiple target regions exist, the region with the largest boundary value can be selected as the ROI based on the boundaries of the multiple target regions, and then semantic segmentation can be performed on the ROI to obtain the event feature region. For example, if an image frame includes three target regions, then... Figure 6 The region enclosed by the thick dashed line is the region of interest, which is then semantically segmented to obtain the event feature region.

[0121] After semantic segmentation, event feature regions representing surgical events, such as bleeding areas and suture lines, can be segmented from the region of interest. Then, the event description features of the surgical event corresponding to the image frame can be determined through the event feature regions.

[0122] Semantic segmentation can be achieved using the Unet network. During feature extraction, its feature changes are as follows: Figure 7 As shown.

[0123] Specifically, the image information of the event feature regions can be arranged according to time to obtain time-series image data. As an example, time-series image data can be represented as: [

[0125] [Area1 t1, Area2 t1, Area3 t1, Area4 t1, Area…t1,]

[0126] [Area1 t2, Area2 t2, Area3 t2, Area4 t2, Area…t2,]

[0127] [Area1 t3, Area2 t3, Area3 t3, Area4 t3, Area…t3,]

[0128] …]

[0129] Where Area n represents the event feature region, and tn represents the time.

[0130] Furthermore, event description features of image frames can be determined based on time-series image data. Specifically, the changes in the area of ​​a certain type of event feature region in the time-series image data can be plotted as a curve and processed using calculus to determine the surgical event features. For example, if Area 1 represents a bleeding area, and the area of ​​the bleeding area increases over time (i.e., the derivative is greater than 1), then a bleeding event is confirmed. As another example, after identifying the suture surgical event feature region, the change in its area from present to absent indicates the end of the suture event. Through the above methods, the event description features corresponding to the image frame can be determined.

[0131] In some embodiments, features of the surgical stages can be extracted from the image frame to determine the stage description features corresponding to the image frame. For example, if the surgery includes K surgical stages, the stage description features corresponding to the K surgical stages of the image frame can be determined by extracting features from the image frame. The stage description features are used to describe the surgical stages corresponding to the image frame. For example, if the surgery includes 7 surgical stages, the stage description feature K of the image frame can be represented as [0, 0.11, 0.04, 0, 0.69, 0.16, 0].

[0132] Step S302: The event description features and the stage description features are fused to obtain the fused features.

[0133] In some embodiments, event description features and stage description features can be fused to obtain fused features. Specifically, this can be achieved through methods such as... Figure 8 The network shown fuses event description features and stage description features to obtain fused features. For example, if the event description feature is represented as P and the stage description feature is represented as K, then the fused feature is K concatP, or simply KP.

[0134] Step S303: Determine the context features corresponding to the image frame based on the fusion features corresponding to the image frame and the fusion features corresponding to the N image frames adjacent to the image frame in the image video stream.

[0135] Where N is an even number, the N image frames adjacent to the image frame can be N / 2 image frames that are chronologically preceding the image frame and N / 2 image frames that are chronologically following the image frame. When N is an odd number, the N image frames adjacent to the image frame can be (N-1) / 2 image frames that are chronologically preceding the image frame and (N+1) / 2 image frames that are chronologically following the image frame; or, the N image frames adjacent to the image frame can also be (N+1) / 2 image frames that are chronologically preceding the image frame and (N-1) / 2 image frames that are chronologically following the image frame.

[0136] For example, when N is 4, the 4 image frames adjacent to the image frame can be the 2 image frames before the image frame and the 2 image frames after the image frame. When N is 5, the 5 image frames adjacent to the image frame can be the 2 image frames before the image frame and the 3 image frames after the image frame, or they can be the 3 image frames before the image frame and the 2 image frames after the image frame.

[0137] If the current image frame is the first image frame, it can be copied into multiple image frames and used as the image frames preceding it. The corresponding fusion feature is the fusion feature of the current image frame. Similarly, if the current image frame is the last image frame, it can be copied into multiple image frames and used as the image frames following it. The corresponding fusion feature is the fusion feature of the current image frame.

[0138] In some scenarios, the N image frames adjacent to the image frame can also be M1 image frames before the image frame and M2 image frames after the image frame, where M1 + M2 = N. The four image frames adjacent to the image frame can be the two image frames before the image frame and the two image frames after the image frame, or the one image frame before the image frame and the three image frames after the image frame, or the four image frames adjacent to the image frame can be either the four image frames before the image frame or the four image frames after the image frame. This application does not make specific limitations on this.

[0139] In this embodiment of the application, the contextual features of an image frame can be determined through a timing processing module, as follows:

[0140] The fusion features of one image frame and the fusion features of N image frames can be concatenated into a fusion feature sequence. The temporal processing module can then perform dilated convolution on this fusion feature sequence. Specifically, for a convolution kernel of size k, the mathematical expression of the dilated convolution function with input x is:

[0141]

[0142] Where s represents the input time-series information; d is the dilation parameter, i.e., the dilation interval size. By adding a dilated mechanism, the current causal convolutional network can expand its receptive field of view by increasing the kernel size or the dilation parameter without changing the network depth (number of layers). Figure 9 As shown.

[0143] The aforementioned temporal feature module fully considers the sequential dependencies between surgical stages (i.e., temporal causal relationships) and employs a causal convolutional neural network with dilated convolution mechanisms. After dilation through dilated convolution, the perceptual field of the next layer neurons in the causal neural network for historical data from the previous neurons is greatly expanded, which also improves the modeling ability of the causal convolutional network for time series inference tasks that require long-term memory.

[0144] In this embodiment, the number of layers and the number of hidden layers can be adjusted. The specific structure used is as follows: Figure 9 As shown. The value of M can be set according to specific circumstances; the value used in this scheme is 64. DilatedResidualCausal is the dilated convolution module (…). Figure 9 (The part inside the dashed box) Figure 9 In the middle, K represents a continuous multi-layer structure (e.g., Figure 9 In Figure (b), there are three layers on the right (K=3), which can be changed according to the actual situation. This application generally uses 8 or 12 layers. The specific number of layers and the expansion index can be adjusted according to the actual surgical type and scenario. In addition, various variants of TCN can also be used in this application to obtain the contextual features of the image frame.

[0145] Before using the network for processing, the convolutional network needs to be trained on data. This training should be conducted using pre-classified surgical videos to evaluate the recognition performance across different stages of the surgical procedure and determine relevant parameters. During training, the dataset can be further enriched by rotating and cropping the data.

[0146] Step S304: Determine the surgical stage and / or surgical event corresponding to the image frame based on contextual features.

[0147] In some embodiments, the surgical stage and / or surgical event corresponding to the image frame is determined based on the context features. Specifically, this can be achieved by: identifying the surgical stage based on the context features to determine the surgical stage corresponding to the image frame; determining the surgical event features of the image frame based on the context features; and determining the surgical event corresponding to the image frame based on the surgical event features and event description features.

[0148] Specifically, when determining the surgical event corresponding to an image frame based on surgical event features and event description features, the event description features and surgical event features can be weighted to obtain weighted features. Then, the surgical event corresponding to the image frame can be determined based on the weighted features.

[0149] As an example, the context features output by the temporal feature module can be passed through a fully connected layer to determine the surgical stage of an image frame, and then passed through another fully connected layer to determine the surgical event features of the image frame, such as... Figure 10 As shown. Then, the surgical event corresponding to the image frame is determined based on the surgical event features and event description features. Specifically, the surgical event features and event description features can be added or weighted to determine the surgical event corresponding to the image frame.

[0150] In some embodiments, after determining the context features, the context features KP can be further enhanced by inputting QKV into the Transformer module to determine the enhanced context features. Specifically, the context features KP' are processed through a multi-head attention mechanism, a residual network, and normalization, followed by a feed forward layer, a residual network, and normalization to obtain the enhanced context features KP''. Then, KP*Seq is input into the decoding module as the query value, and after the same operation, the enhanced context features are output, which can be represented as KP''. Furthermore, the enhanced surgical features KP'' are processed through two fully connected layers to output K' and P' respectively. Further, P' is weighted with P to output the final PF, where K' represents the surgical stage corresponding to the image frame, and PF represents the surgical event corresponding to the image frame.

[0151] In some embodiments, the following operations may also be performed before target recognition is performed on the image frame:

[0152] For each image frame, anomaly frame identification is performed to determine the identification result for each image frame. If the identification result indicates that the corresponding image frame is an anomaly frame, then the image frame is processed according to the anomaly type corresponding to the identification result of the image frame.

[0153] As an example, the Inception network can be used to identify abnormal frames in image frames, thereby determining the identification result corresponding to the image frame. The specific operation is as follows:

[0154] The specific operation is as follows, taking Inception-Resnet-V2 as an example, its network structure is as follows: Figure 11 As shown. This application is not limited to the Inception-Resnet-V2 network; any network capable of classification is acceptable, such as Resnet50, InceptionV1, 2, 3, etc.

[0155] 1. Input the current image: 299*299*3.

[0156] 2. After a 3x3 convolution, with a stride of 2, the output is 149*149*32.

[0157] 3. After a 3x3 convolution, with a stride of 1, the output is 147*147*32.

[0158] 4. After a 3x3 convolution, with a stride of 1, the output is 147*147*64.

[0159] 5. Perform max pooling and 3*3 convolution, output size 73*73*160.

[0160] 6. After multiple 1*1, 7*1, and 3*3 convolutions, the output is 71*71*192.

[0161] 7. Perform max pooling and 3*3 convolution, output size 35*35*384.

[0162] 8. Via the Inception-resnet-A module, such as Figure 12 As shown, the obtained feature map is 35*35*256.

[0163] 9. Features obtained through ReductionA Figure 17 *17*896, where the structure of ReductionA is as follows Figure 13 As shown.

[0164] 10. Features obtained through Inception-ResNet-B Figure 17 *17*896. The network structure of Inception-resnet-B is as follows: Figure 14 As shown.

[0165] 11. After ReductionB, the feature map is obtained as 8*8*1792. The structure of ReductionB is as follows: Figure 15 As shown.

[0166] 12. After passing through Inception-ResNet-C, the feature map is obtained as 8*8*1792. The network structure of Inception-ResNet-C is as follows: Figure 16 As shown.

[0167] 13. The pooling layer transforms the feature map into a 1*1*1792.

[0168] 14. The linear layer transforms 1*1*1792 into 1*1*M, where M is the number of categories to be identified.

[0169] 15. Softmax layer, with M categories, outputs 1*1*M, which is then processed to output a classification result of M.

[0170] In some scenarios, after passing through the aforementioned network, image frames can be identified into any one of M categories. The classification results can be labeled from 0 to N-1, where 0 represents a normal frame and the others are abnormal frames, such as 1 indicating fog, 2 indicating occlusion, and 3 indicating image blurring or jitter.

[0171] Furthermore, abnormal image enhancement and repair can be performed based on the abnormality type corresponding to the image frame. For example, histogram equalization can be performed on blurred images, dehazing can be performed on foggy images, and dehazing and histogram equalization can be performed based on the type of abnormal frame. The histogram equalization uses the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm, and the dehazing algorithm uses the CLAHE+Retinex algorithm.

[0172] As an example, when processing blurry images using the CLAHE algorithm, it can be implemented in the following way:

[0173] 1) Divide the image frame into multiple image blocks, each image block having a size of N×N;

[0174] 2) Perform histogram equalization on each image block to make the pixel value distribution of each image block more uniform;

[0175] 3) Constrain the equalized pixels. Specifically, for each pixel, calculate the average and standard deviation of that pixel relative to its surrounding pixels. If the pixel's value exceeds the average plus a constraint factor multiplied by the standard deviation, then the pixel's value is set to the average plus the constraint factor multiplied by the standard deviation.

[0176] 4) Stitch all the image blocks together to form an image.

[0177] Furthermore, image frames can be replaced with processed image frames to facilitate target detection. For example, multiple image frames can be represented as F1F2F3..., where if there is an abnormal frame, it is replaced with a processed image frame. For example, if the third image frame is an abnormal frame, it is replaced with the processed image frame F3', that is, F1F2F3 is corrected to F1F2F3'.

[0178] In some embodiments, before performing abnormal frame identification on the image frames, the following operations are performed for each image frame:

[0179] Invalid edge information is determined based on the image information of the image frame, and invalid edge is cropped from the image frame to obtain a cropped image of the image frame; the contrast and grayscale of the cropped image of the image frame are adjusted to obtain an adjusted image frame.

[0180] Specifically, in the real-time video stream of a laparoscopic endoscopy, there may be invalid black edges at the periphery. Invalid regions around these black edges can be cropped. Furthermore, the image size can be adjusted to fit subsequent models. For example, the image can be resized to X*X*3 (X can be standard sizes such as 299, 512, 640, etc.). Further, contrast, grayscale values, and uniformization operations can be adjusted to highlight the feature information of the image frames. In addition, other empirical or adaptive preprocessing operations, such as filtering and noise reduction, can be performed, but this application does not specifically limit these operations.

[0181] In some embodiments, when it is determined that the video stream corresponds to multiple surgical stages, the target surgical stage that meets the stage adjustment conditions among the multiple surgical stages is adjusted. The stage adjustment conditions include at least one of the following: the time percentage of the target surgical stage is less than a set threshold, or the time of the surgical stage is less than a set duration; or two surgical stages that are temporally adjacent to the target surgical stage are identical.

[0182] In some embodiments, after determining the surgical event and surgical stage corresponding to each image frame in the video stream, the surgical stages and surgical events are arranged in chronological order. Furthermore, considering the characteristics of surgery, the surgical stages and surgical events are continuous, so they can be corrected by a post-processing module. The specific operation is as follows:

[0183] For example, anomaly identification, removal, and correction can be performed through a post-processing module. Since surgical phases are continuous, if a situation suddenly appears in a series of consecutive image frames that does not conform to other image frames, it is removed and corrected. For example, in the surgical phase sequence A..AAAAAAAAABAAAAAAAAAAAAA..A, it is impossible for a surgical phase to last continuously for milliseconds or seconds. Therefore, B is identified as an anomaly and corrected, replacing B with A.

[0184] In some scenarios, post-processing modules can be used for non-cavity processing. That is, if an image frame shows that the laparoscopic device has not entered or has exited the cavity, an error marker is deleted. During laparoscopic surgery, there are situations where the endoscope needs to be removed from the body. In this stage, the system identifies whether the device is inside or outside the cavity, and for times outside the cavity, the stage marker is deleted and replaced with an outside-cavity marker.

[0185] In some embodiments, the post-processing module can sort the identification results of surgical stages and events corresponding to each image frame in time, and smooth the surgical stages and / or events along the time axis of the video stream. Since a surgical stage or event cannot be a mere one-second occurrence, and considering the special nature of surgery, there will not be extremely short surgical stages or events, the data for surgical stages and events is filtered along the time axis to remove glitches. The filtering method involves mean filtering of the corresponding values ​​for each stage on the time axis, using a filter slider of size L for time axis filtering to eliminate glitches and thus improve accuracy.

[0186] In some scenarios, an abnormal phase removal module can be added to the post-processing module to perform deletion operations on modules with extremely short time fluctuations. This module can greatly eliminate glitch and jitter phenomena.

[0187] For example, regarding the changes in surgical stages A1, B1, A2, B2 (AB represents two stages, i.e., A1 and A2 are the same stage, and B1 and B2 are the same stage), if A2 has a very short duration and a small proportion (e.g., A2 accounts for less than the set threshold of the total surgical time, i.e., (A2 duration / total duration < set threshold and B1 = B2)), then A2 will be corrected to stage B, thereby effectively eliminating abnormal modules.

[0188] For example, if a certain surgical stage has a very short duration, less than the set duration (i.e., less than M frames), it will be corrected, and the correction value will be referenced to the stages before and after it.

[0189] For example, regarding the surgical event changes A1, A2, A1, if the duration of A2 is extremely short and its proportion is very small (the proportion of the duration of the second A1 and the first A1 is less than the set threshold), then the surgical event A2 will be corrected to the surgical event A1.

[0190] In practical applications, after determining the surgical stage and surgical event for each image frame in the video stream, the surgical events and stages can be recorded and labeled. Then, based on the labeled surgical events and stages, the time and duration of the surgical events and stages are recorded and statistically analyzed, allowing operators to review the image playback of specific surgical events and stages.

[0191] As an example, if a bleeding event is detected, the button in the upper right corner of the display screen will flash. The operator can click it to view the specific surgical event, such as bleeding, and the corresponding surgical stage's retrospective. Figure 17 As shown.

[0192] For example, operators can click the "view" button for the corresponding surgical event and stage to directly review the video footage at that specific time point. Figure 17 As shown.

[0193] Furthermore, postoperatively, the system allows for quick navigation to the corresponding video stream, showing the percentage of bleeding time and the relevant index, facilitating surgical review by the operator. Figure 18 As shown.

[0194] The image processing method of this application will be illustrated below with examples, and the specific process is as follows: Figure 19 As shown, the specific steps are as follows:

[0195] Step S1901: Preprocess each image frame in the video stream to obtain an image frame sequence.

[0196] Specifically, the image frames acquired through laparoscopic equipment can be wirelessly cropped at their outer edges. The cropped image frames can then be resized to 299×299×3 pixels. Contrast and grayscale values ​​can then be adjusted to highlight key features within the image frames. Furthermore, filtering or noise reduction operations can be performed, and the image frames can be sorted chronologically to obtain the image frame sequence F1F2F3F4…Fn.

[0197] Step S1902: Identify abnormal frames in the image frame sequence, perform abnormal processing on the abnormal frames, and obtain the abnormal processed image frame sequence.

[0198] Specifically, features of each image frame can be extracted through multi-layer convolution operations, and abnormal frame identification and classification can be performed. When an image frame is an abnormal frame, it is processed, and the abnormal frame in the image frame sequence is replaced with the processed image frame. For example, if the image frame sequence is F1F2F3F4……Fn, and the third image frame is an abnormal frame, then after the third image frame is processed, the resulting image frame sequence is F1F2F3'F4……Fn.

[0199] Step S1903: Extract features from each image frame in the image frame sequence to determine the stage description features of the surgical stage for each image frame.

[0200] In this step, the structure of the feature extraction network can be the same as that used in step S1902. The goal is to output the stage description features corresponding to the image frames, i.e., the classification results of the surgical stages. For example, if the number of surgical stages is K, then the output of the feature extraction network is the classification results for the K surgical stages.

[0201] Step S1904: Perform target detection on each image frame in the image frame sequence and determine the target detection result corresponding to each image frame.

[0202] Specifically, the YOLO network can be used for target detection during surgical events to determine the target area. The target area is the junction between the instrument and the organ.

[0203] Step S1905: Determine the region of interest based on the target detection results.

[0204] In some scenarios, the target detection results may contain multiple target regions. In this case, the bounding rectangles of multiple target regions can be used as regions of interest.

[0205] Step S1906: Perform semantic segmentation on the region of interest to determine the event description feature sequence.

[0206] Specifically, semantic segmentation can be used to segment feature regions such as the initial learning region and the seam line from the region of interest, and these regions can be formed into a time series, as shown below: [

[0208] [Area1 t1, Area2 t1, Area3 t1, Area4 t1, Area…t1,]

[0209] [Area1 t2, Area2 t2, Area3 t2, Area4 t2, Area…t2,]

[0210] [Area1 t3, Area2 t3, Area3 t3, Area4 t3, Area…t3,]

[0211] …].

[0212] For example, if Area1 is a bleeding area, then its area increases over time, i.e., the derivative is greater than 1, which indicates bleeding. Another example is the identification of a suture line, where its area changes from present to absent over time, indicating the end of the suturing event. By using the above methods, preliminary event prediction is made, generating PO*Seq (event type is P, PO*Seq is used to represent the event distribution based on time series).

[0213] Step S1907: The event description feature sequence and the stage description feature sequence are fused to obtain the fused feature sequence.

[0214] In some embodiments, the event description feature sequence PO*Seq and the stage description feature sequence K*Seq can be fused to obtain the fused feature sequence (K concat P)*Seq, abbreviated as KP*Seq.

[0215] Step S1908: Perform dilated convolution on the fused feature sequence to determine the context feature sequence.

[0216] The specific steps for this process can be found in the section on determining contextual features described above, and will not be repeated here. The contextual feature sequence can be represented as KP*Seq'.

[0217] Alternatively, a Transformer model can be used to extract features from the aforementioned KP*Seq' using a multi-head attention mechanism, further extracting the contextual feature sequence. For example, KP*Seq' can be input into the Transformer module as a QKV value, passing through a multi-head attention mechanism, residual network, normalization, feedback layer, residual network, and normalization to obtain KP*Seq". Then, KP*Seq can be input into the decoding module as a query value, and the same operation can be performed to obtain the output KP*Seq"'.

[0218] Step S1909: Identify the surgical stage and surgical events based on the context feature sequence, and determine the surgical event and surgical stage corresponding to each image frame.

[0219] For example, KP*Seq' (or KP*Seq"') can be passed through two fully connected layers to output K'*Seq and P'*Seq respectively. P'*Seq and PO*Seq are added (or weighted) to output the final PF*Seq. At this time, K'*Seq is the recognition result of the surgical stage of the image frame (such as the start of surgery, incision, etc.), and PF*Seq is the recognition result of the surgical event of the image frame (such as surgical bleeding, achievement of surgical safety standards, etc.).

[0220] Step S1910: Based on the identification results of the surgical stage and surgical events in the image frame, perform the elimination and correction of abnormal stages and abnormal events.

[0221] Based on the continuity of surgical stages and events, unreasonable identification results can be eliminated and corrected. For details, please refer to the aforementioned procedures; they will not be repeated here.

[0222] Based on the same technical concept, see [link / reference] Figure 20As shown, this application provides an image processing apparatus 2000. This apparatus 2000 can perform any step of the image processing method described above; to avoid repetition, it will not be described again here. The apparatus 2000 includes a first processing module 2001 and a second processing module 2002.

[0223] The first processing module 2001 performs the following operations for each image frame of the video stream acquired by the laparoscopic equipment during the surgical procedure:

[0224] Target detection is performed on the image frame to identify at least one target region in the image frame; semantic segmentation is performed on the at least one target region in the image frame to obtain event description features for describing the surgical event corresponding to the image frame; and surgical stage features are extracted for the image frame to obtain stage description features for the surgical stage corresponding to the image frame.

[0225] The event description features and the stage description features are fused to obtain fused features;

[0226] Based on the fusion features corresponding to the image frame and the fusion features corresponding to the N image frames adjacent to the image frame in the image video stream, the context features corresponding to the image frame are determined.

[0227] Based on the contextual features, the surgical stage and / or surgical event corresponding to the image frame are determined.

[0228] In one possible implementation, the first processing module 2001 is further configured to: identify abnormal frames for each image frame and determine the identification result of each image frame; if the identification result indicates that the corresponding image frame is an abnormal frame, then perform abnormal processing on the image frame according to the abnormal type corresponding to the identification result of the image frame.

[0229] In one possible implementation, the first processing module 2001, when performing semantic segmentation on at least one target region in the image frame to obtain event description features for representing the surgical event corresponding to the image frame, specifically performs: semantic segmentation on the at least one target region; determines the event feature region in the image frame for characterizing the surgical event based on the semantic segmentation result; arranges the image information of the event feature region according to time order to obtain temporal image data for characterizing the distribution of event features based on time order; and determines the event description features of the image frame based on the temporal image data.

[0230] In one possible implementation, the first processing module 2001, when determining the surgical stage and surgical event corresponding to the image frame based on the context features, is specifically used for:

[0231] Based on the contextual features, surgical stage identification is performed to determine the surgical stage corresponding to the image frame; and

[0232] The surgical event features corresponding to the image frame are determined based on the context features; the surgical event corresponding to the image frame is determined based on the surgical event features and the event description features.

[0233] In one possible implementation, the first processing module 2001, when determining the surgical event of the image frame based on the surgical event features and the event description features, is specifically configured to: weight the event description features and the surgical event features to obtain weighted features; and determine the surgical event corresponding to the image frame based on the weighted features.

[0234] In one possible implementation, the second processing module 2002 is configured to:

[0235] When it is determined that the video stream corresponds to multiple surgical stages, the target surgical stage that meets the stage adjustment conditions among the multiple surgical stages is adjusted.

[0236] The stage adjustment conditions include at least one of the following:

[0237] The time percentage of the target surgical stage is less than a set threshold, or the time of the surgical stage is less than a set duration;

[0238] The target surgical stage is the same for two adjacent surgical stages in time.

[0239] In one possible implementation, the second processing module 2002 is further configured to: perform the following operations for each image frame:

[0240] Invalid edge information is determined based on the image information of the video frame, and invalid edge cropping is performed on the video frame to obtain a cropped image of each video frame;

[0241] The contrast and grayscale of the cropped image of the image frame are adjusted to obtain the adjusted image frame.

[0242] Based on the same inventive concept, this application provides an electronic device 2100, which can realize the functions of the image processing device described above. Please refer to... Figure 21 The device includes a processor 2101 and a memory 2102.

[0243] Memory 2101 is used to store program instructions;

[0244] The processor 2102 is used to call program instructions stored in the memory and execute any step of the above image processing method according to the obtained program instructions.

[0245] In this embodiment, the processor 2102 is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and routes. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 2101, and by calling data stored in the memory 2101. Optionally, the processor 2102 may include one or more processing units. The processor 2102 may be, for example, a processor, microprocessor, controller, or other control component. It may be a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0246] The memory 2101 can be used to store software programs and modules. The processor 2102 executes various functional applications and data processing by running the software programs and modules stored in the memory 2101. The memory 2101 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function, etc.; the data storage area can store data created according to business processing, etc. As a non-volatile computer-readable storage medium, the memory 2101 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 2101 can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, disk, optical disk, etc. The memory 2101 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 2101 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0247] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform any of the image processing methods discussed above. Since the principle by which the computer-readable storage medium solves the problem is similar to that of the image processing method, the implementation of the computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0248] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute the image processing method described above. Since the principle by which the above computer program product solves the problem is similar to that of the image processing method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0249] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0250] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0251] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0252] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0253] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, The method includes: For each frame of the video stream captured by the laparoscopic equipment during the surgical procedure, perform the following operations: Target detection is performed on the image frame to identify at least one target region in the image frame; semantic segmentation is performed on the at least one target region in the image frame to obtain event description features for describing the surgical event corresponding to the image frame; and surgical stage features are extracted for the image frame to obtain stage description features for the surgical stage corresponding to the image frame. The event description features and the stage description features are fused to obtain the fused features; Based on the fusion features corresponding to the image frame and the fusion features corresponding to the N image frames adjacent to the image frame in the image video stream, the context features corresponding to the image frame are determined. Based on the contextual features, the surgical stage and / or surgical event corresponding to the image frame are determined.

2. The method as described in claim 1, characterized in that, The method further includes: For each image frame, anomaly frame identification is performed to determine the identification result for each image frame; If the identification result indicates that the corresponding image frame is an abnormal frame, then the image frame is processed according to the abnormality type corresponding to the identification result of the image frame.

3. The method as described in claim 1, characterized in that, The step of semantically segmenting at least one target region in the image frame to obtain event description features for representing the surgical event corresponding to the image frame includes: Semantic segmentation is performed on the at least one target region, and event feature regions in the image frame used to characterize surgical events are determined based on the semantic segmentation results; The image information of the event feature regions is arranged according to the time order to obtain time-series image data that characterizes the distribution of event features based on the time order. Based on the time-series image data, the event description features of the image frames are determined.

4. The method as described in claim 1, characterized in that, The step of determining the surgical stage and / or surgical event corresponding to the image frame based on the context features includes: Based on the contextual features, surgical stage identification is performed to determine the surgical stage corresponding to the image frame; and The surgical event features of the image frame are determined based on the context features; the surgical event corresponding to the image frame is determined based on the surgical event features and the event description features.

5. The method as described in claim 4, characterized in that, Determining the surgical event of the image frame based on the surgical event features and the event description features includes: The event description features and the surgical event features are weighted to obtain weighted features; Based on the weighted features, the surgical event corresponding to the image frame is determined.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: When it is determined that the video stream corresponds to multiple surgical stages, the target surgical stage that meets the stage adjustment conditions among the multiple surgical stages is adjusted. The stage adjustment conditions include at least one of the following: The time percentage of the target surgical stage is less than a set threshold, or the time of the surgical stage is less than a set duration; The target surgical stage is the same for two adjacent surgical stages in time.

7. The method as described in claim 2, characterized in that, The method further includes: For each image frame, perform the following operations: Invalid edge information is determined based on the image information of the image frame, and invalid edge cropping is performed on the image frame to obtain the cropped image of the image frame; The contrast and grayscale of the cropped image of the image frame are adjusted to obtain the adjusted image frame.

8. An image processing apparatus, characterized in that, include: The processing module performs the following operations for each frame of the video stream acquired by the laparoscopic equipment during the surgical procedure: Target detection is performed on the image frame to identify at least one target region in the image frame; and semantic segmentation is performed on the at least one target region in the image frame to obtain event description features for describing the surgical event corresponding to the image frame; In addition, surgical stage features are extracted from the image frames to obtain stage description features of the surgical stage corresponding to the image frames. The event description features and the stage description features are fused to obtain the fused features; Based on the fusion features corresponding to the image frame and the fusion features corresponding to the N image frames adjacent to the image frame in the image video stream, the context features corresponding to the image frame are determined. Based on the contextual features, the surgical stage and / or surgical event corresponding to the image frame are determined.

9. An execution device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1-7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-7.

Images