ConvNeXt-based multi-stage adaptive class balance operation stage identification method

Abstract

The invention relates to a ConvNeXt-based multi-stage adaptive class balance operation stage identification method, which comprises the following steps of: firstly, preparing a data set, and pre-training the weight of a model of ConvNeXt on an Imagenet-22k large data set; then carrying out data preprocessing; training an image feature extraction module by using the data to obtain a BLConvNeXt model weight; training a time sequence feature module by using data to obtain a multilevel adaptive time convolution module weight; and finally, loading a model training weight, and completely operating the operation stage identification model to obtain a final prediction result. According to the method, the recognition capability of the model on few sample categories can be enhanced through mutual learning of high-dimensional image features among the same batch of images; the overall recognition result accuracy in the few-sample stage and the recognition result accuracy in the difficult-to-classify image stage are improved; the whole method is good in generalization, and the stage of the current operation picture can be efficiently and accurately recognized in different types of operations; the recognition effect of the model on the stage with the small sample size is improved, and therefore the recognition result of the whole operation stage is optimized.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and pattern recognition, and in particular relates to a multi-stage adaptive category-balanced surgical stage recognition method based on ConvNeXt. Background Art

[0002] Automatic surgical stage recognition is an important task in the field of computer-assisted intervention, which aims to objectively identify surgical stages from recorded surgical videos. This technology can help doctors monitor the operation during surgery and issue alarms before dangerous problems occur. It can also be used to record surgical reports, surgeon skill assessment and training after surgery, which is crucial to improving surgeon efficiency and patient safety.

[0003] In reality, the duration of each stage varies greatly due to the different task objectives of different surgical stages. For example, the fifth stage of ultrasonic cataract surgery is to use ultrasonic energy to break up the cloudy lens into small particles and remove them through incisions. This stage is the focus of ultrasonic cataract surgery, so the operation time accounts for the longest proportion. The first stage takes up less time because the surgical goal is relatively simple. This situation leads to the natural imbalance of data distribution in the surgical stage identification dataset.

[0004] The uneven distribution of data during the surgical stage will lead to the following two problems: 1. The image feature extraction module is difficult to extract the difference features of the image in the few-sample stage, so more misclassified images will appear in the few-sample stage. 2. The temporal feature extraction module will be disturbed by the features of the misclassified images, which will affect the temporal feature fitting and the overall stage recognition results.

[0005] The document with application number "202410441127.0" discloses "Surgical stage identification method and device based on global surgical stage positioning". First, the surgical video to be identified is obtained, and the features of the surgical video are extracted to obtain the video features; based on the video features, the surgical video is subjected to regional detection on the video timeline to obtain the various surgical stages and surgical video clips contained in the surgical video. The problem is that it does not take into account that the goals of each surgical stage are different and the duration of each surgical stage is different, which makes it difficult to identify the stage with too short a duration. Summary of the invention

[0006] The present invention provides a multi-stage adaptive category-balanced surgical stage recognition method based on ConvNeXt to overcome the problem in the prior art that it is difficult to recognize a stage with too short a surgical duration in surgical stage recognition.

[0007] In order to achieve the above object, the technical solution of the present invention is: a multi-stage adaptive category-balanced surgical stage recognition method based on ConvNeXt, the method comprising the following steps:

[0008] Step 1. Data preparation: prepare the dataset and the model pre-training weights of ConvNeXt on the Imagenet-22k large dataset;

[0009] Step 2: Data preprocessing: down-sampling the data set and adjusting the resolution to be consistent;

[0010] Step 3: Use the data to train the image feature extraction module to obtain the BLConvNeXt model weights, specifically:

[0011] 3.1 Construction of image feature module: The image feature extraction module uses the ConvNeXt model as the backbone network, uses ZeroLoss to replace the cross entropy loss in the ConvNext model, and adds a batch sample learning module after the average pooling layer of the ConvNext model;

[0012] 3.2 Use data to train the image feature extraction module: During training, use the ConvNeXt model pre-training weights on the Imagenet-22k large dataset, and fine-tune the model parameters on the Cho lec80 and Cataract-101 datasets. The BLConvNeXt training is performed for a total of 30 rounds, and finally the BLConvNeXt model weights with the highest recognition accuracy are retained;

[0013] Step 4: Use the data to train the time series feature module to obtain the weights of the multi-level adaptive time convolution module;

[0014] Step 5: Load the model training weights, run the surgical stage recognition model completely, and obtain the final prediction results:

[0015] The complete surgical stage recognition network model consists of an image feature extraction module and a temporal feature extraction module.

[0016] Furthermore, the specific steps of the above step 4 are:

[0017] 4.1 Construction of the temporal feature module: The structure of the temporal feature extraction module is to add a self-attention module to the causal convolution before two different step lengths, and perform an add operation on the temporal features extracted by the two causal convolutions after the two causal convolutions with the same dimension superimposed. Finally, the prediction result is fine-tuned through normal causal convolution to obtain the final surgical stage recognition result.

[0018] 4.2 Temporal feature module training: The trained BLConvNeXt weights are fixed, the last fully connected layer is removed, and the training is performed in the target dataset to obtain image features. After obtaining the image features, the final surgical stage recognition results are obtained through multi-level adaptive temporal convolution. The multi-level adaptive convolution module is trained for a total of 30 rounds, and the weights of the multi-level adaptive temporal convolution module with the highest recognition accuracy are retained;

[0019] Furthermore, the specific steps of the above step five are: loading the saved BLConvNext module and the multi-level adaptive time convolution module weights, and applying the trained complete surgical stage recognition network model to the Cho lec80 dataset or the Cataract-101 dataset to obtain the final prediction result, that is, the surgical stage recognition result of the dataset.

[0020] Furthermore, in 3.1 above, the process of using ZeroLoss to replace the cross entropy loss in the ConvNext model is: first add Hard Number as the category balance weight of the cross entropy loss function,

[0021] Then add the balance parameter α, and the Hard Number formula is shown in (1):

[0022]

[0023] where p t is the overall recognition accuracy of the image at that stage, and p c is the accuracy rate of the stage recognition result of the image;

[0024] Finally, add the random optimization loss β, and the complete ZeroLoss formula is shown in (2), where Loss is the normal cross entropy loss:

[0025]

[0026] Furthermore, in the above step 3.1, the batch sample learning module consists of two Batch Learning layers and an enhanced residual structure. The Batch Learning layer includes multi-head self-attention (MSA) and MLP blocks, and Layernorm (LN) is used after each layer; at the same time, the features before and after the two Batch Learning layers are superimposed.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1. The method of the present invention proposes a batch sample learning module, in which a multi-head self-attention module and a residual structure are added, so that the high-dimensional image features can be learned from each other among the same batch of images to enhance the model's recognition ability for small sample categories;

[0029] 2. The method of the present invention designs a new loss function, namely, ZeroLoss, to make the model more inclined to learn images that are difficult to classify and have a small number of samples, thereby improving the overall recognition result accuracy of the few-sample stage and the recognition result accuracy of the difficult-to-classify image stage.

[0030] 3. The present invention proposes a new temporal feature extraction module. When extracting temporal features, it avoids fitting image features that are easy to misclassify and fits high-quality temporal features, so that the overall method has good generalization and can efficiently and accurately identify the stage of the current surgical screen in different types of surgeries;

[0031] 4. The method of the present invention aims at the problem of data imbalance in different stages of surgical stage recognition. Without changing the data sampling method, the method is optimized from three aspects: enhancing the recognition ability of few-sample category images, category cost-sensitive learning, and adaptive time series feature fitting method. By improving the recognition effect of few-sample surgical stages, the overall surgical recognition accuracy is effectively improved. Among them, by improving the model structure and designing a new balanced loss function, the recognition effect of the model for the stage with a small sample size is improved, thereby optimizing the overall surgical stage recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of a surgical stage identification process in an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the method structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below by specific implementation methods. It should be understood by those skilled in the art that the embodiments are only to help understand the present invention and should not be regarded as specific limitations of the present invention.

[0035] See also Figure 1 , a multi-stage adaptive category-balanced surgical stage recognition method based on ConvNeXt, the method comprising the following steps:

[0036] Step 1: Data preparation: Prepare the Cataract-101 dataset and the Cho lec80 dataset, and prepare the model pre-training weights of ConvNeXt on the Imagenet-22k large dataset;

[0037] The Cataract-101 dataset consists of 101 ultrasonic cataract surgery videos. The dataset defines the surgical stages as 10 stages. Each video is shot at a frame rate of 25fps per second, and the resolution of these frames is 720×540. The Cho lec80 dataset contains 80 surgical videos of cholecystectomy performed by 13 surgeons. The dataset defines the surgical stages as 7 stages. Each surgical video is shot at a frame rate of 25fps, and the resolution of these frames is 1920*1080 and 854*480.

[0038] Step 2: Data preprocessing:

[0039] The videos in the Cataract-101 dataset and Cho lec80 dataset are downsampled from 25fps to 1fps, and the resolution of the original frame images is adjusted to 250×250;

[0040] Step 3: See Figure 2 , use the data to train the image feature extraction module and obtain the BLConvNeXt model weights; the specific steps are:

[0041] 3.1 Construction of image feature feature module: The image feature extraction module (BLConvNeXt) uses the ConvNeXt model as the backbone network, uses cost-sensitive loss instead of the cross-entropy loss in the ConvNext model, and adds a batch sample learning module after the average pooling layer of the ConvNext model;

[0042] (1) Because the Softmax loss used in ConvNeXt ignores the class imbalance of data size and tends to produce uneven gradients of different classes, it is not conducive to the model's recognition of few-sample categories. Therefore, in order to further enhance the model's attention to images of few-sample categories, we designed a new category cost-sensitive loss (zero loss) that uses category prediction accuracy to weight the loss and rebalance the uneven gradients between classes. ZeroLoss is based on cross-entropy loss. The process of using ZeroLoss to replace the cross-entropy loss in the ConvNext model is:

[0043] First, we add Hard Number as the category balance weight of the cross entropy loss function, which increases the loss of images with high recognition difficulty and small sample size, and reduces the recognition loss of easy-to-classify sample images, so that the model pays more attention to difficult-to-classify samples.

[0044] Using only the Hard Number as the category balance weight will result in too large a proportion of loss of difficult-to-classify samples. Therefore, a balance parameter α is added to avoid an imbalance in the proportion of loss of small sample sizes, thereby improving the model recognition accuracy.

[0045] The obtained Hard Number formula is shown in (1):

[0046]

[0047] where p t is the overall recognition accuracy of the image at that stage, and p c This is the accuracy rate of the stage recognition result of the image. This can increase the loss value weight of images with few samples and difficult to classify, thereby achieving the effect of strengthening the learning of such images.

[0048] Since categories with a small number of samples are prone to local overfitting in the recognition process, a random optimization loss β is added at the end to allow the model to jump out of the local optimal point and perform random optimization when it falls into the local optimal point, so as to find a better solution. The overall loss function formula is shown in (2), where Loss is the normal cross entropy loss:

[0049]

[0050]

[0051] Considering that for models with larger parameters such as ConvNeXt-B, the model will not fully learn images of the minority category due to the small proportion of images of the minority category;

[0052] (2) Add a batch sample learning module after the average pooling layer of the ConvNext model:

[0053] The batch sample learning module provided by the present invention consists of two Batch Learning layers and an enhanced residual structure. The Batch Learning layer includes multi-head self-attention (MSA) and MLP blocks. Layernorm (LN) is used after each layer. In order to prevent the loss of some important features during the batch sample learning process, the features before and after the two Batch Learning layers are superimposed. Such a residual structure ensures the batch sample learning ability while reducing the risk of feature loss. The batch learning module collaboratively learns the important focus point knowledge of image features at different stages in the same batch, so that the few-sample category learns the important focus point knowledge of the multi-sample category, and expands the important focus point knowledge in the multi-sample category, thereby enhancing the model's recognition capabilities for both multi-sample and few-sample images;

[0054] 3.2: Use the data constructed in step 3.1 to train the image feature extraction module and obtain the BLConvNeXt model weights:

[0055] During training, ConvNeXt is used to pre-train the model weights on the Imagenet-22k large dataset, and the Cholec80 and Cataract-101 datasets are used to fine-tune the model parameters. The BLConvNeXt training is performed for a total of 30 rounds, and the BLConvNeXt weights with the highest recognition accuracy are finally saved;

[0056] Step 4: Use the data to train the time series feature module to obtain the weights of the multi-level adaptive time convolution module. The specific steps are:

[0057] 4.1 Construction of the temporal feature module: The structure of the temporal feature extraction module (multi-level adaptive temporal convolution) is to add a self-attention module to the causal convolution before two different step lengths, and perform an add operation on the temporal features (temporal feature 1 and temporal feature 2) extracted by the two causal convolutions in the same dimension after the two causal convolutions. Finally, the prediction result is fine-tuned through normal causal convolution to obtain the final surgical stage recognition result.

[0058] 4.2 Training of temporal feature module: fix the trained BLConvNeXt weights, remove the last fully connected layer, train in the target dataset and obtain image features. The extracted image features are input into the multi-level adaptive temporal convolution to further extract temporal features. Specifically, given a video picture composed of T frames, when the features are input into the module, they first pass through a 1×1 convolution layer, which matches the input feature dimension with the selected feature length forwarded to the next layer, and then calculate the correlation between the input features and the current frame features through the self-attention mechanism, judge the importance of the input features and weight them, and then pass through different expansion layers to capture multi-scale temporal information, and finally get the initial stage recognition results after 1*1 convolution and activation function processing. The multi-level adaptive convolution module is trained for a total of 30 rounds, and the weights of the multi-level adaptive temporal convolution module with the highest recognition accuracy are retained;

[0059] Causal convolution is used in multi-level adaptive temporal convolution, which means that the prediction of the surgical stage does not depend on any future frame, so that this method can predict the current surgical stage in real time during surgery;

[0060] Step 5: Load the model training weights, run the surgical stage recognition model completely, and obtain the final prediction results:

[0061] Load the saved BLConvNext module and the multi-level adaptive temporal convolution module weights, and use the trained complete surgical stage recognition network model in the Cho lec80 dataset or Cataract-101 dataset to obtain the final prediction result, that is, the surgical stage recognition result of the dataset.

[0062] The complete surgical stage recognition network model consists of an image feature extraction module and a temporal feature extraction module.

[0063] Compared with existing methods, our proposed ConvNeXt-based multi-stage adaptive category-balanced surgical stage recognition method (i.e., MACBNet) takes into account the problem of uneven data distribution in the stage recognition problem, and continuously optimizes the model's recognition results for few-sample stage images in the image feature extraction module and the temporal feature extraction module, making MACBNet significantly surpass GLSFormer in overall accuracy, precision, recall and Jaccard, which are 1.4%, 2.93%, 4.06% and 5.06% higher respectively.

[0064] In the Cataract-101 dataset, the ten stages of the dataset are incision, viscous injection, capsular bag tearing, water cutting, phacoemulsification, flushing, polishing, lens implantation, viscous removal, and fluid replenishment. Finally, the stage recognition results of the Cataract-101 dataset are obtained. See Table 1:

[0065] Table 1: Comparison of recognition results of Cataract-101 dataset

[0066]

[0067] The MACBNet we proposed also achieved excellent results in another large public dataset, Cho lec 80. As shown in Table 2, MACBNet achieved better results than the most advanced CMTNet in the Cho lec80 dataset, and also surpassed the existing stage recognition methods in all indicators.

[0068] Table 2: Comparison of recognition results of Cho lec80 dataset

[0069]

[0070]

[0071] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-stage adaptive category-balanced surgical stage recognition method based on ConvNeXt, characterized by: The method comprises the following steps: Step 1. Data preparation: prepare the dataset and the model pre-training weights of ConvNeXt on the Imagenet-22k large dataset; Step 2: Data preprocessing: down-sampling the data set and adjusting the resolution to be consistent; Step 3: Use the data to train the image feature extraction module to obtain the BLConvNeXt model weights, specifically: 3.1 Construction of image feature module: The image feature extraction module uses the ConvNeXt model as the backbone network, uses ZeroLoss to replace the cross entropy loss in the ConvNext model, and adds a batch sample learning module after the average pooling layer of the ConvNext model; 3.2 Use data to train the image feature extraction module: During training, use the model pre-training weights of ConvNeXt on the Imagenet-22k large dataset, and fine-tune the model parameters on the Cholec80 and Cataract-101 datasets. The BLConvNeXt training is performed for a total of 30 rounds, and finally the BLConvNeXt model weights with the highest recognition accuracy are retained; Step 4: Use the data to train the time series feature module to obtain the weights of the multi-level adaptive time convolution module; Step 5: Load the model training weights, run the surgical stage recognition model completely, and obtain the final prediction results: The complete surgical stage recognition network model consists of an image feature extraction module and a temporal feature extraction module.

2. The ConvNeXt-based multi-stage adaptive class-balanced surgical stage recognition method according to claim 1, characterized in that: The specific steps of step 4 are: 4.1 Construction of the temporal feature module: The structure of the temporal feature extraction module is to add a self-attention module to the causal convolution before two different step lengths, and perform an add operation on the temporal features extracted by the two causal convolutions after the two causal convolutions with the same dimension superimposed. Finally, the prediction result is fine-tuned through normal causal convolution to obtain the final surgical stage recognition result. 4.2 Temporal feature module training: The trained BLConvNeXt weights are fixed, the last fully connected layer is removed, and the training is performed in the target dataset to obtain image features. After obtaining the image features, the final surgical stage recognition results are obtained through multi-level adaptive temporal convolution. The multi-level adaptive convolution module is trained for a total of 30 rounds, and the weight of the multi-level adaptive temporal convolution module with the highest recognition accuracy is retained.

3. The ConvNeXt-based multi-stage adaptive class-balanced surgical stage recognition method according to claim 2, characterized in that: The specific steps of step five are: loading the saved BLConvNext module and the multi-level adaptive time convolution module weights, applying the trained complete surgical stage recognition network model to the Cholec80 dataset or the Cataract-101 dataset, and obtaining the final prediction result, i.e., the surgical stage recognition result of the dataset.

4. The ConvNeXt-based multi-stage adaptive class-balanced surgical stage recognition method according to claim 3, characterized in that: In 3.1, the process of using ZeroLoss to replace the cross entropy loss in the ConvNext model is: First, add Hard Number as the category balance weight of the cross entropy loss function; Then add the balance parameter α, and the Hard Number formula is shown in (1): where p t is the overall recognition accuracy of the image at that stage, and p c is the accuracy rate of the stage recognition result of the image; Finally, add the random optimization loss β, and the complete ZeroLoss formula is shown in (2), where Loss is the normal cross entropy loss:

5. The ConvNeXt-based multi-stage adaptive class-balanced surgical stage recognition method according to claim 4, characterized in that: In step 3.1, the batch sample learning module consists of two Batch Learning layers and an enhanced residual structure. The Batch Learning layer includes multi-head self-attention (MSA) and MLP blocks. Layernorm (LN) is used after each layer; at the same time, the features before and after the processing of the two Batch Learning layers are superimposed.

Images