Method for classifying images from videos during surgical procedures

Abstract

A method (20) for processing a surgical video to associate a sequence of images extracted from said video with distinct surgical phases (P1-P11), comprising: (S211) extracting said sequence of images from the video, said images being associated with distinct successive instants, (S212) classifying said images by associating said images with predicted surgical phases from a predefined set of surgical phases, implemented by a classification model, (S224) correcting the classification by a hidden Markov model (S223), implemented on the predicted surgical phases as observations, the actual surgical phases associated with the sequence of images being hidden states of said hidden Markov model. Abstract figure: Figure 1

Description

Title of the invention: Method for classifying video images during surgical procedures technical field

[0001] The present disclosure falls within the field of data processing, including images from videos of surgical operations, to predict and extract distinct surgical phases from videos. Previous technique

[0002] It is known to record, in the form of a series of images or videos, sequences of actions or gestures within the framework of acts or operations following well-defined protocols. In particular, in the medical field, the recording of videos of surgical operations is frequently implemented, for example for training purposes or for the automation of actions (in the operating room, for example) triggered by the real-time detection of certain acts.

[0003] In the context of surgical videos, these often constitute several hours of recording, from the preparation of the surgical procedure to the removal or deactivation of the camera. It is therefore difficult and time-consuming to use these recordings, particularly for effective detection or evaluation purposes.

[0004] Solutions exist for segmenting videos into surgical phases, in order to identify and distinguish surgical periods during which the practitioner(s) perform specific surgical steps from periods of inactivity, for example. Such solutions rely in particular on image processing and segmentation techniques that allow an image or set of recorded images to be associated with a given phase. However, most existing classification solutions struggle to provide satisfactory accuracy. Furthermore, some existing solutions involve cumbersome post-predictive processing, which is resource-intensive and / or requires considerable execution time.Such solutions are therefore not suitable for real-time surgical phase predictions, which require both precise and rapid detection in order to trigger actions upon detection of a given surgical procedure (e.g., switching on a device, controlling the light, automatically activating surgical robot functionalities).

[0005] Other solutions also offer post-predictive processing of surgical video classification predictions by considering feature vectors derived from the classification model used. One such solution is for example implemented in the document [CADENE 2016]. However, such treatment requires, in order to exploit the predicted characteristic vectors, the adoption of assumptions and approximations (in this case a Gaussian assumption in the case of [CADENE 2016]), which bias the accuracy of the predicted results.

[0006] Moreover, such existing solutions lack explainability and transparency regarding the predictions obtained, so that existing surgical video classification models can hardly be enriched by medical or theoretical expertise. Summary

[0007] The present disclosure addresses these problems and improves the performance of surgical phase video cutting solutions, increasing the accuracy of surgical phase predictions, while simplifying the prediction process.

[0008] According to a first aspect of this disclosure, a method for processing a surgical video is proposed to associate a succession of images extracted from said video with distinct surgical phases, comprising: - an extraction of said sequence of images from the video, said images being associated with distinct successive moments, - a classification of said images, by associating said images with predicted surgical phases from a predefined set of surgical phases, implemented by a classification model, - a correction of the classification by a hidden Markov model, implemented on the predicted surgical phases as observations, with the actual surgical phases associated with the sequence of images being hidden states of said hidden Markov model.

[0009] Consequently, the proposed method advantageously allows for the classification of images from a surgical video in such a way as to automate and simplify the identification of key moments in the video corresponding to the surgical phases. Thus, such a method saves practitioners (or more generally, any individual concerned) from having to watch several hours of video and manually edit the video. Furthermore, the proposed method allows for refining the accuracy of the video image classification by combining the implementation of a classification model with the correction of the classification using that model.

[0010] In particular, the classification correction step as proposed advantageously corrects potential prediction errors of the classification model both locally and globally by considering all predicted surgical phases and the image sequence as a whole. Furthermore, such classification correction does not require any adjustments (e.g., hyperparameter settings) related to the classification model, so the proposed correction step can be Implementation is possible regardless of the classification model used. The proposed correction thus provides a further improvement in the accuracy of surgical phase prediction via a separate processing component, compatible with any image classification and therefore easily integrated into an image processing pipeline as a post-prediction correction step. The proposed correction relies solely on the output results of the classification model (specifically, the predicted surgical phases), enabling simple implementation, transparent to the user, and generally independent of the classification step.

[0011] Moreover, such a correction block separate from the classification step makes it possible in particular to refine, adapt or improve the correction phase without necessarily affecting or correcting the parameters and / or hyperparameters of the classification model (for example, without changing the weights of the model, or without requiring retraining of the classification model, when the classification model is a learning model).

[0012] A surgical video is understood to be a succession of consecutive images captured from a scene corresponding to a surgical operation, captured by one or more devices incorporating one or more cameras, typically an endoscope. A surgical operation can be of any type, for example, cataract surgery or a cholecystectomy. Such a succession of images forming the video is captured over a defined duration (or period), which may correspond to several minutes or several hours, for example. An image extraction rate can be chosen for the video, so that two images extracted from the video can be considered successive according to a chosen (temporal) granularity. For example, it is possible to consider that an image is extracted every second or every millisecond.

[0013] By a succession of images extracted from the video and associated with distinct successive moments, it is understood that successive images extracted from the video form all or part of the video. For example, when the succession of extracted images forms all the images of the video, the classification of the images is comparable to a segmentation of the video, so that the video is partitioned into distinct surgical phases. The succession of extracted images may form only part of the video, for example, if one or more portions of the video are not used / usable or cannot be considered surgical phases. Thus, segmentation of the video corresponds to a particular case of the classification of successive images extracted from the video, corresponding to the case where the successive extracted images form all the images of the video.

[0014] Surgical phases are defined as predefined categories or labels that can be associated with images in a surgical video and represent a surgical step captured at a given moment during the surgical operation. The set of distinct surgical phases that can be associated with images in a surgical video is finite and known. Such a set of surgical phases may vary in size and nature depending on the surgical video in question (particularly depending on the type of surgical operation filmed) or not. Examples of surgical phases may include: preparation phase, cleaning phase, incision phase, irrigation phase, dissection phase, suturing phase, etc.

[0015] Furthermore, the surgical phases may be subject to constraints. For example, surgical phases may be mutually exclusive (a given image cannot be associated with two distinct surgical phases), ordered (if surgical phase a follows surgical phase b, the reverse is not possible), linear, or follow each other according to a tree structure, etc. Such constraints may be predefined and depend on several parameters such as the type of surgical operation or procedural rules or protocols.

[0016] Image classification refers to the prediction of surgical phases, as predicted by the classification model, respectively associated with the images in the extracted image sequence. For this purpose, the classification model can typically be a supervised learning prediction model. For example, the classification model used could be the TMRnet model, described in the publication [JIN 2021]. More generally, the classification model can be any known classification model or one adapted from a known classification model (for example, by fine-tuning) configured to predict surgical phases associated with each of the extracted images.The predicted surgical phases therefore correspond to predictions of the classification model, unlike the actual surgical phases, which correspond to the real surgical phases associated with the extracted images and are unknown during the implementation of the process. Thus, the more closely the predicted surgical phases correspond to the actual surgical phases, the more efficient (or accurate) the classification step (and therefore the underlying classification model) is.

[0017] By classification correction, it is understood that an additional processing step is added to the classification step in order to refine the prediction of the surgical phases predicted in the classification step. Such a correction aims to potentially correct the surgical phases predicted by the classification model by modeling the sequence of surgical phases in the surgical video as a hidden Markov model. Thus, the classification correction results in, for a sequence of successive extracted images of a given length, a corresponding sequence of predicted (and corrected) surgical phases of the same length (i.e., with, for each image extracted from the video, a determined predicted corrected surgical phase).

[0018] By correcting the classification using a hidden Markov model, it is understood that the sequence of video images associated with surgical phases is modeled by a hidden Markov chain (or Hidden Markov Model or HMM), which amounts to inferring the actual surgical phases as hidden states from the surgical phases predicted by the classification model, as observations (or observable states). Thus, the method of the present disclosure proposes to rely directly on the categorical predictions (i.e., of the predicted surgical phases) as observations.

[0019] The features described in the following paragraphs may optionally be implemented independently of each other or in combination with each other:

[0020] In one embodiment, the classification model is pre-trained and validated using a set of labeled images, said labeled images corresponding to a sequence of images associated with known surgical phases. Optionally, the classification model is pre-trained, validated, and tested using a set of labeled images.

[0021] Consequently, the prediction of surgical phases based on successive extracted images can exhibit a first level of accuracy, which will be improved through the subsequent correction step. Furthermore, the results and performance of such a classification model can be improved without affecting the intrinsic parameterization of the classification model, thus allowing the correction step (corresponding to post-processing) to be superimposed on the results of any classification model.

[0022] In other words, the classification model used can be any learning model (e.g., deep learning, a neural network). The classification model can be pre-existing and refined for the detection of surgical phases. The classification model can, for example, correspond to the TMRNet model described in the publication [JIN 2021]. The classification model can be a Charnel-Separated Convolutional Network (or CSN) described in the publication [TRAN 2019], in particular the CSN-R101 model.

[0023] Such a classification model is trained on labeled surgical video images (i.e., associated with known surgical phases) forming a training set. Once trained, the classification model is validated on labeled surgical video images (different from the images in the training set) forming a validation (or calibration) set. Such a phase The validation phase then allows for calibration of the classification model's errors on the validation set (which was not seen by the trained classification model) and adjustment of the classification model's parameters / hyperparameters to improve its performance. Finally, the resulting classification model can be tested on labeled surgical video images (different from the images in the training and validation sets) forming a test set.

[0024] Thus, the labeled image set must be of sufficient size to allow at least the training and validation of the classification model. The labeled image set may consist of images of surgical procedures associated with a label corresponding to the known surgical phase. The labeled image set may also be extracted from labeled surgical videos segmented into known surgical phases. For example, the labeled image set may consist of one or more of the following datasets: Cholec80, Cataract-101, which will be detailed later in this disclosure.

[0025] In one embodiment, the method comprises, before the correction step: - a calculation of a matrix of transitions associated with the distinct surgical phases from at least the known surgical phases associated with the labeled images, - a calculation of a confusion matrix from the classification model implemented on at least some of the labeled images for a calibration step, and in which the classification correction depends on the transition matrix and the confusion matrix.

[0026] A transition matrix is ​​understood to be a matrix (or equivalently, a two-way table) whose coefficients correspond to the probabilities of observing (directly) a transition, between two successive images, from one surgical phase (e.g., on the rows of the matrix) to another (on the columns of the matrix). A transition from one surgical phase to another between two successive images is understood to mean the observation of the surgical phases of two successive images, each being associated with a given surgical phase.In particular, the transition between two surgical phases in the context of the transition matrix can correspond to a transition to the same surgical phase (e.g., when the two successive images are associated with the same surgical phase) or to a transition of change of surgical phase (e.g., when the two successive images are respectively associated with two different surgical phases). In the context of this disclosure (which is that of a Markov model), the transition matrix is ​​a theoretically stochastic matrix, reflecting that, from a given surgical phase, only one transition to another surgical phase is possible. (the same or another) among the set of distinct surgical phases defined is possible. The coefficients in the first row correspond to the probabilities of transitioning, for a given image, from a first surgical phase to, for the subsequent image, all other surgical phases. The coefficients on the diagonal of the transition matrix then reflect the probability of repetition of a surgical phase, that is, the probability, between two successive images, of being in the same surgical phase. A coefficient mT>ij of 1.0 (or 100%) in the transition matrix reflects a certain transition, that is, that a surgical phase i is systematically directly followed by a surgical phase j (at least across the set of successive images observed). Conversely, a coefficient mT>ij of 0.0 (or 0%) reflects an impossible transition (at least across all successive observed images), meaning that a surgical phase i is never directly followed by another surgical phase j across all successive observed images. Specifically, in the context of this disclosure, the transition matrix is ​​calculated from labeled images (and more specifically, from the known surgical phases associated with the labeled images), for example, the labeled images in the validation set. Each coefficient mTjijj of the transition matrix can then be calculated, for example, by averaging over all transitions between two successive observed labeled images (i.e., from the number of occurrences of the transition between surgical phases i and j observed across all pairs of successive observed labeled images). In particular, the diagonal coefficients of the transition matrix (i.e.,The coefficients mTjij (where i=j) can be non-nuisance, reflecting that from one labeled image to another, there is a non-zero probability of observing the same known surgical phase. In practice, when surgical phases extend over a certain duration (in any case longer than the acquisition or extraction period of successive video images), such coefficients are non-nuisance, or even rather high (i.e., close to 1 or 100%), since there is a high probability of observing the same surgical phase from one image to the next. The non-diagonal coefficients of the transition matrix (i.e., mTjijj, where i is different) can be non-nuisance and depend in particular on the constraints of the execution of surgical phases within a type of surgical operation.

[0027] A confusion matrix is ​​understood to be a matrix whose coefficients reflect a degree of performance (or similarly, error) of the classification model. Typically, the coefficients of the confusion matrix can be determined from the performance of the classification model trained on the validation set for the purpose of calibrating the subsequent classification correction algorithm. The rows of the confusion matrix can correspond to the phases The known surgical phases and the columns of the confusion matrix can correspond to the predicted surgical phases. Thus, the closer all the coefficients on the diagonal of the confusion matrix are to 1 (and the other coefficients are close to 0), the better the classification model performs.

[0028] In one embodiment, coefficients of the transition matrix depend on at least one element among: a type of surgical operation captured in the surgical video, a scheduling of the surgical phases of the predefined set of surgical phases.

[0029] Advantageously, the transition matrix allows for the reflection of information that can be interpreted and adjusted by humans, particularly medical experts. Indeed, the coefficients of the transition matrix can be determined empirically from known surgical phases associated with labeled images, as described above. Such coefficients of the transition matrix can also be determined, supplemented, refined, or enriched by provided expert knowledge or data.Indeed, certain types of surgical procedures (or even all types of surgical procedures) have specific characteristics (for example, the absence of a surgical phase, in which case any coefficient in the transition matrix representing a transition from, within, or to this surgical phase should be zero; or a systematic sequence of two surgical phases, in which case the corresponding coefficient in the transition matrix should be non-zero). Surgical procedures may also follow a strict sequence due to the protocol-driven nature of surgical interventions. Thus, a linear ordering of surgical phases may exist and allow for the determination or refinement of the coefficients in the transition matrix. Such specific characteristics can vary depending on the type of surgical procedure: the number of surgical phases, their nature, and / or their order may vary.Such a possible manual correction of the transition matrix then makes it possible to enrich the data from the labeled images with external expert data, which improves the relevance of the transition matrix and therefore the correction step.

[0030] Advantageously, the coefficients of the transition matrix can be determined so as to maximize the sparseness of the transition matrix. Indeed, the greater the number of nuisance coefficients, the more linear and therefore strictly sequential the underlying type of surgical operation. A large number of nuisance coefficients in the transition matrix advantageously allows for greater efficiency in the classification correction step, in that prediction errors by the classification model can be more easily identified during the correction step, in the case of strict linearity of the surgical phases.

[0031] More generally, by using the calculation of a transition matrix and a confusion matrix to correct image classification, the proposed method advantageously allows for the use of data that can be humanly interpreted, manually adjusted, or adapted, particularly by medical experts, depending on the type of surgical operation involved or changing operational and / or procedural specificities. Indeed, as described above, the coefficients of the transition matrix can be manually adjusted when specific transitions between certain surgical phases can be defined (typically, when certain transitions between surgical phases are mandatory or, conversely, impossible).Such manual adjustment, which improves the accuracy of the matrices and therefore of the hidden Markov model, can be implemented without altering the classification model or the underlying operation of the hidden Markov model. This allows both for the simple enrichment of the model with additional human data or knowledge and provides interpretability and transparency of the process applied to surgical videos—such interpretability being essential in the medical field. The confusion matrix also provides interpretability of the process by reflecting a degree of performance of the classification model.

[0032] In one embodiment, the hidden Markov model is applied with the transition matrix as the transition matrix of the hidden states of said model and with the confusion matrix as the distribution matrix of the observations of said model.

[0033] In one embodiment, coefficients of the confusion matrix and / or the transition matrix are smoothed by Laplace smoothing.

[0034] Laplace smoothing allows adjustment of transition and / or confusion matrix coefficients for which the labeled image set does not allow for a representative or sufficiently accurate calculation. In particular, Laplace smoothing can replace uncalculated, default-nullified, or default-calculated coefficients based on one or a few labeled images with smoothed, low but non-zero values. Typically, if a surgical phase is not represented in the labeled image set, the transition matrix coefficients corresponding to a transition to or from such a surgical phase and the confusion matrix coefficients for the correct prediction of this surgical phase will be default-nullified, creating a prediction bias (e.g., transitions to, into, or from this surgical phase will be considered impossible when this is not necessarily the case).Laplace smoothing then advantageously compensates for such insufficient or underrepresented information, thereby improving the effectiveness of classification correction. In one embodiment, Laplace smoothing can be implemented on the transition matrix of . selectively, by identifying the nuisance coefficients in the transition matrix but which correspond to theoretically possible transitions (e.g., using a graph of the progress of surgical operations, data from medical experts...) and thus assigning small non-zero values ​​to these specific coefficients.

[0035] In one embodiment, the classification correction includes a maximization of a total likelihood of a sequence of hidden states associated with the image sequence, said maximization resulting in corrected surgical phases associated with the images of the image sequence.

[0036] In particular, such maximization can be implemented via a Viterbi algorithm.

[0037] Advantageously, the implementation of a Viterbi algorithm allows for rapid execution of the classification correction step, compared to other deep learning algorithms. The correction step can then be executed to process large amounts of data, such as videos lasting several hours, and enable rapid analysis of surgical videos. The method can then be advantageously implemented for real-time or near-real-time surgical phase detection applications, for example, for triggering specific actions or processes depending on the detected surgical phase (e.g., issuing alerts or notifications specific to certain surgical phases, automatically adjusting the brightness or parameters of medical equipment, etc.).

[0038] In another embodiment, such maximization can also be implemented by any other equivalent type of algorithm.

[0039] According to a second aspect of this disclosure, a device is proposed comprising means for acquiring surgical video and a computer configured to implement the method as proposed in the first aspect. Such a device corresponds, for example, to a medical device used during surgery, typically an endoscope.

[0040] According to another aspect, a computer program is proposed comprising instructions for implementing all or part of a process as defined in the first aspect when this program is executed by a processor. According to another aspect, a non-transient, computer-readable recording medium is proposed on which such a program is recorded. Brief description of the drawings

[0041] Other features, details and advantages will become apparent from reading the detailed description below and from analyzing the accompanying drawings, in which: Fig. 1

[0042] [Fig.1] illustrates a classification of images from a video in surgical phases. Fig. 2

[0043] [Fig.2] illustrates steps of a process for classifying images from a video into surgical phases according to an embodiment. Fig. 3

[0044] [Fig.3] illustrates steps in constructing a classification model according to one embodiment. Fig. 4

[0045] [Fig.4] shows a graph of the progress of a type of surgical operation according to a first example. Fig. 5

[0046] [Fig.5] shows a confusion matrix according to the first example. Fig. 6

[0047] [Fig.6] shows a confusion matrix smoothed according to the first example. Fig. 7

[0048] [Fig.7] shows a flow diagram of a type of surgical operation according to a second example. Description of the implementation methods

[0049] Reference is made to [Fig. 1]. [Fig. 1] schematically illustrates examples of Dr, Dp, Dc segments of the same video in surgical phases P1-P7. In the remainder of this description, reference will be made to a segmentation of the video, that is, a classification of all the successive images forming the video. However, as previously stated, the classification of successive images may concern only a portion of the video, so that the segmentation of the video can be understood as the segmentation of successive images forming a portion of the video. The video under consideration can be defined as a succession of consecutive segments in time t, extending over a delimited duration. For example, with reference to [Fig. 1], the video under consideration extends from a start time T0 to an end time TN = T7. The successive images forming the video are not discretized and are therefore not represented one by one on the [Fig.1].

[0050] Figure 1 illustrates an actual segmentation Dr of a video. Such an actual segmentation Dr corresponds to an actual classification (or labeling) of the video images. Such an actual segmentation Dr can typically be obtained manually, for example by a competent medical professional, by viewing the entire video between times T1 and TN and associating a surgical phase with each viewed image. Thus, such an actual segmentation Dr can be considered the substantially correct segmentation of the video and serves as a reference segmentation. With reference to Figure 1, the actual segmentation Dr of the video associates a surgical phase PI with the images acquired between times T1 and T1>TO, a surgical phase P2 to images acquired between times T1 and T2 > T1, a surgical phase P3 to images acquired between times T2 and T3 > T2, a surgical phase P4 to images acquired between times T3 and T4 > T3, a surgical phase P5 to images acquired between times T4 and T5 > T4, a surgical phase P6 to images acquired between times T5 and T6 > T5 and a surgical phase P7 to images acquired between times T6 and T7 > T6.

[0051] Figure 1 also illustrates an example of a predicted segmentation Dp of the video. Such a predicted segmentation Dp typically corresponds to a segmentation resulting from a prediction of a classification model. Such a classification model can correspond to any known prediction model configured to predict surgical phases associated with images. As illustrated in Figure 1, the predicted segmentation Dp provides an estimated segmentation of the video into surgical phases. Some predictions may be correct; for example, the images between times T1 and T2 are correctly predicted as corresponding to surgical phase P2. However, the predicted surgical phases may contain several errors. For example, between times T0 and T1, some images are predicted as corresponding to surgical phase P2, P6, or P7, while these images actually correspond to surgical phase P1.Similarly, between times T4 and T6, some images are predicted as belonging to surgical phase P7, which is therefore predicted as taking place between surgical phases P5 and P6, whereas in reality, surgical phases P5 and P6 are consecutive.

[0052] Figure 1 also illustrates an example of a corrected Dc segment of the video, obtained following the implementation of the proposed method, which will be described in more detail in Figure 2. The corrected Dc segment thus makes it possible to obtain an improved classification of the video segmentation into surgical phases compared to the predicted Dp segmentation without a post-prediction correction step. Indeed, the corrected Dc segmentation is closer to the actual Dr segmentation of the video.

[0053] Reference is now made to [Fig. 2]. [Fig. 2] schematically illustrates steps of a method 20 for associating images extracted from a video with distinct surgical phases according to an embodiment proposed by this disclosure. The method 20 can be implemented by a device (not shown in the figures) comprising at least one computer, such computer including at least one processor and one memory unit. Such a computer can in particular be connected to means for acquiring a video, which may be included in the device or in another device, such acquisition means being arranged so as to acquire a stream of images from a surgical procedure, for example by being positioned in an operating room. The device could, for example, be an endoscope.

[0054] In general, the process 20 includes, in particular, a sequence of steps S21 (including steps S211 and S212) for classifying images into surgical phases and a sequence of steps S22 (including steps S221, S222, S223, and S224) for correcting the classification. These sequences of steps S21 and S22 can represent two distinct processing components and can therefore be implemented by two separate processing entities. Figure 2 illustrates these two sequences of steps S21 and S22 consecutively within the same process 20, but those skilled in the art will understand that these two sequences of steps S21 and S22 are independent and that, for example, the sequence of steps S22 can be implemented from predicted surgical phases obtained at the end of any classification sequence.

[0055] In step S211, a sequence of images is obtained from a surgical video. In some embodiments, step S211 may include or be preceded by a step of acquiring the surgical video using acquisition means such as a camera and a step of extracting the sequence of images. Such a sequence of extracted images may result from cutting the video or one or more portions of the video at a chosen extraction rate, for example, one image every second of the video. The extraction rate of the sequence of images may be chosen based on one or more factors such as the total duration of the video or the expected degree of accuracy of the process 20.

[0056] At the end of step S211, a succession of images associated with successive moments is obtained, each image representing a scene acquired at the corresponding moment of the surgical operation. It should be noted that the surgical phases respectively associated with these successive images are not known (these are therefore not labeled images).

[0057] At step S212, the sequence of images is passed to a classification model, typically a learning model trained on a set of labeled surgical images as described above and configured to classify images into predicted surgical phases. The configuration of the classification model for this purpose will be described in more detail in [Fig. 3].

[0058] In step S212, the classification model classifies the extracted images, associating each image with a predicted surgical phase. Thus, at the end of step S212, the classification model can return a sequence of values ​​(for example, in the form of a list) corresponding to the numbering of predicted surgical phases (e.g., 1 for surgical phase PI, 2 for surgical phase P2, etc.). The number of values ​​returned is then equal to the number of extracted images given to the classification model.

[0059] In another embodiment, such a classification can be implemented by any other remote processing entity and step S212 can include receiving values ​​corresponding to the surgical phases predicted by such a remote processing entity.

[0060] At stage S212, a classification of the images into predicted surgical phases took place. Such a classification typically results in the predicted segmentation Dp of [Fig. 1], which potentially includes prediction errors. The subsequent steps of the process 20 then allow for the implementation of a correction to such a classification.

[0061] In steps S221 and S222, a transition matrix and a confusion matrix of the classification problem are respectively calculated.

[0062] In one embodiment, the transition matrix and the confusion matrix can then be calculated empirically from at least a portion of the labeled surgical image set, typically distinct from those used to train the classification model. For example, for the transition matrix, such labeled surgical images represent a sample from which transition probabilities between successive surgical image phases can be empirically calculated. For the confusion matrix, the performance of the classification model on a portion of the labeled surgical images (e.g., the validation set) allows the coefficients of the confusion matrix to be empirically calculated.

[0063] Examples of transition and confusion matrices will be described in detail in two examples.

[0064] In a particular embodiment, the transition and confusion matrices can be calculated or adjusted with the intervention of an optional step S220 for manually adjusting the transition matrix and / or the confusion matrix. Typically, such a step S220 can be implemented using additional information provided by a medical expert (e.g., a surgeon specializing in the surgical procedure filmed in the surgical video). Such information may be specific to the surgical procedure or not, including, for example: - the presence or absence of certain surgical phases, - an impossible, systematic, preferred, or possible ordering of certain surgical phases - specific events within the context of the filmed surgical operation, - surgical procedural or protocol information.

[0065] Based on such additional information, the transition and confusion matrices can be adjusted to reflect the information provided. For example, step S220 may indicate that any surgical operation must begin with a preparation phase, unique in the surgical operation, such that all coefficients of the transition matrix corresponding to a transition from a given surgical phase (other than the preparation phase) to the preparation phase must be harmed.

[0066] Step S220 may in particular include the direct manual adjustment of matrix coefficients.

[0067] In an optional step, not shown in [Fig.2], the matrix coefficients can be smoothed by Laplace smoothing.

[0068] It should be noted that steps S220, S221, and S222 are independent of the extraction of successive images from the surgical video in step S211, and of the surgical phases predicted in step S212. In particular, [Fig. 2] illustrates steps S220, S221, and S222 as being implemented after sequence S21, but in other embodiments, these steps can be implemented before or during sequence S21.

[0069] At a step S223, the problem of classifying successive images extracted from the surgical video at distinct surgical phases is modeled in a hidden Markov model (or chain).

[0070] In such a hidden Markov model, the sequence of images extracted from the surgical video is modeled as a system transitioning from one state to another among a finite set of states, such states corresponding to the surgical phases, which are finite in number. The Markov model is said to be hidden in that such (real) surgical phases are hidden (i.e., unknown to the problem) and are inferred from known observations, corresponding to the surgical phases predicted by the classification model in step S212.

[0071] In summary, the proposed hidden Markov model is described by: - a set of states, i.e., existing surgical phases P = {1,2,.. .N] (or P = {P1,P2,...PN} - instants t, respectively associated with successive extracted images, - hidden states Ht, with values ​​in P, unknown in the problem description, - observable states Xt, with values ​​in P, corresponding to the phases predicted by the classification model - a transition matrix of hidden states MT, which corresponds to the transition matrix calculated in step S221, where each coefficient mTjij represents the transition probability of the hidden state H; to the hidden state Hj: mTij = P(Ht+i= j I Ht= i) - an emission matrix (or observation distribution matrix), which is chosen in this process 20 as the calculated confusion matrix at step S222, where each coefficient represents the probability that a surgical phase i is predicted in surgical phase j: mCjij = P(Xt= j I Ht= i).

[0072] At step S224, the correction of observable states, i.e., surgical phases predicted by the classification model, is implemented in order to determine the hidden states associated with such observable states. In other words, the objective is to determine a sequence of hidden states associated with the succession of images, such hidden states being inferred from the respective observable states.

[0073] To this end, step S224 may include maximizing the total likelihood of the sequence of hidden states associated with the image succession, i.e., finding the sequence of hidden states that maximizes a likelihood, given a sequence of observable states Xt. Such a likelihood maximization step then amounts to refining the predicted surgical phases of the classification model, so as to maximize the probability of their sequence.

[0074] Such maximization can notably be implemented via a Viterbi algorithm or any equivalent type of algorithm. In step S224, such an algorithm constructs and explores a probability tree of each possible sequence of hidden states in order to determine the one that maximizes a likelihood.

[0075] Thus, taking the example of [Fig. 1], the classification model determined a predicted segmentation Dp with certain images between times T2 and T3 associated with predicted surgical phases P4 (such predictions are therefore false since the images extracted between times T2 and T3 are in fact all associated with surgical phase P3, as illustrated by Dr). The proposed correction then allows, in step S224, for the determination that the likelihood of the hidden state sequence is maximized when the images between times T2 and T3 are all associated with phase P3 (and not, for example, when the images around time T3 are typically associated with phase P4, which could be the case with other post-predictive processes that simply smooth or filter the predictions).

[0076] Reference is now made to [Fig. 3]. [Fig. 3] illustrates steps in constructing a classification model configured to determine predicted surgical phases associated with a sequence of images extracted from a surgical video. Such a classification model is typically used to determine the predicted surgical phases in step S212 of process 20.

[0077] In a step S31, successive images extracted from one or more surgical videos are collected. Such a step S31 may include the collection of a plurality of surgical videos and the extraction (or clipping) of successive images from such surgical videos. The successive images may represent different types of surgical operations, including surgical phases of different natures, ordered differently or in different numbers.

[0078] In particular, the images collected in step S31 are labeled, meaning that each image is associated with a known surgical phase from a predefined set of surgical phases. Thus, a succession of labeled images collected in step S31 presents a known sequence of surgical phases. It should be noted that successive labeled images can be associated with the same label (i.e., with the same surgical phase), particularly when a surgical phase extends over a period longer than the extraction time of each labeled image (e.g., if a surgical phase lasts approximately one minute and the extraction rate is one image per second, approximately sixty successive images will be associated with the same label).

[0079] Step S31 may also include preprocessing of the extracted successive images. Such preprocessing may, for example, include data augmentation of the extracted successive images to increase the number of images associated with the same label (i.e., the same surgical phase). To this end, random transformations such as translations, zooms, rotations, and brightness changes can be applied to the sequence of labeled images, resulting in augmented sequences of labeled images at step S31. Such data augmentation is particularly relevant when few or insufficient labeled images are available, or to improve the generalization of the learning model.For example, one or more of the following augmentation methods can be used: RandAugment, CutOut (replacing part of an image with random noise), Mixup or CutMix (mixing two images to form a new image).

[0080] At step S32, the set of successive labeled (and possibly augmented) images is divided into different sets, including a training set and a validation set. A test set may also be included. For example, 40% of the collected labeled images may form the training set, 10% of the collected labeled images may form the validation set, and 50% of the collected labeled images may form the test set.

[0081] At step S33, a classification model is trained on the training set. Such a classification model may correspond to a supervised learning or deep learning model adapted for classifying successive images in surgical phases. In particular, such a classification model may be a pre-trained model (for example, on various images different from the labeled image set, corresponding or not to images). of surgical operations), which is then specialized at the S33 training stage on the training set (by fine tuning so as to be able to specifically classify images from surgical videos into surgical phases.

[0082] At step S34, the trained classification model is validated on the validation set. To do this, the classification model, once trained on the training set, is implemented on the images of the validation set in order to evaluate the prediction performance of the classification model. Based on this performance, the hyperparameters of the classification model can be adjusted.

[0083] As described previously, the validation set can also be used to calibrate the S22 classification correction sequence, particularly for calculating the confusion matrix in step S222, which determines a representative estimate of the errors made by the classification model trained on images that the classification model has never seen during the S33 training step. This explains, in particular, the importance of having a sufficient amount of data in the validation set, so that the calibration is as representative as possible of the reality of the diversity of labeled image sequences.

[0084] In an optional step S35, the classification model can be tested on the test set. Such a test step makes it possible to determine the performance of the classification model. More generally, such a test set can also be used to determine the performance of the entire process 20, i.e., of the classification model but also of the classification correction. Examples

[0085] The following examples, implemented on April 22, 2024, illustrate two detailed examples of implementation of method 20 on two separate datasets, for the classification of successive images from these two datasets.

[0086] For each example, the same classification model is used, namely the CSN-101, which corresponds to a Charnel - Separated Convolutional Network (CSNf) architecture as described in the publication [TRAN 2019]. In each of the examples presented, the implemented classification model is built from the same initial predictive model (backbone) of the ResNet-101 type, pre-trained and then fine-tuned on images from each of the considered datasets.

[0087] In particular, each example dataset is also implemented – for comparison with the proposed method 20 – with the algorithm adopting a Gaussian assumption, described in the publication [CADENE 2016] mentioned in the prior art of this disclosure. Such an algorithm will be referred to hereafter as the “Gaussian algorithm”.

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] To evaluate the respective performance of both the 20-step process and the Gaussian algorithm on each dataset, the following performance metrics are defined. Such metrics are typically evaluated using the test set. The Jaccard index, hereafter referred to as "Jaccard," allows us to evaluate the similarity between two sets. It is calculated by dividing the number of elements at the intersection of the sets by the number of elements at the union of the sets. Overall accuracy, referred to hereafter as "Accuracy," measures the proportion of correct predictions relative to the total number of images examined (typically, the number of images in the test set). Such correct predictions include both true positives (e.g., if a surgical phase PI is predicted as such at the end of procedure 20) and true negatives (here, if a surgical phase absent from successive images is not predicted as present). We have: Number of correct predictions Accuracy = .....---—— Total number of images tested Precision (simple), hereafter referred to as "Precision," measures the proportion of correct positive predictions (e.g., a surgical phase PI is predicted as such) relative to the total number of positive predictions (i.e., surgical phases predicted as present in the sequence, whether correct (true positives) or not (false positives)). In other words, precision allows us to assess what proportion of positive identifications of surgical phases were actually correct. We have: True positives Precision = True positives + False positives Recall, referred to hereafter as "Recall," measures the proportion of correct positive predictions (i.e., true positives) relative to the total number of actual positive cases (whether correctly predicted or not). In other words, recall allows us to assess the proportion of true positives that were correctly identified. We have: True positives ixCClUl — positive yrajs + False negatives Example 1: Cholec80 dataset The first example is based on a Cholec80 dataset, containing eighty (80) videos of cholecystectomy surgeries performed by thirteen (13) different surgeons. Such a dataset is described in the publication [TWINANDA 2016] (section IV, A). The surgical videos in the Cholec80 dataset present a sequence of seven distinct surgical phases (in particular, the cholecystectomy surgical operation presents seven distinct surgical phases): Preparation (PI), CalotTriangleDissection (P2), ClippingCutting (P3), GallbladderDissection (P4), GallbladderPacking (P5), CleaningCoagulation (P6) and GallbladderRetraction (P7). Such surgical phases P1-P7 and their sequence within the cholecystectomy operation are, for illustrative purposes, schematically represented in [Fig. 4] as a flow diagram of the cholecystectomy operation. The flow diagram in [Fig. 4] notably allows us to represent the (theoretical) succession of changes between the 7 surgical phases P1-P7 observable in cholecystectomy operations. For example, starting from the initial state (represented by Start in [Fig. 4]) corresponding to the beginning of the cholecystectomy operation, the flow diagram of the operation illustrated in [Fig. 4] shows the following:[4] indicates two theoretically possible surgical phases to begin a cholecystectomy: the surgical preparation phase (Preparation) PI (in 89% of cases) or the Calot triangle dissection phase (CalotTriangleDissection) P2 (in 11% of cases). In another example, phases PI, P2, P3, and P4 are sequential with a probability of 1, such that each phase necessarily leads to a single possible next surgical phase. Such a flowchart of the cholecystectomy operation can, for example, be represented from theoretical or experimental information from medical experts and allows visualization of the theoretical sequence of surgical phases for the operation in question.

[0099] As a point of reference, such a flow diagram illustrates that the surgical phases considered in the cholecystectomy operation are not all ordered. For example, it is possible to move from phase P6 to phase P7 and vice versa, and the same applies to phases P5 and P6.

[0100] From such a dataset, a plurality of sequences of thirty-two (32) successive images were extracted, with an extraction frequency of 1 image per second. The sequences of successive images are labeled and are divided into training, validation, and test sets.

[0101] The pre-trained classification model described above is then trained (specialized by fine tuning) on ​​the training set, and then validated on the validation set.

[0102] A transition matrix associated with the present example can then be calculated (corresponding to a transition matrix calculated specifically for the cholecystectomy surgical operation, represented by the Cholec80 dataset).

[0103] The transition matrix calculated on the basis of the Cholec80 dataset for cholecystectomy surgery, denoted MTjChOiec80, represents the probabilities (means), between two successive images from the validation set of the Cholec80 dataset, of transitioning from a starting state (represented on the rows of MTjChoiec8o) towards an arrival state (represented in the columns of MTjChOiec80). For example, a coefficient mTjij represents the average proportion observed across all image sequences in the validation set of the Cholec80 dataset of a transition between surgical phases i and j. In particular, when i differs from j, the coefficient at position i, j (row i, column j) of MTjChOiec80 represents the probability of a surgical phase change between surgical phases i and j from one successive image to the next. When i equals j, the coefficient of MTjChoiec8o is a diagonal coefficient and then represents the probability of remaining in state i from one successive image to the next. Such states include an initial state (represented by Start on [Fig.4]), corresponding to the beginning of the surgical operation, the surgical phases P1-P7 of the cholecystectomy operation and a final state (represented by End on [Fig.4]), corresponding to the end of the surgical operation.

[0104] For example, such a surgical phase transition matrix associated with successive images of the validation set of the Cholec80 dataset can be represented in such a striking way:

[0105] 'NA 8.3 le-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NA 0.99 1.05e-3 0.00 0.00 0.00 0.00 0.00 0.00 NA 0.00 1.00 5.93e-3 0.00 0.00 0.00 0.00 0.00 NA 0.00 0.00 0.99 5.91e-3 0.00 0.00 0.00 0.00 ^TÇholecm = NA 0.00 0.00 0.00 1.00 9.03e-4 2.62e-4 0.00 0.00 NA 0.00 0.00 0.00 0.00 0.99 6.66e-3 3.59e-3 0.00 NA 0.00 0.00 0.00 0.00 1.27e-3 1.00 3.66e-3 1.29e-3 NA 0.00 0.00 0.00 0.00 0.00 0.00 1.00 3.63e- 3,NA NA NA NA NA NA NA NA NA

[0106] The MTjChoiec8o transition matrix is ​​a 9x9 matrix, with rows representing starting states (current states or surgical phase of the current image) and columns representing ending states (next states, surgical phase of the next image). Thus, the coefficient in position i, j (row i, column j) represents the probability of transitioning from state i to state j from one image to the next. Such states include an initial state (represented by Start in [Fig. 4]), corresponding to the beginning of the surgical operation, surgical phases P1-P7, and a final state (represented by End in [Fig. 4]), corresponding to the end of the surgical operation.

[0107] It should be noted that the MTjChOiec80 transition matrix exhibits non-null diagonal coefficients, even having values ​​in the vicinity of 1. Indeed, such diagonal coefficients reflect that, from one labeled image to another, the probability of observing the same surgical phase is almost certain (and given that a surgical phase generally lasts more than one second). The probability of moving to a (subsequent) distinct surgical phase is low and non-zero. In particular, the non-null off-diagonal coefficients of such an MTjChOiec80 transition matrix are logically related to the probabilities of observable surgical phase changes. theoretically during the course of a cholecystectomy operation as represented in [Fig.4]: if a transition is observed between two distinct states i and j of two successive images (i.e., that the coefficient mTjijj of the matrix MTjChOiec80 is non-zero), then a priori the change of state is theoretically observable (i.e., that an edge connecting the nodes corresponding to phases i and j in the operation progress graph of [Fig.4] exists).

[0108] Furthermore, the MTjChOiec8o transition matrix is ​​theoretically stochastic (the sum of each row is 1): it therefore reflects all possible transitions from each surgical phase (even within the same surgical phase). However, presenting the transition matrix with coefficients limited to a few decimal places can lead to sums of coefficients on each row differing slightly from 1, as illustrated by the MTjChOiec80 transition matrix. In practice, the precision of the transition matrix used in the correction step is sufficiently high (64-bit cgJloal) that such an approximation is largely negligible.

[0109] The transition matrix MTjChoiec8o thus calculated (at step S221) and used for modeling the problem according to a hidden Markov model.

[0110] The confusion matrix associated with this example is also calculated. Such a confusion matrix is ​​shown, for example, in [Fig. 5]. The confusion matrix is ​​a 7x7 square matrix (since the type of surgical operation considered in this example contains 7 surgical phases) and represents, in rows, the known surgical phases (i.e., labels of successive images in the validation set) and, in columns, the surgical phases predicted by the classification model trained for these labeled successive images in the validation set, which were never seen by the classification model during training. Indeed, such a confusion matrix is ​​calculated, in particular, from the validation set formed with a portion of the Cholec80 dataset.

[0111] Thus, with reference to the confusion matrix illustrated in [Fig.5], it is observed that the trained classification model best recognizes the surgical phase P4 (among the images in the validation set associated with the surgical phase P4, 90% were correctly classified), while the surgical phase PI is the least well identified (among the images in the validation set associated with the surgical phase PI, 46% were correctly classified).

[0112] With reference to [Fig. 5], it is observed that several coefficients are undefined (noted NA in the confusion matrix). Such undefined coefficients typically correspond to predictions that did not occur based on the validation set. For example, the coefficient ruc ^i in the third row and first column This reflects the proportion of images labeled with surgical phase P3 that would be predicted to be associated with surgical phase PI. This coefficient mc,3,i is not defined here because such a prediction scenario did not occur. The same applies, for example, to the coefficients mc,4,1, nie,5,1, nie,6,1, nie,7,1, and mc,7,3. Such a confusion matrix would then be exploited in subsequent steps (particularly during the classification correction sequence) by interpreting the undefined coefficients NA as zero values. Such an interpretation would bias the results, as it amounts to overestimating the performance of the classification model. Indeed, the coefficients mc,3,i and mc,i,3 nuis are equivalent to concluding that the model never confuses surgical phases PI and P3.Such an assumption is not necessarily true, for example if the validation set is not sufficiently representative of all surgical phases and / or if the performance of the classification model on such a validation set is insufficient to reflect the actual predictive capacity of the model for certain surgical phases.

[0113] To limit this interpretation bias, the confusion matrix can be smoothed so as to replace the undefined NA coefficients with low (close to zero) but non-zero values. Such smoothing can, for example, be implemented by Laplace smoothing. The confusion matrix smoothed by such Laplace smoothing is illustrated, for example, in [Fig. 6], where undefined NA coefficients no longer appear.

[0114] The data from the first example are then modeled into a problem according to a hidden Markov model as described previously and a sequence of surgical phases (hidden states) is determined (for example by Viterbi's algorithm), in correction of the succession of surgical phases predicted by the classification model.

[0115] The following table gives a summary of the comparative average performance of process 20 and the Gaussian algorithm on the Cholec80 dataset. Jaccard Algorithm Accuracy Precision Recall Procedure 20 (proposed in this disclosure) 90.18 95.26 95.27 95.17 Gaussian ([CADENE 2016]) 89.17 93.22 94.43 94.48

[0116] The following table gives the detailed scores per predicted surgical phase (numbered according to [Fig.4]) of the comparative performance of procedure 20 and the Gaussian algorithm on the Cholec80 dataset. Method 20 Gaussian Algorithm (as proposed in this disclosure) ([CADENE 2016]) Surgical Phases Jaccard Precision Recall Jaccard Precision Recall PI 90.4737 97.3587 91.6677 94.5789 97.2044 95.9409 P2 93.8034 97.5425 96.3291 92.0838 98.7617 93.4425 P3 86.8653 94.0433 93.0921 81.9997 90.3500 90.2381 P4 96.2441 98.0478 98.3570 92.7252 95.0402 97.8201 P5 91.6667 95.6185 96.7166 87.3143 91.2263 96.9537 P6 85.883 93.1348 94.4853 85.1635 91.7518 94.9539 P7 86.3345 91.1576 95.5599 90.3437 96.684 91.9852 Average 90.1815 95.2719 95.1725 89.1727 94.4312 94.4763

[0117] We therefore observe better overall performance with the method 20 proposed in this disclosure compared to the Gaussian algorithm.

[0118] Example 2: Cataract-101 Dataset

[0119] The second example is based on a Cataract-101 dataset, containing one hundred and one (101) videos of cataract surgeries performed by four (4) different surgeons. Such a dataset is described in the publication [SCHOEFFMANN 2018] (Abstract).

[0120] The surgical videos in the Cataract-101 dataset present a sequence of eleven distinct surgical phases, including a phase associated with downtime (in particular, cataract surgery has ten distinct (active) surgical phases and one downtime phase): None (PI, downtime phase), Incision (P2), Viscous Agent Injection (P3), Rhexis (P4), Hydrodissection (P5), Phacoemulsification (P6), Irrigation and Aspiration (P7), Capsule Polishing (P8), Lens Implant Setting Up (P9), Viscous Agent Removal (P10), and Tonifying and Antibiotics (PI 1). Similar to the Cholec80 dataset, the sequence of such surgical phases, and thus the theoretical flow of cataract surgery, can be illustrated using a cataract surgery flow diagram as shown in [Fig. 7]. The unfolding graph shown in [Fig.Figure 7 then allows us to represent the theoretical possibilities of sequencing between the surgical phases among the 11 surgical phases Pl-Pll observable in cataract surgery. For example, starting from the initial state (represented by Start in Figure 7) corresponding to the beginning of the cataract operation, the sequence graph illustrates that two phases... The following are possible to start the operation: a dead time phase (None) PI (in 96% of cases) or a surgical incision phase (Incision) P2 (in 4% of cases).

[0121] In particular, such a sequence graph illustrates, in comparison with the cholecystectomy sequence graph in [Fig. 4], the diversity of surgical phases and possible sequences depending on the type of surgical operation considered. Indeed, here, we observe greater flexibility in the possible sequencing of the surgical phases, whereas the cholecystectomy operation exhibits a stricter sequencing between the surgical phases.

[0122] As a point of note, the surgical phases considered are not ordered. The elements described in the first example apply to the second example.

[0123] From such a Cataract-101 dataset, a plurality of sequences of thirty-two (32) successive images were extracted, with an extraction frequency of 2.5 images per second. Such a plurality of successive image sequences are labeled and divided into training, validation, and test sets.

[0124] The same pre-trained classification model as for the first example and described above is then trained (specialized by fine tuning) on ​​the Cataract-101 training set, and then validated on the Cataract-101 validation set.

[0125] In the same way as for the Cholec80 dataset, a transition matrix associated with the present example can then be calculated (corresponding to the transition matrix calculated for the cataract surgery represented by the dataset). Such a transition matrix, denoted MT>Cataract-iob, can be calculated from the validation set of the Cataract-101 dataset and defined as follows:

[0126] 'NA NA 0.96 0.99 0.04 6.3e-3 0.00 2.7e-4 0.00 6.8e-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.9e-3 NA 0.00 0.98 1.7e-2 1.7e-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NA 0.00 9. le-4 9.7e-7 1.3e-2 0.00 1.3e-4 0.00 0.00 1.3e-2 0.00 0.00 0.00 NA 0.00 0.00 1.7e-4 0.99 8.5e-3 0.00 0.00 0.00 8.4e-5 0.00 0.00 0.00 0.00 NA 0.00 0.00 0.00 0.00 0.99 1.2e-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NA 0.00 0.00 l.le-4 0.00 0.00 1.00 2.6e-3 0.00 0.00 0.00 0.00 0.00 NA 0.00 0.00 6.3e-4 0.00 0.00 0.00 0.99 7.0e-3 0.00 0.00 0.00 0.00 NA 0.00 0.00 1.8e-2 0.00 0.00 0.00 3.7e-3 0.98 1.9e-4 0.00 0.00 0.00 NA 3.8e-3 0.00 2.8e-4 0.00 0.00 1.4e-4 0.00 0.00 0.99 l.le-2 0.00 0.00 NA 6.1e-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 8.0e-4 0.00 NA 8.5e-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.5e-3 0.99 0.00 \NA NA NA NA NA NA NA NA AU 'VA VA NA NA

[0127] The MT> cataract 101 transition matrix is ​​a 13x13 matrix with rows representing the starting states (current states or possible surgical phases on . a given image) and the columns represent the arrival states (next states, possible surgical phases in the following image). Thus, the coefficient at position i, j (row i, column j) represents the average probability of transitioning from state i to state j between two successive images of the Cataract-101 validation set (such a transition can be understood as a stagnation in the same state when i equals j). Such states include an initial state (represented by Start in [Fig. 7]), corresponding to the beginning of the surgical operation, the surgical phases Pl-Pll and a final state (represented by End on the [Fig.7]), corresponding to the end of the surgical operation.

[0128] Similarly, a confusion matrix (not detailed here) can be calculated for the second example.

[0129] The implementation of method 20 on the data of the present example is similar to that of the first example.

[0130] The following table gives a summary of the comparative average performance of process 20 and the Gaussian algorithm on the Cataract-101 dataset. Jaccard Algorithm Accuracy Precision Recall Procedure 20 (proposed in this disclosure) 80.91 90.04 88.35 88.13 Gaussian ([CADENE 2016]) 79.61 92.34 83.27 84.28

[0131] The following table gives the detailed scores by predicted surgical phase (numbered according to [Fig.7]) of the comparative performance of the procedure 20 and the Gaussian algorithm on the Cataract-101 dataset. Procedure 20 (proposed in this disclosure) Gaussian Algorithm ([CADENE 2016]) Surgical Phases Jaccard Precision Recall Jaccard Precision Recall PI 72.4027 80.9249 82.9738 75.9315 84.0690 85.7554 P2 76.1418 91.5747 79.5010 74.2002 87.2836 80.8290 P3 76.2342 83.5272 85.4539 81.2424 94.2249 90.9431 P4 95.2858 97.2371 96.4944 96.3047 98.1428 96.7086 P5 90.0247 92.4760 95.3933 91.4843 93.836 95.3573 P6 97.6229 98.2644 98.8250 97.6891 99.7078 97.4719 P7 96.2835 97.0598 97.7151 95.3960 95.6580 98.4390 P8 88.0190 86.1480 96.9070 89.9395 88.1408 98.0103 P9 93.9096 95.8653 94.7464 86.9860 88.6776 94.9086 P10 73.7818 91.8067 73.9418 86.4942 96.2475 88.6519 Fold 30.2976 56.9979 67.5 0 0 0 Average 80.9094 88.3529 88.1320 79.6062 83.27 84.2796

[0132] It should be noted that the examples described above are purely illustrative and not intended to limit the implementation of this disclosure and the steps described in Figures 2 and 3. In particular, other datasets and other classification models (pre-trained and trained and then validated) can be implemented within the framework of the proposed process 20. List of documents cited

[0133] [CADENE 2016]: Cadène et al. December 2016, “M2CAI Workflow Challenge: Convolutional Neural Networks with Time Smoothing and Hidden Markov Model for Video Frame Classification”

[0134] [JIN 2021]: Y. Jin et al. March 2021, IEEE Transactions on Medical Imaging, “Temporal Memory Relation Network for Workflow Recognition from Surgical Video”

[0135] [TRAN 2019]: D. Tran et al. November 2019, “Video Classification with Channel-Separated Convolutional Networks”

[0136] [TWINANDA 2016] : A.P. Twinanda et al. May 2016, « EndoNet: A Deep Architecture for Récognition Tasks on Laparoscopie Videos »

[0137] [SCHOEFFMANN 2018] : K. Schoeffmann et al. June 2018, MMSys, « Cataract-101 :video dataset of 101 cataract surgeries »

Claims

Demands

1. Method (20) of processing a surgical video to associate a sequence of images extracted from said video with distinct surgical phases (P1-P11), comprising: - (S211) an extraction of said sequence of images from the video, said images being associated with distinct successive instants, - (S212) a classification of said images, by associating said images with predicted surgical phases from a predefined set of surgical phases, implemented by a classification model, - (S224) a correction of the classification by a hidden Markov model (S223), implemented on the predicted surgical phases as observations, the actual surgical phases associated with the sequence of images being hidden states of said hidden Markov model.

2. Method (20) according to claim 1, wherein the classification model is previously trained (S33), validated (S34) and preferably tested (S35) from a set of labeled images, said labeled images corresponding to a succession of images associated with known surgical phases.

3. Method (20) according to claim 2 and further comprising, before the correction step (S224): - (S221) a calculation of a transition matrix associated with the distinct surgical phases from at least the known surgical phases associated with the labeled images, - (S222) a calculation of a confusion matrix from the classification model implemented on at least a part of the labeled images for a calibration step, and wherein the correction of the classification (S224) depends on the transition matrix and the confusion matrix.

4. Method (20) according to claim 3, wherein coefficients of the transition matrix depend on at least one element among: a type of surgical operation captured in the surgical video, a scheduling of the surgical phases of the predefined set of surgical phases.

5. Method (20) according to any one of claims 3 and 4, wherein the hidden Markov model is applied with the transition matrix as the transition matrix of the hidden states of said model and with the confusion matrix as the distribution matrix of the observations of said model.

6. A method (20) according to any one of claims 3 to 5, wherein coefficients of the confusion matrix and / or the transition matrix are smoothed by Laplace smoothing.

7. A method (20) according to any one of the preceding claims, wherein the classification correction (S224) includes a maximization of a total likelihood of a sequence of hidden states associated with the image succession, said maximization resulting in corrected surgical phases associated with the images in the image succession.

8. Method (20) according to claim 7, wherein said maximization is implemented via a Viterbi algorithm.

9. Device comprising means for acquiring surgical video and a computer configured to implement the method (20) according to any one of the preceding claims.

10. Computer program comprising instructions for carrying out the method (20) according to any one of claims 1 to 8 when this program is executed by a processor.

11. A non-transient, computer-readable recording medium on which is recorded a program for implementing the method (20) according to any one of claims 1 to 8 when this program is executed by a processor.

Images