Endoscopic control method, model and training method

Abstract

Embodiments of the present application relate to an endoscope control method, a model and a training method. The endoscope control model training method comprises: obtaining a first sample number of sample frame sequences and corresponding target field of view requirements in a target sample video; training a field of view judgment sub-model based on the sample frame sequences and the target field of view requirements, so that the field of view judgment sub-model is used to generate corresponding target field of view requirements according to each sample frame sequence; obtaining a second sample number of sample image frames, corresponding lens control sequences and target field of view requirements in the target sample video; training an action control sub-model based on the sample image frames, the lens control sequences and the target field of view requirements, so that the action control sub-model is used to generate corresponding lens control sequences according to each sample image frame and the corresponding target field of view requirements. The technical scheme of the embodiments of the present application can reduce the difficulty of model training while improving the performance of the model, and improve the operation stability, safety and intelligent level of the endoscopic surgery.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an endoscope control method, model and training method. Background Technology

[0002] Endoscopic minimally invasive surgery is a high-end minimally invasive surgical procedure that utilizes endoscopic visualization technology and establishes surgical approaches based on the body's natural cavities. It generally has the core advantages of minimal trauma, intraoperative visualization, and rapid postoperative recovery. It is widely used in the clinical treatment of diseases in multiple disciplines such as otolaryngology, skull base, and neurosurgery. For example, endoscopic sinus surgery uses the natural nasal cavity and sinus passages as surgical approaches and is widely used in the treatment of chronic sinusitis, nasal polyps, nasal skull base tumors, and related skull base diseases.

[0003] The core feature of endoscopic minimally invasive surgery is the real-time acquisition of surgical field images of human cavities and adjacent key tissues using endoscopic equipment. This allows surgeons to perform precise treatment of diseased tissues while minimizing external trauma. Such surgeries involve narrow operating spaces and complex surgical field layers, with the operating area often adjacent to important blood vessels, nerves, bones, and functional organs. Therefore, the stability of the surgical field, operational precision, and control of safety boundaries are all crucial during the procedure.

[0004] In recent years, the development of embodied intelligence technology has provided new technical pathways for high-precision surgical robots. Existing methods include autonomous navigation based on safe space constraints, motion restriction based on virtual force constraints, and instrument tracking and field of view adjustment based on visual servoing. These methods can achieve endoscopic motion control to a certain extent, but they are mostly concentrated on field of view tracking or local motion constraints in fixed modes, and it is difficult to autonomously adjust the endoscope holding strategy according to the continuous changes in the surgical process. Summary of the Invention

[0005] This application provides an endoscopic control method, model, and training method, aiming to achieve the model's ability to understand the surgical process, judge visual field requirements, and generate lens control strategies through decomposed learning and efficient collaboration. This reduces the difficulty of model training while improving model performance, thereby enhancing the operational stability, safety, and intelligence level of endoscopic surgery. It has extremely high practical application potential, especially for transnasal endoscopic surgery scenarios with limited space and large depth.

[0006] In a first aspect, embodiments of this application provide a method for training an endoscopic control model, including: In the target sample video, obtain a first number of sample frame sequences and the target field of view requirements corresponding to each sample frame sequence; A field of view judgment sub-model is trained based on the sample frame sequence and the target field of view requirement, so that the field of view judgment sub-model is used to generate the corresponding target field of view requirement according to each sample frame sequence; In the target sample video, a second number of sample image frames, a lens control sequence corresponding to each sample image frame, and the target field of view requirement are obtained; The motion control sub-model is trained based on the sample image frames, the lens control sequence, and the target field of view requirements, so that the motion control sub-model can generate the corresponding lens control sequence according to each sample image frame and the corresponding target field of view requirements.

[0007] Optionally, obtaining a first number of sample frame sequences and the target field of view requirements corresponding to each sample frame sequence in the target sample video includes: In the target sample video, a temporal continuous image frame of the length of the first number of target frame sequences is obtained as the sample frame sequence; each sample frame sequence is composed of the target field-view frame at the temporal end of the sample frame sequence and the temporal preceding associated frame of the target field-view frame. Obtain the target field of view label for each target field of view frame as the target field of view requirement for each sample frame sequence; the target field of view label includes a field of view distance label and a field of view direction label.

[0008] Optionally, obtaining the target field-of-view labels for each of the target field-of-view frames includes: Obtain a reference field-of-view frame within the continuous temporal domain of the target field-of-view frame; Based on the temporal distance between each reference field of view frame and the target field of view frame, the field of view fusion weight of each reference field of view frame is determined. Based on the field of view fusion weight, the target field of view frame and the reference field of view frame are fused by weighted field of view labels to obtain the target field of view label of the target field of view frame.

[0009] Optionally, training the field-of-view judgment sub-model based on the sample frame sequence and the target field-of-view requirement includes: The sample frame sequence is input into the field of view determination sub-model to obtain the field of view prediction result output by the field of view determination sub-model; Based on the consistency of logical value classification and the difference in logical value distribution between the predicted field of view and the target field of view requirement, the target cross-entropy loss is obtained. Based on the difference in logical value distribution between the vision prediction result and the corresponding temporal adjacent preceding prediction result, calculate the label jump cross-entropy. Based on the visual label jump position of the sample frame sequence, obtain the label consistency constraint weight of the label jump cross-entropy; Calculate the vision judgment loss based on the target cross-entropy loss, the label jump cross-entropy, and the label consistency constraint weight; The parameters of the vision judgment sub-model are iteratively updated based on the vision judgment loss.

[0010] Optionally, obtaining the target cross-entropy loss based on the logistic value classification consistency and logistic value distribution difference between the vision prediction result and the target vision requirement includes: If the predicted field of view is consistent with the logical value classification result of the target field of view requirement, the target cross-entropy loss is determined to be 0. If the predicted field of view is inconsistent with the logical value classification result of the target field of view requirement, the cross-entropy between the predicted field of view and the logical value of the target field of view requirement is calculated as the target cross-entropy loss.

[0011] Optionally, obtaining the label consistency constraint weight of the label jump cross-entropy based on the view label jump position of the sample frame sequence includes: Search for field-view label jump frames in the target field-view frame and the reference field-view frame corresponding to the sample frame sequence; If the field-of-view label jump frame does not exist, the label consistency constraint weight is set to 1; When the field of view label jump frame exists, the field of view fusion weight corresponding to the field of view label jump frame is obtained, and the label consistency constraint weight is determined to be the difference between 1 and the field of view fusion weight.

[0012] Optionally, training the motion control sub-model based on the sample image frames, the camera control sequence, and the target field of view requirements includes: The sample image frame and the target field of view requirement are input into the motion control sub-model to obtain the predicted control sequence output by the motion control sub-model; Calculate the motion control loss based on the target statistical difference between the predictive control sequence and the lens control sequence; The parameters of the motion control sub-model are iteratively updated based on the motion control loss.

[0013] Optionally, training the motion control sub-model based on the sample image frames, the camera control sequence, and the target field of view requirements further includes: The sample image frame and the target field of view requirement are input into the motion control sub-model to obtain the posterior latent space distribution features generated by the motion control sub-model based on the shot control sequence; Based on the aforementioned posterior latent space distribution characteristics, the distribution divergence loss is calculated; The parameters of the action control sub-model are iteratively updated based on the distributed divergence loss.

[0014] Secondly, embodiments of this application provide an endoscopic control model, trained using the endoscopic control model training method provided in any embodiment of this application, comprising: a visual field judgment sub-model and a motion control sub-model; wherein... The field of view determination sub-model is used to generate target field of view requirements based on the associated frame sequence at the current moment; The motion control sub-model is used to generate a lens control sequence based on the target field of view requirements and the real-time image frame at the current moment.

[0015] Thirdly, embodiments of this application provide an endoscope control method, executed by an endoscope control model trained by the endoscope control model training method provided in any embodiment of this application, including: Generate target field of view requirements based on the current frame sequence; Based on the target field of view requirements and the real-time image frames at the current moment, a lens control sequence is generated.

[0016] This application provides an endoscopic control method, model, and training method. By acquiring a first number of sample frame sequences and corresponding target field-of-view requirements from a target sample video, a field-of-view judgment sub-model is trained to generate corresponding target field-of-view requirements based on each sample frame sequence. A second number of sample image frames, corresponding lens control sequences, and target field-of-view requirements are acquired from the target sample video, and a motion control sub-model is trained to generate corresponding lens control sequences based on each sample image frame and corresponding target field-of-view requirements. This solves the problem in existing technologies where it is difficult to autonomously adjust the endoscope holding strategy according to continuous changes in the surgical process. It achieves decomposed learning and efficient collaboration of the model's ability to understand the surgical process, judge field-of-view requirements, and generate lens control strategies. This reduces the difficulty of model training while improving model performance, enhancing the operational stability, safety, and intelligence level of endoscopic surgery. It has particularly high practical application potential for transnasal endoscopic surgery scenarios with limited space and significant depth. Attached Figure Description

[0017] Figure 1 A flowchart of an endoscopic control model training method provided in Embodiment 1 of this application; Figure 2 A schematic diagram of an endoscope control architecture provided in Embodiment 1 of this application; Figure 3 This is a schematic diagram of the structure of an endoscope control model provided in Embodiment 2 of this application; Figure 4 This is a flowchart of an endoscope control method provided in Embodiment 3 of this application. Detailed Implementation

[0018] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0019] Example 1 Figure 1 This is a flowchart of an endoscopic control model training method provided in Embodiment 1 of this application. This embodiment is applicable to training endoscopic control models to learn active perception, understanding, and collaboration capabilities. The method can be executed by an endoscopic control model training device, which can be implemented in hardware and / or software and is generally integrated into an electronic device, such as a computer. The method specifically includes: Step 110: Obtain the first number of sample frame sequences and the target field of view requirements corresponding to each sample frame sequence in the target sample video.

[0020] The target sample video can be selected from endoscopic videos of actual endoscopic surgeries, with each frame labeled by professionals. The initial sample size can be a pre-set amount of sample data based on factors such as the total duration of the target sample video, the richness of the surgical scene, and the evenness of the distribution of various visual field labels. The sample frame sequence can be a sequence of multiple temporally continuous image frames obtained from the target sample video. Its sequence length can be pre-determined according to the temporal perception requirements of the surgical field. Typically, the model needs to rely on multiple consecutive frames to completely capture the dynamic changes in instrument movement, tissue deformation, and surgical area switching, balancing the accuracy of temporal feature extraction with the computational time of the model. The target visual field requirements corresponding to any sample frame sequence can describe the endoscopic field of view required for the surgical operation in that sample frame sequence, and can be determined based on the visual field annotations of the image frames in the sample frame sequence.

[0021] In an optional implementation, obtaining a first number of sample frame sequences and the target field of view requirements corresponding to each sample frame sequence in the target sample video includes: obtaining a first number of target frame sequence length of continuous image frames in the target sample video as a sample frame sequence; and obtaining the target field of view label of each target field of view frame as the target field of view requirement of each sample frame sequence.

[0022] Each sample frame sequence consists of the target field-of-view frame at the end of the time domain and the preceding time-of-time associated frame of the target field-of-view frame. The total number of frames of the target field-of-view frame and the preceding time-of-time associated frame is the length of the target frame sequence. Preferably, the length of the target frame sequence can be set to 32 frames, corresponding to a video length of 6.4 seconds. Then, the sample frame sequence consists of 31 consecutive preceding time-of-time associated frames and the last target field-of-view frame.

[0023] Furthermore, the target field of view (PDV) label can be a label describing the endoscopic field of view presented by the target PPV frame, and can be based on pre-annotation by professionals. Specifically, the target PPV label includes a PPV distance label and a PPV direction label. The PPV distance label is used to mark the depth position of the endoscopic field of view at the time of image frame acquisition, and can include categories such as "extremely near," "proximal," "middle," "far," "extremely far," and "other." The PPV direction label is used to mark the planar orientation of the endoscopic field of view at the time of image frame acquisition, and can include categories such as "middle," "directly left," "lower left," "directly down," "lower right," "directly right," "upper right," "directly up," "upper left," and "other." The target PPV requirement for any sample frame sequence can be determined based on the target PPV labels of the target PPV frames within it.

[0024] In one optional implementation, obtaining the target field-view label for each target field-view frame includes: obtaining reference field-view frames within the continuous temporal range of the target field-view frame; determining the field-view fusion weight of each reference field-view frame based on the temporal distance between each reference field-view frame and the target field-view frame; and performing weighted fusion of the target field-view frame and the reference field-view frames based on the field-view fusion weight to obtain the target field-view label for the target field-view frame.

[0025] The continuous temporal range of the visual field can be a range of specific time lengths before and after the target visual field frame. Generally, it can be assumed that the exact moment when the semantic change of the endoscopic visual field occurs within this range does not need to be precise. In general, image frames within this range can be used as reference visual field frames. Preferably, the continuous temporal range of the visual field can also be determined by the number of reference visual field frames. For example, the temporal range consisting of the first three frames and the last three frames of the target visual field frame can be determined as the continuous temporal range of the visual field, thereby obtaining six frames within this range as reference visual field frames.

[0026] Furthermore, based on the temporal distance between each reference field-of-view frame and the target field-of-view frame, the field-of-view fusion weight of each reference field-of-view frame can be determined. This weight reflects the degree of correlation and proximity between the endoscopic field of view of the corresponding reference frame and the endoscopic field of view of the target field-of-view frame. Generally, the larger the temporal distance of a reference field-of-view frame from the target field-of-view frame, the lower its field-of-view fusion weight, and the weight of the target field-of-view frame can be 1. Accordingly, in the field-of-view label weighted fusion based on the field-of-view fusion weight, the pre-labeled field-of-view labels of each frame can be fused and calculated according to their corresponding weight ratios in the form of a label one-hot code logical value (logit) to obtain the target field-of-view label of the target field-of-view frame.

[0027] In a preferred embodiment, the first three frames and the last three frames of the target field of view are selected to form a reference field of view frame, and the weights of each frame are determined according to the temporal order as (0.2, 0.4, 0.6, 1.0, 0.6, 0.4, 0.2). The unique hot code logic value of the tag of each frame is calculated by weighting and superimposing according to the weight ratio to obtain the target field of view tag of the target field of view frame.

[0028] The above implementation uses time-series soft labels to construct training labels, which can mitigate the performance loss caused by the ambiguity of the data scenario itself. Furthermore, a specialized data filtering strategy can effectively improve the model's ability to identify long-tail labels and its sensitivity to label switching scenarios.

[0029] Step 120: Train the field of view judgment sub-model based on the sample frame sequence and the target field of view requirements, so that the field of view judgment sub-model can be used to generate the corresponding target field of view requirements according to each sample frame sequence.

[0030] Among them, the field of view judgment sub-model can be a pre-constructed neural network model with a defined structure and parameters to be learned, which can be used to automatically identify the ideal endoscopic field of view required for the current surgical operation based on the input of a continuous multi-frame endoscopic image sequence.

[0031] In an optional implementation, the vision judgment sub-model can use a transformer encoder as the backbone, with the encoder consisting of two 8-layer attention layers. For image frames, a ResNet50 convolutional network can be used to extract image features. This network uses pre-trained weights from the existing work SurgVLP (Surgical Vision-Language Pre-training) as initial weights. Each frame's image feature value is pooled and used as a token. The tokens from all frames are concatenated in their original sequence order to obtain a 32-bit image feature token sequence. Additionally, a learnable cls token (Classification Token) is added and concatenated to the end of the sequence. The concatenated sequence is input into the transformer architecture and forward-propagated to obtain output features. The output features corresponding to the cls token are mapped to a 16-dimensional tensor using a linear layer, which is then divided into two labels of 6 and 10 dimensions. Each label is passed through a softmax layer to generate two predicted labels. This part of the model has a dropout of 0.1.

[0032] Specifically, each image frame in an arbitrary sample frame sequence records the endoscopic movement and image acquisition process during a corresponding time period in the target sample video. The corresponding target field of view requirement records the field of view that the endoscope needs to be controlled to obtain during that surgical procedure. Training the field of view judgment sub-model based on the sample frame sequence and target field of view requirement allows the sub-model to perceive the scene and behavior during the surgical process, understand the corresponding surgical intent, and learn to determine the required endoscopic field of view. For example, the field of view judgment sub-model can detect the surgical instruments performing fine cutting operations based on the sample frame sequence and learn the target field of view requirement constituted by the "proximal intermediate field of view" marked in the last frame of the sample frame sequence in the time domain, thereby iteratively updating the corresponding model network parameters. After training, the field of view judgment sub-model can determine the currently required endoscopic field of view based on real-time surgical operations, which is used to guide the motion control sub-model in determining the motion control strategy for the endoscope.

[0033] In one optional implementation, training a vision judgment sub-model based on a sample frame sequence and target vision requirements includes: inputting the sample frame sequence into the vision judgment sub-model to obtain the vision prediction result output by the vision judgment sub-model; obtaining the target cross-entropy loss based on the logistic value classification consistency and logistic value distribution difference between the vision prediction result and the target vision requirements; calculating the label jump cross-entropy based on the logistic value distribution difference between the vision prediction result and the corresponding temporally adjacent preceding prediction result; obtaining the label consistency constraint weight of the label jump cross-entropy based on the vision label jump position of the sample frame sequence; calculating the vision judgment loss based on the target cross-entropy loss, the label jump cross-entropy, and the label consistency constraint weight; and iteratively updating the parameters of the vision judgment sub-model based on the vision judgment loss.

[0034] The process involves inputting a sample frame sequence into the visual field determination sub-model. Based on the current network results and parameters, the sub-model outputs a corresponding visual field prediction, typically a logistic distribution. This prediction is then compared with the target visual field labels (also in a logistic distribution) in the target visual field requirements, using a highest-distribution category comparison and cross-entropy calculation. This yields logistic classification consistency and logistic distribution difference, reflecting the discrepancy between the current visual field determination sub-model's prediction of endoscopic visual field requirements and the actual requirements. The target cross-entropy loss determined accordingly can then guide the iterative update of the visual field determination sub-model's parameters.

[0035] In an optional implementation, the target cross-entropy loss is obtained based on the consistency of the logical value classification and the difference in logical value distribution between the field of view prediction result and the target field of view requirement. This includes: when the field of view prediction result is consistent with the logical value classification result of the target field of view requirement, the target cross-entropy loss is determined to be 0; when the field of view prediction result is inconsistent with the logical value classification result of the target field of view requirement, the logical value cross-entropy between the field of view prediction result and the target field of view requirement is calculated as the target cross-entropy loss.

[0036] In this model, the logistic classification result can refer to the category with the highest logistic probability (top 1). If the top 1 of the vision prediction result equals the top 1 of the ground truth label in the target vision requirement, the target cross-entropy loss can be directly set to 0. If the top 1 of the vision prediction result does not equal the top 1 of the ground truth label in the target vision requirement, the logistic cross-entropy between the two is calculated and used as the target cross-entropy loss. By using a modified cross-entropy loss to control label behavior and only taking the top 1 category as the desired result, the model can avoid being dragged down to a more uniform state between the two distributions by cross-entropy in scenarios with ambiguous boundaries, thus enabling the isolated model to make effective and stable judgments.

[0037] Furthermore, the temporal adjacent preceding prediction result, i.e., the prediction result of the previous frame sequence corresponding to the vision prediction result, can serve as training supervision, stabilizing the model's label output for nearby moments. Specifically, the corresponding label jump cross-entropy can be obtained by calculating the distribution cross-entropy, which is used to quantify the difference in logistic value distribution between the vision prediction result and the temporal adjacent preceding prediction result. Based on the vision label jump position of the sample frame sequence, the importance of the label consistency constraint before and after the difference in training can be further determined, and the label consistency constraint weight in which the label jump cross-entropy constitutes the training loss can be obtained accordingly. The closer the vision label jump position of the sample frame sequence is to that moment, the lower the degree of label consistency constraint required at that moment, and the lower the corresponding label consistency constraint weight. For example, if that moment is the vision label jump position, the label consistency constraint weight can be 0, corresponding to the fact that it does not need to be consistent with the label of the previous moment.

[0038] In an optional implementation, the label consistency constraint weight of the label jump cross-entropy is obtained based on the view label jump position of the sample frame sequence, including: searching for view label jump frames in the target view frame and reference view frame corresponding to the sample frame sequence; if the view label jump frame does not exist, the label consistency constraint weight is determined to be 1; if the view label jump frame exists, the view fusion weight corresponding to the view label jump frame is obtained, and the label consistency constraint weight is determined to be the difference between 1 and the view fusion weight.

[0039] In this context, a field-of-view (POV) label-jumping frame can be an image frame where the pre-labeled POV label changes compared to the POV label of the previous temporal frame. If no POV label-jumping frame exists, it indicates that the label consistency between the preceding and following time steps needs to be constrained. In this case, the label consistency constraint weight can be set to 1, meaning the label jump cross-entropy is directly used as the label consistency loss, proportionally superimposed with the target cross-entropy loss to form the POV decision loss. If a POV label-jumping frame exists, the corresponding POV fusion weight is obtained. The label consistency constraint weight is set to 1 minus the difference between the POV fusion weight and the label consistency constraint weight. Since the POV fusion weight decreases with increasing temporal distance, the corresponding label consistency constraint weight can increase with increasing temporal distance, thus determining the ratio of the label jump cross-entropy to the target cross-entropy loss when constituting the POV decision loss.

[0040] The above implementation uses temporal consistency loss to control label behavior. In some ambiguous situations, the model may determine that certain labels have similar probabilities of occurrence, leading to rapid switching between different category outputs within a short period. Temporal consistency loss encourages the model to adhere to previous category judgments, making it less prone to repeated label jumps.

[0041] In one optional implementation, training loss functions can be calculated for the prediction results of the field distance label and the field direction label, respectively; further optionally, since the depth label is considered to be more valuable than the orientation label, in the final loss function, the target cross-entropy loss and label consistency loss for the field direction label can be multiplied by a weight less than 1, for example, 0.6.

[0042] In one optional implementation, the training data can be cleaned and filtered before training the vision judgment sub-model. Specifically, many data points are consecutive frames, which can lead to information redundancy. Furthermore, some long operations may cause a certain label category to appear for an excessively long time, and the overall data statistics also show a significant imbalance in category distribution. During data cleaning, the data is categorized into two types based on whether label jumps occur in the reference vision frame window of the sequence: those with label jumps and those without. All jump data is retained, while half of the non-jump data is removed. After aggregating the data, half of the data for the most frequently occurring category is removed, resulting in the final training sample data.

[0043] In an optional implementation, data augmentation can be performed during training, specifically including image scaling, cropping, color dithering, and zero-value filling. For example, during training, the image is first scaled to a height of 256, then cropped from the centered region of (384, 256), and then randomly cropped again with a random area of ​​60% to 100% and a random aspect ratio of 2:33:2 before being scaled to 224×224. Next, random color dithering is performed, including a 20% perturbation of brightness, contrast, and saturation, and a 5% perturbation of hue. Finally, zero values ​​are used to fill the image into a 224×224 square.

[0044] In an optional implementation, the optimizer used during training is AdamW (Adam with Weight Decay), which can be configured with a global learning rate of 2e-5, betas default (0.9, 0.999), and weight decay of 0.1; the learning rate for the image feature encoder is multiplied by 0.1; cls_token and pos_embed do not undergo weight decay. Gradient clipping is set to max_norm=40, L2 norm. Mixed-precision training is used. The learning rate strategy involves a linear warm-up for the first 1% of steps, increasing linearly from (0.1 × base learning rate) to 2e-5, followed by cosine annealing for the remaining steps, decaying from 2e-5 to 5e-8. Training lasts for a total of 2 epochs.

[0045] Step 130: Obtain the second number of sample image frames, the lens control sequence corresponding to each sample image frame, and the target field of view requirements from the target sample video.

[0046] The second sample size can be the same as or different from the first sample size, and can be a preset sample data volume based on the corresponding model learning requirements. A sample image frame can be an image frame acquired at a single moment in the target sample video. The lens control sequence corresponding to any sample image frame can be a time-domain sequence describing the endoscopic motion process within a certain period after the acquisition time of the sample image frame during the actual surgery; optionally, it can be composed of motion information corresponding to eight consecutive frames after the sample image frame; further optionally, the trajectory prediction model Endo-FASt3r can be used to infer the target sample video to obtain the estimated results of the camera motion parameters corresponding to each frame. The target field of view requirement corresponding to any sample image frame can be marked accordingly after the surgeon determines the required field of view at the corresponding moment of the sample image frame; optionally, the target field of view requirement of that frame can be determined by weighted fusion of the labels of the reference field of view frames of any frame.

[0047] Step 140: Train the motion control sub-model based on sample image frames, lens control sequences, and target field of view requirements, so that the motion control sub-model can generate corresponding lens control sequences according to each sample image frame and the corresponding target field of view requirements.

[0048] Among them, the motion control sub-model can be a pre-built neural network model with a defined structure and parameters to be learned. It can be used to determine the target field label of the sub-model based on the input single-frame endoscopic image and the field of view, predict the endoscopic motion control strategy at subsequent time steps, and output the corresponding endoscopic control sequence.

[0049] In one optional implementation, the motion control sub-model can use a transformer encoder-decoder structure as the backbone, with each encoder and decoder consisting of two layers and eight heads. Further optionally, at each time step, for the current video frame, a convolutional network ResNet50 can be used to extract image features. This convolutional network uses pre-trained weights from the existing work SurgVlp as initial weights and extracts the image feature sequence. Further optionally, the motion control sub-model can extract the posterior latent space distribution token, i.e., latent z, from the shot control sequence and the current pose qpos (Generalized JointPosition) in the sample image frame through a 7-layer, 8-head VAE (Variational Autoencoder) module. This latent z is concatenated with the image feature sequence and input into the transformer main model; the posterior latent z can be trained to approximate a Gaussian distribution, so that during model inference, it can sample from the Gaussian distribution as a latent space token and use it to generate the motion sequence.

[0050] Optionally, in the transformer decoder layer of the main model, a learnable query token of length 8 is used as input to the query side to obtain the feature sequence output, preferably of length 8. This part of the transformer architecture model has a dropout of 0.3, and the VAE module has a dropout of 0.1.

[0051] Optionally, the feature sequence can be fed into a MoE decoding head structure with two expert groups. The outputs of all experts in each expert group are gated and weighted to obtain the final output of that group. The two final outputs from the two expert groups are then fused and mapped using an MLP (Multilayer Perceptron) network. Each feature is mapped to a 6-dimensional tensor, corresponding to motion parameters at one time scale. A sequence of length 8 corresponds to 8 motion parameters, representing the motion parameters at 8 time scales after prediction. Correspondingly, the gating signal for the expert group can be obtained by fusing features with external labels. For an action sequence of length 8, an equal-length sequence of external labels for the next 8 frames is taken. The labels are mapped to an embedding through a linear layer. This embedding is then dropped by 0.3 and mapped again through a linear layer to logits of the same dimension as the number of label categories. These logits are multiplied by a weight of 0.6. Simultaneously, the feature data of the main model is also mapped to logits through a linear layer. The sum of these two logits is then processed through a softmax layer to obtain the weights of each expert in the expert group. For dual-label tasks, the weights of the two expert groups will be obtained using the same pattern. Further, optionally, to combat the averaging problem and strengthen the posterior influence, the latent z of the latent space will be projected through a linear layer and added to all learnable query tokens in the decoder as a bias term.

[0052] In an optional implementation, training a motion control sub-model based on sample image frames, camera control sequences, and target field of view requirements includes: inputting sample image frames and target field of view requirements into the motion control sub-model to obtain a predicted control sequence output by the motion control sub-model; calculating a motion control loss based on the target statistical difference between the predicted control sequence and the camera control sequence; and iteratively updating the parameters of the motion control sub-model based on the motion control loss.

[0053] In this process, the sample image frames and their corresponding target field of view requirements are input into the motion control sub-model. Based on the current model network structure and parameters, the motion control sub-model can derive the motion control strategy required to move the endoscopic field of view from the state corresponding to the sample image frame to the state required by the target field of view, thus obtaining a predicted control sequence. The target statistical difference between the predicted control sequence and the lens control sequence can be a numerical value that quantifies the difference between the two, preferably the L1 distance between them. Based on the target statistical difference, a motion control loss is obtained, which allows the motion control sub-model to gradually bring the predicted control sequence closer to the actual lens control sequence through iterative parameter updates during training.

[0054] In one optional implementation, different L1 distances can be configured to form the weights of the training loss based on the positions of the predicted control sequence and the camera control sequence. Specifically, for an 8-frame camera control sequence, the weight value of the first 2 frames in the temporal domain can be set to 1, and the weight value of the last 6 frames can be set to 0.2.

[0055] In an optional implementation, training the motion control sub-model based on sample image frames, camera control sequences, and target field of view requirements further includes: inputting the sample image frames and camera control sequences into the motion control sub-model to obtain the posterior latent space distribution features generated by the motion control sub-model according to the camera control sequences; calculating the distribution divergence loss based on the posterior latent space distribution features; and iteratively updating the parameters of the motion control sub-model based on the distribution divergence loss.

[0056] The sample image frames and camera control sequences are input into the motion control sub-model. The motion control sub-model extracts the posterior latent space distribution features from the current pose corresponding to the sample image frames and the camera control sequences. Specifically, this can refer to the posterior latent space distribution token, i.e., latent z. The distribution divergence loss of this posterior latent space distribution feature can be the KL divergence loss. Therefore, iteratively updating the parameters of the motion control sub-model based on the distribution divergence loss can make the posterior latent space distribution feature approximate a Gaussian distribution. Optionally, the weight of the KL divergence loss can be set to 1.5, and it can have 0.01 free bits, meaning that no KL divergence loss is added when the information content of latent z is less than 0.01.

[0057] In one optional implementation, the training data can be cleaned and filtered before training the motion control sub-model. Specifically, during most periods of the data, the camera remains relatively still, which can easily lead to model collapse during training. Data cleaning is performed to remove data with low motion intensity and data with significant jitter during motion. The filtering process involves taking the sequence of 17 frames before and after each sample, determining the longest monotonically unidirectional cumulative interval for each dimension, obtaining the maximum cumulative net motion value, and then taking the largest cumulative net value across all dimensions and directions as the motion score for that window. Data with motion intensity exceeding a certain threshold and data with label jumps within the window are used as training data, while the remaining data is filtered out. Before training begins, the data is also subjected to z-score normalization.

[0058] In one optional implementation, data augmentation can be added during the training of the motion control sub-model, specifically including frame scaling, cropping, and padding. For example, during training, the original frame is first scaled to 256 on the shorter side, then randomly cropped and scaled again to 224x224. During validation and testing, random cropping is not performed; instead, the frame is centered at 224x196 and then padded to form a 224x224 square.

[0059] In an optional implementation, training can employ mixed-precision AdamW for the optimizer, for example, setting the learning rate to 1e-5, Betas defaults to (0.9, 0.999), and weight decay to 0.05; for the pre-trained image backbone in the encoder, the learning rate is multiplied by 0.2. Gradient clipping is set to max_norm=40, L2 norm. The learning strategy involves linear warm-up, increasing linearly from (0.1 × base learning rate) to 2e-5, with warm-up steps accounting for 1% of the total steps. After warm-up, cosine annealing is used, gradually decreasing the learning rate from 1e-5 to 5e-7. A total of twenty training epochs are performed, scheduled by iteration steps rather than by training epochs. In the first training epoch, the KL divergence loss weights are set to 0, and the retrained model weights are retrained in another epoch, this time with the KL divergence loss added, making the posterior latent z distribution approximate a Gaussian distribution.

[0060] For example, Figure 2 This is a schematic diagram of an endoscope control architecture provided in Embodiment 1 of this application. Figure 2 As shown, based on the above implementation method, a trained endoscopic control model can be obtained, which is divided into an upper-layer visual field judgment sub-model and a lower-layer motion control sub-model. This effectively splits the task; the upper-layer model only needs to make judgments based on label understanding, while the lower-layer model only needs to output motion based on the visual field content of a single frame. This design clearly reduces the training difficulty of the two models and improves the model performance. At the same time, this design effectively maintains the mutual cooperation ability between models, integrating the process understanding ability of the upper-layer model through the expected visual field labels, and effectively transmitting it to the lower-layer model to guide its motion.

[0061] The technical solution of this embodiment obtains a first number of sample frame sequences and corresponding target field of view requirements from the target sample video, trains a field of view judgment sub-model, and enables the field of view judgment sub-model to generate corresponding target field of view requirements based on each sample frame sequence. It also obtains a second number of sample image frames, corresponding lens control sequences, and target field of view requirements from the target sample video, and trains a motion control sub-model, enabling the motion control sub-model to generate corresponding lens control sequences based on each sample image frame and corresponding target field of view requirements. This solves the problem of existing technologies being unable to autonomously adjust the endoscope holding strategy according to the continuous changes in the surgical process. It realizes the separate learning and efficient collaboration of the model's ability to understand the surgical process, judge field of view requirements, and generate lens control strategies, reducing the difficulty of model training while improving model performance, and improving the operational stability, safety, and intelligence level of endoscopic surgery.

[0062] Example 2 Figure 3 This is a schematic diagram of the structure of an endoscope control model provided in Embodiment 2 of this application, as shown below. Figure 3 As shown, the endoscopic control model includes: a field of view determination sub-model 310 and a motion control sub-model 320, wherein, The field of view determination sub-model 310 is used to generate target field of view requirements based on the associated frame sequence at the current moment; wherein, the associated frame sequence at the current moment can refer to the sequence of endoscopic image frames acquired in the temporal domain order within a period of time adjacent to the current moment. The motion control sub-model 320 is used to generate a lens control sequence based on the target field of view requirements and the real-time image frame at the current moment.

[0063] The endoscopic control model provided in this application embodiment can be trained by the endoscopic control model training method provided in any embodiment of this application, and can execute the endoscopic control method provided in any embodiment of this application, possessing the corresponding functional modules and beneficial effects of executing the method.

[0064] Example 3 Figure 4 This is a flowchart of an endoscope control method provided in Embodiment 3 of this application. This embodiment is applicable to situations where the endoscope is automatically controlled during surgery. This method can be executed by the endoscope control model provided in any embodiment of this application. The model can be trained by the endoscope control model training method provided in any embodiment of this application. The method specifically includes: Step 410: Generate target field of view requirements based on the associated frame sequence at the current moment.

[0065] Step 420: Generate a lens control sequence based on the target field of view requirements and the real-time image frame at the current moment.

[0066] The technical solution of this embodiment generates target field of view requirements based on the associated frame sequence at the current moment, and generates a lens control sequence based on the target field of view requirements and the real-time image frames at the current moment. This solves the problem that existing technologies are unable to autonomously adjust the lens holding strategy according to the continuous changes in the surgical process. It realizes the separate learning and efficient collaboration of the model's ability to understand the surgical process, judge field of view requirements, and generate lens control strategies. This reduces the difficulty of model training while improving model performance, and enhances the operational stability, safety, and intelligence level of endoscopic surgery.

[0067] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0068] It is worth noting that in the embodiments of the above-mentioned endoscopic control model, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0069] Although this application has been described in detail above with general descriptions, specific embodiments, and experiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of this application fall within the scope of protection claimed in this application.

Claims

1. A method for training an endoscopic control model, characterized in that, include: In the target sample video, obtain a first number of sample frame sequences and the target field of view requirements corresponding to each sample frame sequence; A field of view judgment sub-model is trained based on the sample frame sequence and the target field of view requirement, so that the field of view judgment sub-model is used to generate the corresponding target field of view requirement according to each sample frame sequence; In the target sample video, a second number of sample image frames, a lens control sequence corresponding to each sample image frame, and the target field of view requirement are obtained; The motion control sub-model is trained based on the sample image frames, the lens control sequence, and the target field of view requirements, so that the motion control sub-model can generate the corresponding lens control sequence according to each sample image frame and the corresponding target field of view requirements.

2. The method according to claim 1, characterized in that, The step of obtaining a first number of sample frame sequences and the target field of view requirements corresponding to each sample frame sequence in the target sample video includes: In the target sample video, a temporal continuous image frame of the length of the first number of target frame sequences is obtained as the sample frame sequence; each sample frame sequence is composed of the target field-view frame at the temporal end of the sample frame sequence and the temporal preceding associated frame of the target field-view frame. Obtain the target field of view label for each target field of view frame as the target field of view requirement for each sample frame sequence; the target field of view label includes a field of view distance label and a field of view direction label.

3. The method according to claim 2, characterized in that, The step of obtaining the target field-of-view labels for each of the target field-of-view frames includes: Obtain a reference field-of-view frame within the continuous temporal domain of the target field-of-view frame; Based on the temporal distance between each reference field of view frame and the target field of view frame, the field of view fusion weight of each reference field of view frame is determined. Based on the field of view fusion weight, the target field of view frame and the reference field of view frame are fused by weighted field of view labels to obtain the target field of view label of the target field of view frame.

4. The method according to claim 3, characterized in that, The training of the field-of-view judgment sub-model based on the sample frame sequence and the target field-of-view requirement includes: The sample frame sequence is input into the field of view determination sub-model to obtain the field of view prediction result output by the field of view determination sub-model; Based on the consistency of logical value classification and the difference in logical value distribution between the predicted field of view and the target field of view requirement, the target cross-entropy loss is obtained. Calculate the label jump cross-entropy based on the logical value distribution difference between the vision prediction result and the corresponding temporal adjacent preceding prediction result; Based on the visual label jump position of the sample frame sequence, obtain the label consistency constraint weight of the label jump cross-entropy; Calculate the vision judgment loss based on the target cross-entropy loss, the label jump cross-entropy, and the label consistency constraint weight; The parameters of the vision judgment sub-model are iteratively updated based on the vision judgment loss.

5. The method according to claim 4, characterized in that, The step of obtaining the target cross-entropy loss based on the logistic value classification consistency and logistic value distribution difference between the predicted field of view and the target field of view requirement includes: If the predicted field of view is consistent with the logical value classification result of the target field of view requirement, the target cross-entropy loss is determined to be 0. If the predicted field of view is inconsistent with the logical value classification result of the target field of view requirement, the cross-entropy between the predicted field of view and the logical value of the target field of view requirement is calculated as the target cross-entropy loss.

6. The method according to claim 4, characterized in that, The step of obtaining the label consistency constraint weight of the label jump cross-entropy based on the view label jump position of the sample frame sequence includes: Search for field-view label jump frames in the target field-view frame and the reference field-view frame corresponding to the sample frame sequence; If the field-of-view label jump frame does not exist, the label consistency constraint weight is set to 1; When the field of view label jump frame exists, the field of view fusion weight corresponding to the field of view label jump frame is obtained, and the label consistency constraint weight is determined to be the difference between 1 and the field of view fusion weight.

7. The method according to claim 1, characterized in that, The training of the motion control sub-model based on the sample image frames, the camera control sequence, and the target field of view requirements includes: The sample image frame and the target field of view requirement are input into the motion control sub-model to obtain the predicted control sequence output by the motion control sub-model; Calculate the motion control loss based on the target statistical difference between the predictive control sequence and the lens control sequence; The parameters of the motion control sub-model are iteratively updated based on the motion control loss.

8. The method according to claim 7, characterized in that, The training of the motion control sub-model based on the sample image frames, the lens control sequence, and the target field of view requirements also includes: The sample image frame and the shot control sequence are input into the motion control sub-model to obtain the posterior latent space distribution features generated by the motion control sub-model based on the shot control sequence. Based on the aforementioned posterior latent space distribution characteristics, the distribution divergence loss is calculated; The parameters of the action control sub-model are iteratively updated based on the distributed divergence loss.

9. An endoscopic control model, characterized in that, The endoscopic control model is trained by the training method described in any one of claims 1-8, and includes: a visual field judgment sub-model and a motion control sub-model; wherein, The field of view determination sub-model is used to generate target field of view requirements based on the associated frame sequence at the current moment; The motion control sub-model is used to generate a lens control sequence based on the target field of view requirements and the real-time image frame at the current moment.

10. An endoscope control method, characterized in that, The endoscopic control model trained by the endoscopic control model training method according to any one of claims 1-8 is executed, including: Generate target field of view requirements based on the current frame sequence; Based on the target field of view requirements and the real-time image frames at the current moment, a lens control sequence is generated.

Images