A vesicoperitoneal fold cut open site positioning navigation system and methods of use thereof

Abstract

The application relates to the technical field of surgical assistance, and discloses a bladder peritoneal reflection incision site positioning navigation system and a use method thereof, which comprises a scene recognition module, a cut-in point recognition module and a real-time tracking module.The scene recognition module is used for acquiring a real-time surgery video stream of a target surgery process, and performing region recognition according to the real-time surgery video stream to obtain a surgery region recognition result.The cut-in point recognition module is used for, in the case that the surgery region recognition result indicates that the real-time surgery video stream contains a target surgery region, performing cut-in point recognition on the real-time surgery video stream to determine a target cut-in point in the target surgery region.The real-time tracking module is used for performing real-time tracking on the target cut-in point to obtain a cut-in point mark, so as to perform cut-in point navigation on the target surgery process.The application has the beneficial effects of improving the positioning accuracy of a bladder peritoneal reflection incision cut-in point and improving the safety and standardization degree of a surgery process.

Description

Technical Field

[0001] This application relates to the field of surgical assistance technology, and in particular to a vesicoperitoneal reflection incision site positioning and navigation system and its usage method. Background Technology

[0002] Vesicoperitoneal reflection incision is a crucial gynecological surgical procedure. It involves making an incision at the peritoneal reflection at the top of the bladder during surgery. This creates a surgical pathway to the target surgical area, such as the uterus, cervix, or bladder itself, while protecting the bladder from damage. It is commonly used in procedures such as total hysterectomy, radical hysterectomy, repair of uterine scar diverticulum after cesarean section, and removal of lesions from cesarean scar pregnancy. The core of vesicoperitoneal reflection incision lies in the precise selection of the initial incision point. Incision points that are too high or too low increase intraoperative risks and affect the normal progress of the surgery. Currently, the location of the incision point in vesicoperitoneal reflection incision relies heavily on the surgeon's subjective experience and visual judgment, lacking objective quantitative standards. Summary of the Invention

[0003] This application provides a vesicoperitoneal reflection incision site localization navigation system and its usage method. It utilizes real-time surgical video stream of the target surgical procedure for region identification and incision point identification, and generates incision point markers to guide the surgeon to the incision point, thereby improving the localization accuracy of the vesicoperitoneal reflection incision point and enhancing the safety and standardization of the surgical procedure.

[0004] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a vesicoperitoneal reflection incision site positioning and navigation system, the system including a scene recognition module, an entry point recognition module and a real-time tracking module; The scene recognition module is used to acquire a real-time surgical video stream of the target surgical procedure and perform region recognition based on the real-time surgical video stream to obtain the surgical region recognition result. The entry point identification module is used to identify the entry point of the real-time surgical video stream when the surgical area identification result indicates that the real-time surgical video stream contains a target surgical area, so as to determine the target entry point in the target surgical area. The real-time tracking module is used to track the target incision point in real time and obtain the incision point identifier to provide incision point navigation for the target surgical procedure.

[0005] The vesicoperitoneal reflection incision site localization and navigation system proposed in this application includes a scene recognition module, an incision point recognition module, and a real-time tracking module. The scene recognition module is used to identify the target surgical area based on the real-time surgical video stream of the target surgical procedure, determining whether the target surgical area appears in the real-time surgical video stream. The incision point recognition module is used to identify the target incision point in the target surgical area when the target surgical area appears in the real-time surgical video stream. The real-time tracking module is used to track the target incision point in real time and generate an incision point identifier for incision point navigation. Compared with related technologies, this application can utilize the real-time surgical video stream to perform real-time analysis of the surgical process, automatically identify the incision point of the vesicoperitoneal reflection incision and prompt the surgeon, ensuring that the surgeon can quickly and accurately determine the optimal initial incision position during the operation, reducing the operational difficulty of the vesicoperitoneal reflection incision, and improving the safety and standardization of the surgical procedure.

[0006] Optionally, the scene recognition module includes: The field of view feature extraction unit is used to extract field of view features from any video frame in the real-time surgical video stream to obtain the region-related features of any video frame. The region correlation identification unit is used to perform region correlation identification on any video frame based on the region correlation features, determine whether any video frame contains the target surgical region, and obtain the surgical region identification result.

[0007] Optionally, the region-related identification unit includes: The region probability classification subunit is used to perform logical mapping on any video frame based on the region-related features to obtain the logical probability that any video frame contains the target surgical region. A threshold comparison subunit is used to compare the logical probability with a preset classification threshold. If the logical probability exceeds the preset classification threshold, it is determined that the real-time surgical video stream contains the target surgical region; otherwise, it is determined that the real-time surgical video stream does not contain the target surgical region.

[0008] Optionally, the entry point identification module includes: The video image segmentation unit is used to segment the real-time surgical video stream using an incision point localization model in order to determine the specific range of the target surgical region from the real-time surgical video stream. A region entry point identification unit is used to identify entry points in the target surgical region to obtain the target entry point.

[0009] Optionally, the entry point identification module further includes a model training unit for training the entry point localization model, comprising: The dataset acquisition subunit is used to acquire medical image segmentation datasets and incision point localization datasets; The model pre-training subunit is used to train the initial localization model by segmenting key regions using the medical image segmentation dataset to obtain an intermediate localization model. The main training subunit of the model is used to perform region segmentation optimization training and entry point localization training on the intermediate localization model using the entry point localization dataset to obtain the entry point localization model.

[0010] Optionally, the real-time tracking module includes: An initial coordinate acquisition unit is used to acquire the initial region coordinates of the target incision point in an initial video frame; wherein, the initial video frame is the video frame in the real-time surgical video stream in which the target incision point is determined; An image motion calculation unit is used to perform image motion calculation on subsequent video frames based on the initial video frame to obtain the image motion vector of the subsequent video frame relative to the initial video frame. The coordinate motion update unit is used to update the initial region coordinates according to the image motion vector, obtain the subsequent region coordinates of the target entry point in the subsequent video frame, and move the entry point identifier to the subsequent region coordinates.

[0011] Optionally, the real-time tracking module further includes: A field-of-view abrupt change detection unit is used to perform field-of-view abrupt change detection on subsequent video frames based on the initial video frame, and obtain the abrupt change detection result of the subsequent video frames; The coordinate initialization unit is used to, when the mutation detection result indicates that a sudden change in the field of view occurs in the subsequent video frame, use the subsequent video frame as the initial video frame to redetermine the target entry point.

[0012] Secondly, embodiments of this application provide a method for using a vesicoperitoneal reflection incision site positioning and navigation system, applied to the system described in any of the above embodiments; the method includes: The scene recognition module acquires the real-time surgical video stream of the target surgical procedure, and performs region recognition based on the real-time surgical video stream to obtain the surgical region recognition result. If the surgical region identification result indicates that the real-time surgical video stream contains a target surgical region, the incision point identification module is used to identify the incision point in the real-time surgical video stream to determine the target incision point in the target surgical region. The target incision point is tracked in real time by a real-time tracking module to obtain an incision point identifier, thereby providing incision point navigation for the target surgical procedure.

[0013] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the system described in any of the above embodiments.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to implement the system described in any one of the above embodiments.

[0015] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which are used to cause a computer to implement the system described in any of the above embodiments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A block diagram of a vesicoperitoneal reflection incision site positioning and navigation system provided in an embodiment of this application; Figure 2 A step diagram illustrating the usage of the vesicoperitoneal reflection incision site positioning and navigation system provided in this application embodiment; Figure 3 This is a diagram illustrating the usage steps of the scene recognition module in this application embodiment; Figure 4 This is a diagram illustrating the steps of using the region-related identification unit in the embodiments of this application; Figure 5 This is a diagram illustrating the usage steps of the entry point identification module in the embodiments of this application; Figure 6 This is a diagram illustrating the training steps of the incision point localization model in this application embodiment; Figure 7 This is a diagram illustrating the usage steps of the real-time tracking module in the embodiments of this application; Figure 8 This is a diagram illustrating the usage steps of the real-time tracking module in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Vesicoperitoneal reflection incision is a crucial gynecological surgical procedure. It involves making an incision at the peritoneal reflection at the top of the bladder during surgery. This creates a surgical pathway to the target surgical area, such as the uterus, cervix, or bladder itself, while protecting the bladder from damage. It is commonly used in procedures such as total hysterectomy, radical hysterectomy, repair of uterine scar diverticulum after cesarean section, and removal of lesions from cesarean scar pregnancy. The key to successful vesicoperitoneal reflection incision lies in the precise selection of the initial incision point. If the incision point is too high, difficulties in space dissection and tissue layer identification may arise, hindering the normal progress of the surgery. Conversely, if the incision point is too low, the vesicoperitoneal reflection incision procedure is highly susceptible to damaging the bladder wall, leading to intraoperative and postoperative complications and increasing surgical risks.

[0020] In related techniques, the location of the incision point for vesicoperitoneal reflection incision mainly relies on the surgeon's subjective experience and visual judgment, which may lead to errors in the location of the incision point, making it difficult to guarantee the safety and consistency of the surgical procedure. In addition, traditional teaching methods for vesicoperitoneal reflection incision often rely on experiential mnemonics such as "yellow and white alternating, yellowish area," lacking objective quantitative standards, making it difficult for beginners to accurately master the technique and prolonging the training time required.

[0021] To address the aforementioned issues, this application provides a vesicoperitoneal reflection incision site localization and navigation system and its usage method, comprising: a scene recognition module for acquiring a real-time surgical video stream of the target surgical procedure and performing region recognition based on the real-time surgical video stream to obtain a surgical region recognition result; an incision point recognition module for performing incision point recognition on the real-time surgical video stream when the surgical region recognition result indicates that the real-time surgical video stream contains the target surgical region, thereby determining the target incision point within the target surgical region; and a real-time tracking module for real-time tracking of the target incision point to obtain an incision point identifier for incision point navigation of the target surgical procedure.

[0022] The vesicoperitoneal reflection incision site localization and navigation system provided in this application includes a scene recognition module, an incision point recognition module, and a real-time tracking module. The scene recognition module is used to identify the region based on the real-time surgical video stream of the target surgical procedure and determine whether the target surgical region appears in the real-time surgical video stream. The incision point recognition module is used to identify the target incision point in the target surgical region when the target surgical region appears in the real-time surgical video stream. The real-time tracking module is used to track the target incision point in real time and generate an incision point identifier for incision point navigation.

[0023] Compared with related technologies, this application can use real-time surgical video streams to analyze the surgical process in real time, automatically identify the incision point for vesicoperitoneal reflection incision and prompt the surgeon, ensuring that the surgeon can quickly and accurately determine the optimal initial incision position during the operation, reducing the difficulty of vesicoperitoneal reflection incision and improving the safety and standardization of the surgical process.

[0024] Reference Figure 1 As shown, this embodiment provides a vesicoperitoneal reflection incision site localization and navigation system. The system includes a scene recognition module, an incision point recognition module, and a real-time tracking module. The scene recognition module acquires a real-time surgical video stream of the target surgical procedure and performs region recognition based on the real-time surgical video stream to obtain a surgical region recognition result. The incision point recognition module, when the surgical region recognition result indicates that the real-time surgical video stream contains the target surgical region, performs incision point recognition on the real-time surgical video stream to determine the target incision point within the target surgical region. The real-time tracking module tracks the target incision point in real-time to obtain an incision point identifier for incision point navigation of the target surgical procedure.

[0025] The target surgical procedure can be any surgical procedure including a vesicoperitoneal reflection incision. The real-time surgical video stream can be image data acquired in real time during the target surgical procedure, including the real-time surgical area. The real-time surgical area can include both the target surgical area and unrelated surgical areas, which can be areas other than the target surgical area. The real-time surgical video stream can be acquired based on the surgeon's field of vision during the target surgical procedure. The surgeon's field of vision can be the area within which the surgeon directly or indirectly observes the target surgical area.

[0026] The target surgical area can be the area where the surgeon needs to perform operations such as vesicoperitoneal reflection incision during the target surgery. The target incision point can be the optimal initial incision point for vesicoperitoneal reflection incision operation identified in the video frame containing the target surgical area.

[0027] Specifically, during the target surgical procedure, the image acquisition device is oriented towards the target surgical area. Continuous real-time image acquisition is performed within the surgeon's observation range using the image acquisition device. The image acquisition results are then standardized to obtain a real-time surgical video stream of the target surgical procedure, which is then input into the scene recognition module. The image acquisition device may include laparoscopes and hysteroscopes, and the standardized processing operations may include, but are not limited to, timestamp alignment, format normalization, and noise reduction.

[0028] Furthermore, in the scene recognition module, the acquired real-time surgical video stream is divided into multiple video frames. Region recognition is performed sequentially on each video frame according to time order to determine whether the target surgical region is contained in each video frame, thus obtaining the surgical region recognition result. In some embodiments, the process of performing region recognition based on the real-time surgical video stream to obtain the surgical region recognition result can be performed using an image classification model. This model classifies regions based on the image features of each video frame and outputs the surgical region recognition result. For example, the image classification model can be a binary classification model based on a deep convolutional neural network, such as EfficientNet-B3.

[0029] The surgical region identification result can be a binary classification result, including whether the target surgical region is contained in the video frame or not. If the target surgical region is identified in any video frame, it means that the image acquisition device capturing the real-time surgical video stream is aligned with the target surgical region and has acquired a clear image of it. The optimal initial incision point for the vesicoperitoneal reflection incision operation can be identified in that video frame. If the target surgical region is not identified in any video frame, it means that the image acquisition device capturing the real-time surgical video stream is not aligned with the target surgical region and has only acquired image data of an unrelated surgical region, which is irrelevant to the target surgical region and the vesicoperitoneal reflection incision operation. In this case, that video frame is skipped, and region identification is performed again after the next video frame is input.

[0030] Understandably, before entry point identification, multiple video frames in the real-time surgical video stream are pre-identified as regions. If a video frame does not contain the target surgical region, that video frame is skipped without entry point identification. Only when the video frame contains the target surgical region does the entry point identification process begin. This achieves scene filtering during entry point localization and navigation. On the one hand, it reduces the probability of false alarms in operations unrelated to the target surgical region, improving the reliability and trustworthiness of entry point navigation. On the other hand, it also reduces the computational load in unrelated operations, optimizing computational resource requirements.

[0031] Furthermore, if the surgical region identification result indicates that the real-time surgical video stream contains the target surgical region, the real-time surgical video stream is input into the incision point identification module. The incision point identification module performs incision point identification on the real-time surgical video stream to identify the target incision point within the target surgical region shown in the real-time surgical video stream, and outputs the coordinates of the target incision point to the real-time tracking module. In some embodiments, the process of performing incision point identification on the real-time surgical video stream to determine the target incision point within the target surgical region can be performed using an incision point localization model. This involves pixel-level fine segmentation and coordinate regression on the video frame containing the target surgical region to determine the specific location of the target incision point within the video frame. For example, the incision point localization model can be a semantic segmentation network based on an encoder-decoder structure, where the encoder-decoder structure can be a U-Net++ architecture with an attention gating mechanism.

[0032] Furthermore, after receiving the coordinates of the target incision point, the real-time tracking module generates an incision point marker at those coordinates to display the location of the target incision point to the surgeon. The target incision point is tracked in real-time in subsequent video frames of the surgical video stream, and the incision point marker is adjusted in real-time based on the tracking results, ensuring that the incision point marker points to the optimal initial incision point for the vesicoperitoneal reflection incision operation. In some embodiments, the real-time tracking of the target incision point can be performed using feature point tracking algorithms, including but not limited to optical flow, particle filtering, and correlation filtering. The incision point marker can be a visual marker with high visual resolution compared to the target surgical area, including but not limited to colored concentric rings with arrow guidance and crosshairs.

[0033] It is understandable that when the surgeon indirectly observes the target surgical area through an image acquisition device, this image acquisition device and the image acquisition device that acquires and displays the real-time surgical video stream can be the same. That is, the surgeon uses the real-time surgical video stream as their field of view and performs surgical operations based on the target surgical area displayed in the real-time surgical video stream. In this case, the incision point marker can be superimposed on the display area of ​​the image acquisition device, allowing the surgeon to directly observe the incision point marker, determine the specific location of the target incision point, and improve the indication effect of the target incision point. In other embodiments, auxiliary instructions can also be given to the surgeon using sound or tactile feedback, including but not limited to indicating that the target incision point has been located or that the surgeon's surgical operation is approaching a danger zone.

[0034] The vesicoperitoneal reflection incision site localization and navigation system provided in this embodiment includes a scene recognition module, an incision point recognition module, and a real-time tracking module. The scene recognition module is used to identify the region based on the real-time surgical video stream of the target surgical procedure and determine whether the target surgical region appears in the real-time surgical video stream. The incision point recognition module is used to identify the target incision point in the target surgical region when the target surgical region appears in the real-time surgical video stream. The real-time tracking module is used to track the target incision point in real time and generate an incision point identifier for incision point navigation.

[0035] Compared with related technologies, this application can use real-time surgical video streams to analyze the surgical process in real time, automatically identify the incision point for vesicoperitoneal reflection incision and prompt the surgeon, ensuring that the surgeon can quickly and accurately determine the optimal initial incision position during the operation, reducing the difficulty of vesicoperitoneal reflection incision and improving the safety and standardization of the surgical process.

[0036] As one embodiment of this application, the scene recognition module includes a field-of-view feature extraction unit and a region correlation recognition unit. The field-of-view feature extraction unit is used to extract field-of-view features from any video frame in the real-time surgical video stream to obtain the region correlation features of any video frame. The region correlation recognition unit is used to perform region correlation recognition on any video frame based on the region correlation features, determine whether any video frame contains the target surgical region, and obtain the surgical region recognition result.

[0037] Specifically, in the scene recognition module, the acquired real-time surgical video stream is divided into multiple video frames, and these multiple video frames are sequentially input into the visual field feature extraction unit according to time sequence. For any one of the multiple video frames, the visual field feature extraction unit performs visual field feature extraction on the image data of that video frame to obtain the image features of the real-time surgical region in that video frame, which are used as the region-related features of that video frame. In some embodiments, the process of performing visual field feature extraction on any video frame to obtain the region-related features of that video frame can be performed by feature extraction algorithms, including but not limited to Convolutional Neural Networks (CNN), Region-CNN (RCNN), and Transformer architecture.

[0038] Furthermore, the field-of-view feature extraction unit inputs the region-related features in any video frame into the region-related recognition unit to perform region-related recognition based on the region-related features. In the region-related recognition unit, the real-time surgical region in any video frame is classified according to the region-related features to obtain the classification probability that the real-time surgical region is classified as the target surgical region, and the classification probability is used to determine whether any video frame contains the target surgical region, thus obtaining the surgical region recognition result.

[0039] In some embodiments, the visual field feature extraction unit and the region correlation recognition unit may be configured with an image classification model for extracting visual field features from any video frame to obtain region correlation features of any video frame, and for performing region correlation recognition on any video frame based on the region correlation features to determine whether any video frame contains a target surgical region, thereby obtaining a surgical region recognition result. The image classification model may be a binary classification model based on a deep convolutional neural network, such as EfficientNet-B3.

[0040] The image classification model can be a model pre-trained on a scene classification dataset. The training data in the scene classification dataset can be actual surgical videos, divided into positive and negative samples. Positive samples are actual surgical videos that clearly and completely display the target surgical area, while negative samples are actual surgical videos that include close-ups of instruments, blood coverage, sutures, or other anatomical areas. Video frames in the actual surgical videos related to the target surgical area have corresponding video annotations to indicate whether the target surgical area is clear in that video frame. These annotations can be performed by medical experts or by an automatic annotation tool. The process of obtaining the scene classification dataset can include: filtering all video frames in the actual surgical videos to identify keyframes containing the target surgical area and generating preliminary labels for these keyframes; filtering keyframes based on the clarity or visual information content of the target surgical area in the keyframes to obtain target frames; and combining consecutive target frames into multiple video segments to obtain the actual surgical video, thus obtaining the scene classification dataset.

[0041] An image classification model can include a visual backbone network and a classification head. The visual backbone network extracts region-related features from video frames, while the classification head classifies regions based on these features to obtain surgical region identification results. The visual backbone network can employ a backbone structure based on convolutional neural networks or visual Transformers, combined with feature fusion neck structures to integrate multi-scale features from video frames, characterizing tissue morphology and spatial relationships within the real-time surgical region. The classification head includes fully connected layers and an output layer. The fully connected layers map region-related features to the classification result, and the output layer outputs the classification result as the surgical region identification result. For example, the decision function of the classification head can be expressed as: in, For real-time surgical video stream A video frame at a given moment; For feature extraction functions; This is the weight matrix of the fully connected layer; Use the Sigmoid activation function; This is a bias term.

[0042] As one embodiment of this application, the region correlation identification unit includes a region probability classification subunit and a threshold comparison subunit. The region probability classification subunit is used to perform a logical mapping on any video frame based on region correlation features to obtain the logical probability that any video frame contains the target surgical region. The threshold comparison subunit is used to compare the logical probability with a preset classification threshold. If the logical probability exceeds the preset classification threshold, it is determined that the real-time surgical video stream contains the target surgical region; otherwise, it is determined that the real-time surgical video stream does not contain the target surgical region.

[0043] Specifically, in the region correlation identification unit, the region probability classification subunit receives the region correlation features output by the field-of-view feature extraction unit, and performs a logical mapping on any video frame based on the region correlation features to obtain the logical probability that any video frame contains the target surgical region. In some embodiments, the process of performing a logical mapping on any video frame based on the region correlation features to obtain the logical probability that any video frame contains the target surgical region can be performed by the classification head of an image classification model. The classification head performs a linear transformation on the region correlation features through a fully connected layer to obtain the classification logical value corresponding to the region correlation features, and then uses a Sigmoid activation function to convert the classification logical value into the logical probability that any video frame contains the target surgical region.

[0044] Further, the logical probability output by the threshold comparison subunit receiving the region probability classification subunit is compared with a preset classification threshold to obtain the surgical region identification result. If the logical probability exceeds the preset classification threshold, it is determined that the real-time surgical video stream contains the target surgical region, and the surgical region identification result containing the target surgical region in the real-time surgical video stream is output. If the logical probability does not exceed the preset classification threshold, it is determined that the real-time surgical video stream does not contain the target surgical region and only contains irrelevant surgical regions, and the surgical region identification result not containing the target surgical region in the real-time surgical video stream is output. For example, the preset classification threshold can be 0.8.

[0045] As one embodiment of this application, the incision point recognition module includes a video image segmentation unit and a region incision point recognition unit. The video image segmentation unit is used to segment the real-time surgical video stream using an incision point localization model to determine the specific range of the target surgical region from the real-time surgical video stream. The region incision point recognition unit is used to perform incision point recognition in the target surgical region to obtain the target incision point.

[0046] The video image segmentation unit and the region cut-in point recognition unit can be configured with a cut-in point localization model. This model can include a semantic segmentation network based on an encoder-decoder structure, which can be a U-Net++ architecture with an attention gating mechanism. In some embodiments, the encoder of the cut-in point localization model can employ a VGG16 (Visual Geometry Group 16) backbone network, and the decoder can employ a recursive upsampling pyramid composed of densely connected attention-gated nodes. Each node in this recursive upsampling pyramid can selectively fuse multi-scale features and semantic features from the encoder through an attention mechanism. The semantic segmentation network is further linked to a cut-in point regression head, which performs coordinate regression based on the cut-in point recognition results to return the coordinates of the target cut-in point in the video frame.

[0047] Specifically, when the surgical area recognition result indicates that the real-time surgical video stream contains the target surgical area, the real-time surgical video stream is divided into multiple video frames and output sequentially to the incision point recognition module in chronological order, and the video image segmentation unit receives the multiple video frames of the real-time surgical video stream.

[0048] For any given video frame from multiple video frames, an incision point localization model is used to extract image features from that frame, resulting in its image features. Based on these features, all pixels in that frame are classified into regions, yielding the region classification results for each pixel. Pixels classified as belonging to the same region type are then segmented into the same region. This image segmentation result allows the generation of an image mask for the target surgical region, thereby determining the specific extent of the target surgical region from the real-time surgical video stream for subsequent incision point identification.

[0049] Furthermore, the region entry point recognition unit receives the image segmentation result output by the video image segmentation unit, performs entry point recognition in the target surgical region according to the specific range of the target surgical region, determines the best initial entry point in the target surgical region as the target entry point, and returns the coordinates of the target entry point in any video frame.

[0050] As one embodiment of this application, the entry point recognition module further includes a model training unit for training the entry point localization model, comprising a dataset acquisition subunit, a model pre-training subunit, and a main model training subunit. The dataset acquisition subunit acquires a medical image segmentation dataset and an entry point localization dataset. The model pre-training subunit uses the medical image segmentation dataset to train the initial localization model for key region segmentation, obtaining an intermediate localization model. The main model training subunit uses the entry point localization dataset to perform region segmentation optimization training and entry point localization training on the intermediate localization model, obtaining the entry point localization model.

[0051] The model training unit is used to pre-train the incision point localization model. The incision point localization model can be trained using a medical image segmentation dataset and an incision point localization dataset. The medical image segmentation dataset can be a publicly available medical dataset, including but not limited to Medical Segmentation Decathlon (MSD), MedSegBench, and the interactive medical image segmentation benchmark dataset (IMed-361M).

[0052] The incision point localization dataset can be obtained by hospitals through image segmentation and annotation of actual surgical videos. The methods for obtaining the incision point localization dataset can include: for scene classification datasets, extracting keyframes containing the target surgical region from actual surgical videos used as positive samples; having doctors perform multiple annotations on the keyframes, including region segmentation and incision point annotation, to obtain the incision point localization dataset. Region segmentation of keyframes can involve annotating the portions of the keyframes that belong to the target surgical region, obtaining a region mask for the target surgical region. Incision point annotation can include: determining the optimal initial incision position within the target surgical region.

[0053] In some embodiments, the incision point localization dataset can be continuously updated based on the real-time target surgical procedure. In this case, the incision point localization dataset corresponds to a localization data memory, which can be an incremental, high-value sample-priority storage, and write-in-time intraoperative data management system used to store newly added incision point localization data during the target surgical procedure. It is understood that after the incision point localization dataset is updated, the incision point localization model can be fine-tuned and trained based on the updated dataset, enabling the model to continuously learn from the incision point localization data and improve the clinical safety of the target surgical procedure.

[0054] Specifically, the medical image segmentation dataset and the incision point localization dataset for training are obtained through the dataset acquisition sub-unit. The medical image segmentation dataset is output to the model pre-training sub-unit, and the incision point localization dataset is output to the model main training sub-unit.

[0055] Furthermore, for the untrained initial localization model, the model pre-training subunit uses a medical image segmentation dataset to train the initial localization model for key region segmentation. The initial localization model then segments the training data in the medical image segmentation dataset. Based on the image segmentation results and the data labels of the training data, the parameters of the initial localization model are adjusted using a segmentation loss function to obtain an intermediate localization model. It can be understood that the model pre-training subunit is mainly used to train the initial localization model from the perspective of image segmentation, enabling the initial localization model to learn the general features of the target surgical region. For example, the model pre-training subunit can train the encoder in the initial localization model, and the segmentation loss function can be the Dice loss function.

[0056] Furthermore, the main training subunit receives the intermediate localization model output from the pre-training subunit. It then uses the incision point localization dataset to perform region segmentation optimization training and incision point localization training on the intermediate localization model. The intermediate localization model performs image segmentation and incision point localization on the training data in the incision point localization dataset. Based on the image segmentation and incision point localization results, the parameters of the intermediate localization model are adjusted using segmentation loss functions and localization loss functions to obtain the incision point localization model. It can be understood that the main training subunit is primarily used to train the intermediate localization model at the incision point localization angle, enabling the intermediate localization model to learn how to locate the incision point in the target surgical region and optimize its image segmentation capabilities, thereby improving its adaptability to specific surgical scenarios. For example, the localization loss function can be the Mean Squared Error (MSE) loss function.

[0057] For example, the loss function used to train the intermediate localization model in the main training subunit of the model can be expressed as: in, For the segmentation loss function; A predicted segmentation map for the training data; The true segmentation map of the training data; The localization loss function; The coordinates of the predicted entry point for the training data; The coordinates of the true entry point for the training data; and These are the weighting coefficients. Segmentation loss function. It can be represented as: in, Let be the pixel index of the i-th pixel in any video frame of the training data; The classification result of the i-th pixel in the predicted segmentation map; This represents the classification result of the i-th pixel in the ground truth segmentation map. Localization loss function. It can be represented as: in, This represents the j-th training data; Let be the coordinates of the predicted cut-in point in the j-th training data; Let J represent the coordinates of the true cut-in point in the j-th training data. This represents the total number of keypoints in all training data.

[0058] As one embodiment of this application, the real-time tracking module includes an initial coordinate acquisition unit, an image motion calculation unit, and a coordinate motion update unit. The initial coordinate acquisition unit is used to acquire the initial region coordinates of the target incision point in an initial video frame; wherein, the initial video frame is the video frame in the real-time surgical video stream from which the target incision point is determined. The image motion calculation unit is used to perform image motion calculations on subsequent video frames based on the initial video frame to obtain the image motion vector of the subsequent video frames relative to the initial video frame. The coordinate motion update unit is used to update the initial region coordinates based on the image motion vector to obtain the subsequent region coordinates of the target incision point in subsequent video frames, and move the incision point identifier to the subsequent region coordinates.

[0059] In this process, the first video frame in the real-time surgical video stream that contains the target surgical area and identifies the target incision point is called the initial video frame. Starting from the initial video frame, the target incision point is tracked in real time to generate an incision point identifier. In the initial video frame and subsequent video frames, the incision point identifier is used to provide real-time indication of the target incision point for incision point navigation during the target surgical procedure.

[0060] Specifically, after obtaining the target entry point, the initial coordinate acquisition unit obtains the initial region coordinates of the target entry point in the initial video frame from the entry point recognition module, and outputs the initial region coordinates to the image motion calculation unit.

[0061] Furthermore, for subsequent video frames of the initial video frame, the gradient change (spatial gradient or temporal gradient, etc.) of the subsequent video frame relative to the initial video frame is calculated by the image motion calculation unit, and the motion mode of each pixel in the initial video frame between frames is analyzed based on the gradient change to obtain the image motion vector of the subsequent video frame relative to the initial video frame.

[0062] In some embodiments, the process of calculating image motion for subsequent video frames based on an initial video frame to obtain the image motion vector of the subsequent video frames relative to the initial video frame can be performed using optical flow. It is understood that optical flow estimates the motion of pixels based on pixel brightness changes between the initial and subsequent video frames. The specific process of image motion calculation may include: for any pixel in the initial video frame, calculating the brightness gradient of that pixel in the horizontal direction to obtain the horizontal brightness gradient; calculating the brightness gradient of that pixel in the vertical direction to obtain the vertical brightness gradient; determining the subsequent pixel corresponding to that pixel in the subsequent video frames, calculating the brightness change between that pixel and the subsequent pixel to obtain the temporal brightness gradient; constructing an optical flow constraint equation based on the horizontal, vertical, and temporal brightness gradients, and introducing constraint conditions to solve the optical flow constraint equation to obtain the image motion vector. These constraint conditions may include, but are not limited to, local window consistency constraints, global smoothness constraints, and motion continuity constraints.

[0063] For example, when the introduced constraints are global constraints, the process of solving the optical flow constraint equation can be to calculate the dense optical flow field between the initial video frame and subsequent video frames. The method for calculating the dense optical flow field can be to use a lightweight optical flow network, such as the lightweight RAFT optical flow algorithm (Recurrent All-Pairs FieldTransforms Lite, RAFT-lite). In this case, the image motion vector can be represented as: in, For located The horizontal displacement vector of the pixel; For located The vertical displacement vector of the pixel.

[0064] Furthermore, the image motion calculation unit outputs the image motion vector to the coordinate motion update unit. The coordinate motion update unit updates the initial region coordinates of the target cut-in point based on the image motion vector, determines the subsequent region coordinates of the target cut-in point in subsequent video frames, and moves the cut-in point marker to the position of the subsequent region coordinates, thereby achieving smooth tracking and indication of the target cut-in point.

[0065] Understandably, for multiple video frames following subsequent video frames, image motion calculations can be performed between adjacent video frames to obtain image motion vectors between adjacent video frames. Based on these image motion vectors, the coordinates of the target cutting point in the previous video frame are updated to obtain the updated coordinates of the target cutting point in the next video frame, and the cutting point is then moved to the updated coordinates. For example, the updated coordinates can be expressed as: in, These are the coordinates of the target's entry point in the video frame at the previous moment.

[0066] As one embodiment of this application, the real-time tracking module further includes a field-of-view change detection unit and a coordinate initialization unit. The field-of-view change detection unit is used to perform field-of-view change detection on subsequent video frames based on the initial video frame to obtain the change detection result of the subsequent video frames. The coordinate initialization unit is used to use the subsequent video frame as the initial video frame when the change detection result indicates that a field-of-view change has occurred in the subsequent video frame, so as to redetermine the target entry point.

[0067] Specifically, during actual surgery, the field of view corresponding to the real-time surgical video stream may change drastically due to acquisition conditions, such as rapid camera movement or other environmental conditions. This can cause significant changes in the field of view between adjacent video frames, severely impacting the continuity of the image and making it impossible to accurately track the target entry point. To address this issue, this embodiment, after obtaining the image motion vector, uses a field of view mutation detection unit to detect drastic changes in the field of view between the initial and subsequent video frames. This determines whether a drastic change in the field of view has occurred between the initial and subsequent video frames, obtaining the mutation detection result of the subsequent video frame relative to the initial video frame. In some embodiments, the method for detecting drastic changes in the field of view may include, but is not limited to, optical flow consistency calculation and color histogram distance calculation.

[0068] Furthermore, the coordinate initialization unit receives the abrupt change detection result from the field-of-view abrupt change detection unit. If the abrupt change detection result indicates that no abrupt change in field of view occurs in subsequent video frames, it means that the image data changes smoothly between the initial video frame and subsequent video frames, and the target entry point can be accurately tracked without needing to relocate it. If the abrupt change detection result indicates that abrupt change in field of view occurs in subsequent video frames, it means that a drastic change in field of view has occurred between the initial video frame and subsequent video frames, making it impossible to accurately track the target entry point. In this case, the coordinate data of the current target entry point is discarded, and the subsequent video frame is used as the initial video frame. The process of region identification and entry point identification is then performed again on the new initial video frame to redetermine the target entry point.

[0069] It is understood that this embodiment can detect interference in the real-time surgical video stream during the entry point navigation process. If these interferences significantly affect the entry point navigation, the current entry point tracking process is stopped, and the steps of region identification and entry point identification are re-executed to redetermine the target entry point. This improves the anti-interference capability of the entry point navigation process and enhances the credibility and reliability of the target entry point and entry point identifier.

[0070] Accordingly, please refer to Figure 2 This application provides a method for using a vesicoperitoneal reflection incision site positioning and navigation system, applied to the system described in any of the above embodiments; the method includes: S100. Obtain the real-time surgical video stream of the target surgical procedure through the scene recognition module, and perform region recognition based on the real-time surgical video stream to obtain the surgical region recognition result.

[0071] S200. If the surgical area identification result indicates that the real-time surgical video stream contains the target surgical area, the incision point identification module is used to identify the incision point in the real-time surgical video stream to determine the target incision point in the target surgical area.

[0072] S300. The target incision point is tracked in real time through the real-time tracking module to obtain the incision point identifier, so as to provide incision point navigation for the target surgical procedure.

[0073] Reference Figure 3 As shown, in some optional implementations, a scene recognition module acquires a real-time surgical video stream of the target surgical procedure, and performs region recognition based on the real-time surgical video stream to obtain surgical region recognition results, including: S110. The field of view feature extraction unit extracts the field of view features of any video frame in the real-time surgical video stream to obtain the region-related features of any video frame.

[0074] S120. The region correlation identification unit performs region correlation identification on any video frame based on the region correlation features, determines whether any video frame contains the target surgical region, and obtains the surgical region identification result.

[0075] Reference Figure 4 As shown, in some optional embodiments, the region correlation identification unit performs region correlation identification on any video frame based on region correlation features to determine whether any video frame contains the target surgical region, thereby obtaining the surgical region identification result, including: S122. By using the regional probability classification subunit to perform logical mapping on any video frame based on regional correlation features, the logical probability that any video frame contains the target surgical region is obtained.

[0076] S124. By comparing the logical probability and the preset classification threshold in the threshold comparison sub-unit, if the logical probability exceeds the preset classification threshold, it is determined that the real-time surgical video stream contains the target surgical area; otherwise, it is determined that the real-time surgical video stream does not contain the target surgical area.

[0077] Reference Figure 5 As shown, in some optional implementations, the incision point identification module identifies the incision point in the real-time surgical video stream to determine the target incision point in the target surgical area, including: S210. The video image segmentation unit uses the incision point localization model to perform image segmentation on the real-time surgical video stream to obtain the key incision area in the target surgical region.

[0078] S220. The target entry point is obtained by identifying the entry point in the key entry area through the regional entry point identification unit.

[0079] Reference Figure 6 As shown, in some optional implementations, the cut-point localization model is trained in the following manner: S202. Obtain the medical image segmentation dataset and the incision point localization dataset through the dataset acquisition sub-unit.

[0080] S204. The initial localization model is trained by segmenting key regions using a medical image segmentation dataset through a model pre-training subunit to obtain an intermediate localization model.

[0081] S206. Using the incision point localization dataset, the intermediate localization model is trained by region segmentation optimization and incision point localization through the main training subunit of the model to obtain the incision point localization model.

[0082] Reference Figure 7 As shown, in some optional implementations, the target entry point is tracked in real time by a real-time tracking module to obtain the entry point identifier, including: S310. Obtain the initial region coordinates of the target incision point in the initial video frame through the initial coordinate acquisition unit; wherein, the initial video frame is the video frame in the real-time surgical video stream in which the target incision point is determined.

[0083] S320. The image motion calculation unit performs image motion calculation on subsequent video frames based on the initial video frame to obtain the image motion vector of the subsequent video frame relative to the initial video frame.

[0084] S330. The coordinate motion update unit updates the initial region coordinates according to the image motion vector to obtain the subsequent region coordinates of the target entry point in subsequent video frames, and moves the entry point marker to the subsequent region coordinates.

[0085] Reference Figure 8 As shown, in some optional implementations, the method further includes: S340. The field of view mutation detection unit performs field of view mutation detection on subsequent video frames based on the initial video frame to obtain the mutation detection results of the subsequent video frames.

[0086] S350. When the mutation detection result indicates that a sudden change in the field of view occurs in subsequent video frames, the coordinate initialization unit uses the subsequent video frames as the initial video frames to redetermine the target entry point.

[0087] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application, such as... Figure 9 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.

[0088] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0089] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0090] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0091] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0092] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0093] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0094] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of any embodiment of this application.

[0095] The systems, devices, modules, or sub-modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0096] For ease of description, the above devices are described in terms of function, divided into various sub-modules. Of course, in implementing this application, the functions of each sub-module can be implemented in one or more software and / or hardware.

[0097] Those skilled in the art will understand that embodiments of this application can be provided as a system or a computer program product. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] This application is described with reference to flowchart illustrations and / or block diagrams of devices (systems) and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0102] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A navigation system for locating the site of a vesicoperitoneal reflection incision, characterized in that, The system includes a scene recognition module, an entry point recognition module, and a real-time tracking module; The scene recognition module is used to acquire a real-time surgical video stream of the target surgical procedure and perform region recognition based on the real-time surgical video stream to obtain a surgical region recognition result; wherein, the surgical region recognition result is output by an image classification model based on the image features of each video frame in the real-time surgical video stream. The entry point identification module is used to identify the entry point of the real-time surgical video stream when the surgical area identification result indicates that the real-time surgical video stream contains a target surgical area, so as to determine the target entry point in the target surgical area. The incision point identification module includes a video image segmentation unit, which is used to segment the real-time surgical video stream using an incision point localization model to determine the specific range of the target surgical region from the real-time surgical video stream. The incision point localization model is trained using a medical image segmentation dataset and an incision point localization dataset. The incision point localization dataset is continuously updated according to the real-time target surgical process. After the incision point localization dataset is updated, the incision point localization model is fine-tuned and trained based on the updated incision point localization dataset. The real-time tracking module is used to track the target incision point in real time and obtain the incision point identifier to navigate the target surgical procedure. The real-time tracking module includes a visual field change detection unit and a coordinate initialization unit. The visual field change detection unit is used to perform visual field change detection between the initial video frame and subsequent video frames, determine whether a drastic change in visual field occurs between the initial video frame and the subsequent video frame, and obtain the change detection result of the subsequent video frame relative to the initial video frame. The initial video frame is the video frame in the real-time surgical video stream that first contains the target surgical region and determines the target incision point. The coordinate initialization unit is used to use the subsequent video frame as the initial video frame when the change detection result indicates that a visual field change has occurred in the subsequent video frame, so as to redetermine the target incision point.

2. The system according to claim 1, characterized in that, The scene recognition module includes: The field of view feature extraction unit is used to extract field of view features from any video frame in the real-time surgical video stream to obtain the region-related features of any video frame. The region correlation identification unit is used to perform region correlation identification on any video frame based on the region correlation features, determine whether any video frame contains the target surgical region, and obtain the surgical region identification result.

3. The system according to claim 2, characterized in that, The region-related identification unit includes: The region probability classification subunit is used to perform logical mapping on any video frame based on the region-related features to obtain the logical probability that any video frame contains the target surgical region. A threshold comparison subunit is used to compare the logical probability with a preset classification threshold. If the logical probability exceeds the preset classification threshold, it is determined that the real-time surgical video stream contains the target surgical region; otherwise, it is determined that the real-time surgical video stream does not contain the target surgical region.

4. The system according to claim 1, characterized in that, The entry point identification module includes: A region entry point identification unit is used to identify entry points in the target surgical region to obtain the target entry point.

5. The system according to claim 4, characterized in that, The entry point identification module further includes a model training unit for training the entry point localization model, including: The dataset acquisition subunit is used to acquire medical image segmentation datasets and incision point localization datasets; The model pre-training subunit is used to train the initial localization model by segmenting key regions using the medical image segmentation dataset to obtain an intermediate localization model. The main training subunit of the model is used to perform region segmentation optimization training and entry point localization training on the intermediate localization model using the entry point localization dataset to obtain the entry point localization model.

6. The system according to claim 1, characterized in that, The real-time tracking module includes: An initial coordinate acquisition unit is used to acquire the initial region coordinates of the target incision point in the initial video frame; An image motion calculation unit is used to perform image motion calculation on subsequent video frames based on the initial video frame to obtain the image motion vector of the subsequent video frame relative to the initial video frame. The coordinate motion update unit is used to update the initial region coordinates according to the image motion vector, obtain the subsequent region coordinates of the target entry point in the subsequent video frame, and move the entry point identifier to the subsequent region coordinates.

7. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to implement the system of any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to implement the system of any one of claims 1 to 6.

Images