Interface display control method and device, computer equipment and system, and medium

Abstract

This application provides an interface display control method and apparatus, a computer device and system, and a computer storage medium. The method includes: determining the current display area of ​​a target instrument in a surgical environment field of view; determining an empty field of view area in the surgical environment field of view based on the current display area of ​​the target instrument; obtaining the current position of an interface icon; and if the position determines that the interface icon is obstructing the field of view, adjusting the interface icon to the empty field of view area. This achieves real-time optimization of the position of the interface icon in the surgical environment field of view, supporting the display of a richer number of icons representing the positions and execution states of more types of instruments in the surgical environment field of view, while ensuring that the interface icons do not obstruct the surgeon's view during the operation, allowing the surgeon to perform the operation more effectively.

Description

[0001] This application is a divisional application filed with the Chinese Patent Office on November 8, 2021, with application number 202111316069.1 and title "Interface display control method and apparatus, computer equipment and system, and medium", the full text of which is incorporated herein by reference. Technical Field

[0002] This application relates to the field of surgical robot technology, and in particular to an interface display control method and apparatus, computer equipment and system, and computer-readable storage medium. Background Technology

[0003] Minimally invasive surgery refers to a surgical procedure performed inside the human body using modern medical instruments and equipment such as laparoscopes and thoracoscopes. Compared to traditional surgical methods, minimally invasive surgery has advantages such as less trauma, less pain, and faster recovery. With the advancement of technology, minimally invasive surgical robot technology has gradually matured and is widely used. A minimally invasive surgical robot typically includes a main control panel and slave operating devices. The main control panel sends control commands to the slave operating devices based on the surgeon's instructions, controlling the slave operating devices. The slave operating devices respond to the control commands sent by the main control panel and perform the corresponding surgical operations. The surgeon can observe the surgical environment in real time through the display interface on the main control panel to facilitate better surgical execution. To assist the surgeon in obtaining the necessary information, various icons are usually displayed on the interface to represent the position and status of corresponding instruments.

[0004] However, as the surgical procedure progresses dynamically, the areas of focus for the doctor on the display interface also change dynamically, which may result in icons obscuring these areas and affecting the doctor's operation. Summary of the Invention

[0005] To address the existing technical problems, this application provides an interface display control method and apparatus, computer equipment and system, and computer-readable storage medium capable of dynamically optimizing the position of interface icons.

[0006] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide an interface display control method, applied to a computer device, comprising:

[0008] Determine the current display area of ​​the target instrument in the surgical environment field of view interface;

[0009] Based on the current display area of ​​the target instrument, determine the free field of view area in the surgical environment field of view interface;

[0010] Obtain the current position of the interface icon. If it is determined from the position that the interface icon is obstructed from view, adjust the interface icon to the free view area.

[0011] Secondly, embodiments of this application provide an interface display control device, comprising:

[0012] The determination module is used to determine the current display area of ​​the target instrument in the surgical environment field of view interface;

[0013] The field of view segmentation module is used to determine the free field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument.

[0014] The adjustment module is used to obtain the current position of the interface icon. If it is determined from the position that the interface icon is obstructed from view, the interface icon is adjusted to the free view area.

[0015] Thirdly, embodiments of this application provide a computer device, including a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor. When the computer program is executed by the processor, it implements the interface display control method described in any embodiment of this application.

[0016] Fourthly, embodiments of this application provide a remote-operated medical system, including a computer device as described in any embodiment of this application and a slave operating device connected to the computer device. The slave operating device includes multiple instruments of different types and a drive component for driving the instruments to perform specified actions.

[0017] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by the processor, implements the interface display control method as described in any embodiment of this application.

[0018] The interface display control method and apparatus provided in the above embodiments determine the current display area of ​​the target instrument in the surgical environment field of view, determine the free field of view area in the surgical environment field of view based on the current display area of ​​the target instrument, obtain the current position of the interface icon, and if it is determined that the interface icon is obstructed by the field of view based on the position of the interface icon, then the interface icon is adjusted to the free field of view area. In this way, by distinguishing the free field of view area in the surgical environment field of view according to the doctor's current focus area, and adjusting the interface icon to the free field of view area when it is identified that the interface icon is obstructed by the field of view, the position of the interface icon in the surgical environment field of view is optimized in real time. Under the premise of supporting the surgical environment field of view to display a richer number of icons that represent the positions and execution states of more types of instruments, it also ensures that the interface icon does not obstruct the doctor's field of view during the operation, so that the doctor can perform the operation better.

[0019] In the above embodiments, the computer equipment and system, the computer-readable storage medium, and the corresponding interface display control method embodiments belong to the same concept, and thus have the same technical effects as the corresponding interface display control method embodiments, which will not be repeated here. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of an optional application scenario for the interface display control method in one embodiment;

[0021] Figure 2 This is a flowchart of an interface display control method in one embodiment;

[0022] Figure 3 This is a schematic diagram of the interface icons in one embodiment;

[0023] Figure 4 This is a schematic diagram of the surgical environment field of view in one embodiment;

[0024] Figure 5 This is a schematic diagram of a surgical environment field of view interface that determines the working field of view area and the idle field of view area based on the display area of ​​the target instrument in one embodiment.

[0025] Figure 6 The flowchart of the interface control method is provided as an optional specific example.

[0026] Figure 7 This is a schematic diagram of the interface display control device in one embodiment;

[0027] Figure 8 This is a schematic diagram of the structure of a computer device in one embodiment. Detailed Implementation

[0028] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0031] In the following description, the terms "first, second, and third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0032] Please see Figure 1 This diagram illustrates an optional application scenario of the interface display control method provided in this application embodiment applied to a remote-operated medical system. The remote-operated medical system includes a master operating device 10 and slave operating devices 12 connected to the master operating device 10. The slave operating devices include multiple instruments 121 of different types and drive components 122 that drive the instruments 121 to perform specified actions. Typically, the instruments 121 include endoscopes and surgical tools related to the surgical procedure, such as electrocautery devices, clamps, staplers, scissors, and ultrasound probes. The endoscope is used to acquire images of the surgical environment and send them to the master operating device 10 for display. The master operating device 10 can be used as a master control panel on the doctor's side. For example, the master operating device may include a client for the user, typically including a client for the doctor to remotely observe the current surgical progress and remotely manage the slave operating devices. The client can be an application client (such as a mobile APP) or a web client, which is not limited here. The slave operating devices 12 are used to receive operating instructions issued by the doctor based on the client and perform corresponding actions. Optionally, the instrument 121 includes an electrocautery device, a clamp, a stapler, a scissors, and an ultrasound probe. The drive assembly 122 may include a hinged component (e.g., a joint assembly) connected to the instrument 121, allowing the position and orientation of the instrument 121 to be manipulated with one or more mechanical degrees of freedom relative to an instrument axis. Optionally, the instrument 121 may also include functional mechanical degrees of freedom with further morphological variations, such as clamps that can be opened and closed.

[0033] Please see Figure 2 This application provides an interface display control method applied to a computer device, which may include one or more physically independent and separate intelligent devices with computing capabilities. In an optional specific example, the computer device includes, for example... Figure 1 The main operating device shown, the interface display control method includes the following steps:

[0034] S101, Determine the current display area of ​​the target instrument in the surgical environment field of view interface.

[0035] The target instrument includes one or more pre-defined instruments, such as instrument 1 and instrument 2, or one or more instruments determined according to pre-set rules, such as one or more activated instruments determined based on whether they are in use. The activated instrument includes the instrument currently being used in the surgery; it can be one or more of a plurality of instruments included in the operating equipment. The surgical environment field of view includes visual scene information for the surgeon to observe the current state of the surgery in real time, such as the position of the activated instrument on the human body, the current action of the activated instrument, and the characteristics of the surgical site on the human body.

[0036] Optionally, taking a computer device executing the interface display control method described in this application embodiment as the main operating device, the main operating device acquires surgical environment field-of-view images through an endoscope. The surgical environment field-of-view interface can be a display page in the client of the remotely operating medical device that displays the surgical environment field-of-view images acquired in real time by the endoscopic imaging system. Determining the current display area of ​​the activated device in the surgical environment field-of-view interface includes the main operating device determining the imaging area of ​​a designated part of the activated device as the current display area of ​​the activated device in the surgical environment field-of-view interface based on the position of the corresponding image of the activated device in the surgical environment field-of-view interface. It should be noted that the imaging area of ​​the designated part of the activated device can include the imaging area of ​​the end effector of the activated device, or an imaging area determined by expanding outward from the end effector of the activated device according to a preset strategy.

[0037] S103, determine the free field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument.

[0038] The idle field of view refers to the area in the surgical environment displayed on the surgical scene view interface that the surgeon does not need to focus on. The main operating device displays the surgical environment view image acquired by the endoscope system on the client's surgical scene view interface, making it easier for the surgeon to clearly and completely observe various indicators in the current surgical environment. As the surgery continues and changes dynamically, the surgeon's focus on image information in different areas of the surgical scene view interface may change. For example, the surgeon may gradually shift from being more concerned with the central part of the surgical scene view interface to being more concerned with the right side of the surgical scene view interface. That is, the idle field of view area gradually changes from the outer part surrounding the central part to the left side of the surgical scene view interface.

[0039] In an optional embodiment, determining the free field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument includes:

[0040] Based on the current display area of ​​the target instrument, determine the working field of view area in the surgical environment field of view interface;

[0041] Based on the operational field of view area, the unoccupied field of view area in the surgical environment field of view interface is determined.

[0042] The working field of view includes the area predicted based on the areas the surgeon will focus on during the procedure. The main operating device displays the surgical environment images acquired by the endoscopic system on the client's surgical scene interface, allowing the surgeon to clearly and completely observe various indicators within the current surgical environment. As the surgery progresses and changes dynamically, the surgeon's focus on different areas of the surgical scene interface may shift. For example, the surgeon may initially be more concerned with the central part of the surgical scene interface but gradually shift their attention to the right side, meaning the position of the working field of view within the client's view changes. The idle field of view includes the portion of the surgical scene interface other than the working field of view.

[0043] The computer device predicts the areas of focus for the surgeon during the operation based on the current display area of ​​the target instrument, thereby determining the working field of view and the idle field of view in the surgical environment's field of view interface.

[0044] S105, obtain the current position of the interface icon; if it is determined from the position that the interface icon is obstructed from view, adjust the interface icon to the free view area.

[0045] Interface icons include virtual icons in the surgical environment's field of view used for human-computer interaction, logical operation, interface aesthetics, and the status of operating equipment and / or instruments. For example, the endoscope icon in the surgical environment's field of view represents status information such as the endoscope's rotation angle and mirror angle. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of an endoscope icon in an optional specific example, where the upward field of view of the endoscope icon corresponds to the upward angle of the mirror, such as... Figure 3 .a and Figure 3 As shown in .b; a downward field of view indicates that the mirror angle is downward, as shown in .b. Figure 3 .c and Figure 3 As shown in .d. Rotation of 0 degrees is as follows: Figure 3 .a and Figure 3 .c; Rotation degree indicator, such as Figure 3 b and d (rotation not exceeding ±90 degrees). Optionally, the mirror angle can be obtained from user-set parameters, and the rotation angle is calculated from the position of the rotation motor controlling the endoscope's actions. The surgeon can monitor the endoscope's status in the surgical environment in real time through the state of the endoscope icon in the surgical environment's field of view interface. The computer device obtains the current position of the interface icon, determines whether the icon obstructs the surgeon's view, and if so, adjusts the icon to an empty area within the surgical environment's field of view interface.

[0046] In the above embodiments, by distinguishing the idle field of view areas in the surgical environment field of view interface according to the doctor's current focus area, when the interface icon is obstructed, the interface icon is adjusted to the idle field of view area, thereby realizing real-time optimization of the position of the interface icon in the surgical environment field of view interface. While supporting the surgical environment field of view interface to display a richer number of icons that represent the positions and execution status of more types of instruments, it also ensures that the interface icon does not obstruct the doctor's field of view during the operation, so that the doctor can perform the operation better.

[0047] In some embodiments, determining the free field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument includes:

[0048] Acquire a surgical environment field of view image, perform target detection on the surgical environment field of view image, and determine the position and size of the target instrument contained in the surgical environment field of view image;

[0049] The current display area of ​​the target instrument is determined based on its position and size, and the working field of view and the idle field of view in the surgical environment field of view interface are determined based on the current display area.

[0050] Computer devices can determine the position and size of target instruments within a surgical environment field of view image by performing target detection. The target detection can employ known target detection algorithms that detect whether a specified object is present in the image. Optionally, taking the computer device executing the interface display control method described in this application's embodiments as the main operating device, acquiring the surgical environment field of view image may include: an endoscope acquiring surgical environment field of view video data and sending it to the main operating device; and the main operating device extracting one or more image frames from the surgical environment field of view video data.

[0051] The target device may include an activating device; and / or, the target device may include a designated non-activating device.

[0052] In the above embodiments, the computer device acquires surgical environment field-of-view images and uses a target detection method based on the surgical environment field-of-view images to determine the position and size of the target instrument in the surgical environment field-of-view images, thereby determining the current display area of ​​the target instrument. Based on the current display area of ​​the target instrument, the parts that the doctor will focus on during the surgical operation are predicted, thereby determining the working field of view area and the idle field of view area in the surgical environment field of view interface.

[0053] Optionally, acquiring a field-of-view image of the surgical environment, performing target detection on the field-of-view image, and determining the position and size of the target instrument contained in the field-of-view image includes:

[0054] A surgical environment field of view image is obtained through a neural network model, and target detection is performed on the surgical environment field of view image to determine the position and size of the target instrument contained in the surgical environment field of view image.

[0055] A neural network model based on image recognition is used to perform target detection on surgical environment field images. The neural network model can be a known neural network architecture trained on, such as a convolutional neural network (Fast R-CNN) based on the fast image recognition algorithm FastYOLO (YOUONLYLOOKONCE). This target detection algorithm using a neural network model on surgical environment field images can achieve end-to-end detection of the position and size of target instruments within the surgical environment field images, ensuring rapid target detection and recognition with high accuracy.

[0056] In some embodiments, before acquiring the field image of the surgical environment using a neural network model, performing target detection on the field image, and determining the position and size of the target instrument contained in the field image, the process includes:

[0057] Construct the initial neural network model;

[0058] The initial neural network model is trained using a training sample set of surgical environment field images containing target object annotations to obtain a trained neural network model; the target object annotations include the position and size annotations of the target instruments contained in the field images.

[0059] Neural network models include those that, through deep learning, can extract key features from images to indicate whether they contain a target object. Deep learning (DL) is a new research direction in machine learning (ML), introduced to bring it closer to its original goal: AI. Deep learning learns the inherent patterns and hierarchical representations of sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, recognizing data such as text, images, and sound. Deep learning has enabled machines to mimic human brain activities such as sight, hearing, and thought, achieving significant results in search technology, data mining, machine learning, machine translation, natural language processing, speech recognition, recommendation and personalization technologies, and other related fields. It has solved many complex pattern recognition problems, leading to significant progress in artificial intelligence-related technologies. The initial neural network model can utilize known convolutional neural networks, including convolutional layers for extracting image features, pooling layers for dimensionality reduction and redundant information removal of the extracted image features, and an output layer that classifies and identifies the target object based on the image features output by the pooling layers.

[0060] The training sample set may include positive sample images and negative sample images. In this embodiment, positive sample images include surgical environment field images containing target object annotations, and negative sample images include surgical environment field images that do not contain target objects, surgical environment field images with incorrect target object annotations, or other images.

[0061] Based on the training sample set, the neural network model can be trained in the following way: The sample images are labeled with categories. This labeling can be done using tags that uniquely represent the category identity, location, and size of the device. For example, an image containing a specified device 1 is labeled with a category of 1, and the corresponding location and size of the device in the image are also marked. An image containing a specified device 2 is labeled with a category of 2, and the corresponding location and size of the device in the image are also marked. An image not containing any specified device is labeled with a category of 0, thus obtaining sample images with target object labels. These sample images with target object labels are then input into the neural network model to predict the category of the target object in the sample images. The predicted category is compared with the standard target category. Based on the difference between the predicted category and the standard target category, the value of the loss function of the neural network model is determined. The value of the loss function is then passed back to each layer of the neural network model, and the model parameters of each layer are updated using stochastic gradient descent (SGD) until the loss function converges, thus completing the training of the neural network model. Optionally, the initial neural network model may also include a regression layer, which can be a backpropagation neural network. Through training with sample data, the network weights and thresholds are continuously adjusted to make the error function decrease along the negative gradient direction, thus approaching the desired output.

[0062] The trained neural network model extracts features from the surgical environment field of view image acquired by the main operating device. Based on the extracted image features, it forms a feature vector for classification prediction to determine the corresponding classification label. This outputs information about the type, location, and size of the target instruments contained in the surgical environment field of view image. In one optional example, the target instrument location and size information includes the coordinates and dimensions of the target bounding box containing the active instrument. By using the active instrument as the target instrument, its corresponding display area is determined based on its location and size, thus distinguishing between the working field of view and the idle field of view, ensuring that the interface icon does not obscure the active instrument. In another optional example, the target instrument location and size information may also include the coordinates and dimensions of the target bounding box containing a specified inactive instrument. By using the specified inactive instrument as the target instrument, its corresponding display area is determined based on its location and size, thus distinguishing between the working field of view and the idle field of view, ensuring that the interface icon does not obscure the specified inactive instrument. The target instrument may also simultaneously include both the active instrument and the specified inactive instrument, thereby ensuring that the interface icon does not obscure any instrument.

[0063] In the above embodiments, the neural network model can be obtained through training. The training sample set can be constructed and continuously enriched based on the surgical environment field images collected during actual applications. Based on the self-learning and iteration of the neural network model, it can be upgraded and replaced as the sample data increases, so that the recognition results of the surgical environment field images can be better and better.

[0064] The method for determining the current display area of ​​the target instrument in the surgical environment field of view is not limited to image recognition. For example, in another optional embodiment, determining the free field of view area in the surgical environment field of view based on the current display area of ​​the target instrument includes:

[0065] Determine the relative distance between the target part on the target instrument and the endoscope mirror surface;

[0066] Based on the relative distance and the transformation relationship between the image coordinate system and the world coordinate system corresponding to the surgical environment field of view of the endoscope, the position and size of the target instrument contained in the field of view image are determined;

[0067] The current display area of ​​the target instrument is determined based on its position and size, and the working field of view and the idle field of view in the surgical environment field of view interface are determined based on the current display area.

[0068] The target part of a target instrument can be any pre-defined reference part used to characterize the location of the target instrument, such as the distal end of the instrument. Different types of target instruments can have different pre-defined parts as their corresponding target parts. The world coordinate system includes a reference coordinate system established based on the surgical environment scene to facilitate the measurement of the relative positional relationships between entities. For example, it can be a world coordinate system established based on the plane of the endoscope mirror, determining the relative distance between the target part of the target instrument and the endoscope mirror in the world coordinate system. The image coordinate system includes a coordinate system established based on the relationship between entities and imaging points when the image acquisition device acquires images of the surgical environment field of view. For example, it can be a three-dimensional coordinate system established with the focal center of the image acquisition device as the origin and the optical axis of the image acquisition device as the vertical axis. The transformation relationship between the image coordinate system and the world coordinate system can usually be represented using rotation and translation matrices. The main operating device acquires the relative distance between the target part on the target instrument and the endoscope surface. Utilizing the projection imaging relationship during the acquisition of the surgical environment field of view image, and based on the relative distance and the transformation relationship between the image coordinate system and the world coordinate system corresponding to the surgical environment field of view of the endoscope, it determines the position and size of the target instrument contained in the field of view image. Furthermore, based on the position and size of the target instrument in the surgical environment field of view image, it determines the current display area of ​​the target instrument in the surgical environment field of view interface.

[0069] In the above embodiments, by obtaining the distance between the target part of the target instrument and the endoscope mirror, the current display area of ​​the target instrument in the surgical environment field of view is determined using optical projection imaging relationships. This provides more optional implementation methods for determining the working field of view area and the idle field of view area in the surgical environment field of view interface. During execution, the interface display control method can simultaneously employ both a target detection method based on the surgical environment field of view image and a method that calculates the position and size of the target instrument in the surgical environment field of view interface using optical projection imaging relationships to determine the current display area of ​​the target instrument in the surgical environment field of view interface. By combining the two methods to correct the results, the working field of view area and the idle field of view area in the surgical environment field of view interface can be determined more accurately.

[0070] The relative distance between the target part of the target instrument and the endoscope surface can be determined using various technical means. In some embodiments, determining the relative distance between the target part of the target instrument and the endoscope surface includes:

[0071] Obtain endoscope structural parameters;

[0072] Determine the type of the target device, and determine the device structure parameters of the target device based on the type of the target device;

[0073] Obtain the driving parameters corresponding to the target instrument;

[0074] The relative distance between the endoscope tip and the endoscope mirror surface is determined based on the endoscope structural parameters, the instrument structural parameters, and the drive parameters.

[0075] The endoscope structural parameters include the shape and dimensions of each part of the endoscope to determine the position of the endoscope mirror. The target instrument's instrument structural parameters include the shape and dimensions of the target part of the instrument, such as the shape and dimensions of the end of the active instrument. The driving parameters corresponding to the target instrument can be obtained from the driving components that drive the target instrument to perform the corresponding actions, such as the number of rotations of the motor shaft of the drive motor. The posture of the target instrument can be determined accordingly through the driving parameters, such as the rotation angle and extension distance of the active instrument. During execution, the interface display control method can calculate the relative distance between the end of the target instrument and the endoscope mirror based on the endoscope structural parameters, the target instrument's instrument structural parameters, and their corresponding driving parameters.

[0076] In the above embodiments, by determining the real-time posture of the target instrument based on the structural parameters of the target instrument, the structural parameters of the endoscope, and the driving parameters that drive the target instrument to perform its current action, the distance between the target part of the target instrument and the endoscope mirror can be calculated. The current display area of ​​the target instrument in the surgical environment field of view is determined by using the optical projection imaging relationship, providing more optional implementation methods for determining the working field of view area and the idle field of view area in the surgical environment field of view interface.

[0077] In some embodiments, obtaining the current position of the interface icon, and if it is determined from the position that the interface icon is obstructed from view, adjusting the interface icon to the free view area includes:

[0078] Get the current position of the interface icon;

[0079] Based on the position of the interface icon, determine whether the interface icon overlaps with the working field of view area, and / or determine whether the idle field of view ratio of the area where the interface icon is located is less than a set threshold.

[0080] If so, adjust the interface icon to the free field of view area.

[0081] The conditions for determining whether an interface icon obstructs the field of view can be that the interface icon and the working field of view area at least partially overlap, or that the proportion of the free field of view in the area where the interface icon is located is less than a set threshold. During execution, the interface display control method can determine whether the interface icon obstructs the field of view based on either of these two conditions. When either condition is met, the interface icon is considered to be obstructing the field of view. The proportion of the free field of view can include the ratio of the free field of view area to the area of ​​the region, in which case the corresponding set threshold includes a set area ratio threshold; or it can include the minimum distance between the edge of a non-free area and the region, in which case the corresponding set threshold includes a set distance threshold.

[0082] In the above embodiments, by optimizing the judgment conditions for whether the interface icon is obstructed by the field of vision, the position control of the interface icon in the surgical environment visual interface is optimized to ensure the integrity of the doctor's field of vision.

[0083] In some embodiments, adjusting the interface icon to the free field of view includes:

[0084] Based on the relative position of the idle field of view area with respect to the working field of view area in the surgical environment field of view interface, the idle field of view area is divided into multiple idle field of view blocks;

[0085] Calculate the percentage of free field of view in multiple free field of view blocks;

[0086] Adjust the interface icon to a target free field of view block that meets the preset requirements for free field of view ratio.

[0087] The unused field of view is divided into multiple unused field of view blocks. Please refer to [link / reference]. Figure 4 This diagram illustrates the possible division of the idle field of view (IFD) blocks. Based on the relative position of the IFD area to the working field of view, it is divided into eight blocks: upper left, upper, upper right, left, right, lower left, lower, and lower right. It should be noted that as the surgical procedure progresses and the position of the working field of view changes within the surgical environment's visual interface, the number of IFD blocks may decrease, and the size of each block may increase or decrease. Depending on the design of the surgical environment's visual interface, different IFD blocks may contain virtual buttons for the surgeon to operate. When adjusting the position of the interface icons, the main operating device can calculate the IFD percentage of multiple IFD blocks based on the real-time situation and adjust the interface icons to a target IFD block where the IFD percentage meets preset requirements. For example, adjusting the interface icons to the IFD block with the largest IFD percentage can reduce the frequency of adjusting the interface icon positions.

[0088] In the above embodiments, by dividing the idle field of view area into multiple idle field of view blocks, when adjusting the position of the interface icon, the idle field of view block with the largest proportion of idle field of view is selected to set the interface icon, so as to optimize the overall layout of the surgical environment field of view interface, reduce the frequency of adjusting the position of the interface icon, and avoid the phenomenon of frequent jumping of the position of the interface icon caused by frequent changes in the idle field of view blocks.

[0089] In some embodiments, before determining the current display area of ​​the target instrument in the surgical environment field of view interface, the following steps are included:

[0090] Determine whether the icon display mode is dynamic or fixed;

[0091] If in dynamic mode, then perform the step of determining the current display area of ​​the target instrument in the surgical environment field of view interface;

[0092] If in fixed mode, wait for interface icon adjustment instructions, and adjust the position of the interface icons accordingly based on the interface icon adjustment instructions.

[0093] The interface icon repositioning feature offers both dynamic and fixed modes. Doctors can choose either mode based on their needs. In dynamic mode, the icon is automatically adjusted by determining the target instrument's current display area, the working field of view, and the idle field of view. In fixed mode, the icon is adjusted based on the doctor's specific actions, such as manually moving an icon to a specific position on the surgical field of view. The icon adjustment command can be a command generated from a doctor dragging an icon to a specific position on the surgical field of view.

[0094] In the above embodiments, dynamic and fixed modes are set for doctors to choose from according to the needs of different application scenarios, so as to meet the needs of more different application scenarios.

[0095] In some embodiments, the free field of view area includes multiple free field of view blocks located at different positions within the surgical environment field of view interface, and the adjustment of the position of the interface icon based on the interface icon adjustment command includes:

[0096] Obtain a selection instruction for an empty field of view block in the surgical environment field of view interface of the candidate location, and adjust the interface icon to the empty field of view block of the corresponding location according to the selection instruction.

[0097] In the fixed mode, adjustment selection buttons are provided according to the position and number of available field-of-view blocks. By clicking the designated adjustment selection button, the interface icon can be adjusted to the corresponding available field-of-view block. The available field-of-view area includes multiple available field-of-view blocks located in different directions within the surgical environment field of view interface. For example, the surgical environment field of view interface can be divided into eight available field-of-view blocks (upper left, upper, upper right, left, right, lower left, lower, and lower right) based on the relative position of the available field-of-view area to the working field of view area. The surgical environment field of view interface has eight adjustment selection buttons (upper left, upper, upper right, left, right, lower left, lower, and lower right) corresponding to the eight available field-of-view blocks. By clicking one of the adjustment selection buttons, the user obtains a selection instruction for the available field-of-view block in the desired direction within the surgical environment field of view interface, and adjusts the interface icon to the corresponding available field-of-view block according to the selection instruction.

[0098] In the above embodiments, a fixed mode for adjusting the position of interface icons is provided. For the relative position of the idle field of view block to the working field of view area, multiple options for the idle field of view block are provided so that users can customize the display position of the interface icons to avoid obstructing the field of view.

[0099] In some embodiments, after adjusting the interface icon to the free field of view area, the method further includes:

[0100] Update and save the current position of the interface icon.

[0101] Whenever the display position of the interface icon changes within the surgical environment's field of view, the current position of the interface icon is updated and saved to ensure real-time updates. It should be noted that this includes situations where the interface icon automatically adjusts its position based on field-of-view obstruction in dynamic mode, and situations where the interface icon adjusts its position based on user commands in fixed mode.

[0102] In the above embodiments, the position changes of the interface icons are updated and recorded in real time to ensure that the latest position of the interface icons can be obtained in real time during the process of adjusting the position of the interface icons in the surgical environment field of view.

[0103] Optionally, before determining the current display area of ​​the target instrument in the surgical environment field of view interface, the process includes:

[0104] Acquire surgical environment field images captured by an endoscope, and display the surgical environment field images on the surgical environment field interface;

[0105] The interface icons include an endoscope icon.

[0106] To gain a more comprehensive understanding of the interface display control method provided in the embodiments of this application, please refer to the following: Figure 5 and Figure 6 The following example uses an endoscope icon as an optional specific example for illustration. The endoscope icon includes an icon displayed in the surgical environment field of view interface to represent the current posture of the endoscope when the doctor controls the endoscope. This icon can indicate the current rotation angle and mirror angle of the endoscope in real time. The interface display control method includes the following steps:

[0107] S10, begin controlling the endoscope;

[0108] S11, determine whether the position adjustment mode of the interface icon is fixed mode or dynamic mode; if it is dynamic mode, execute S121; if it is fixed mode, execute S122.

[0109] S121, determine the working field of view area and the idle field of view area in the surgical environment field of view interface; taking the target instrument as the active instrument as an example, in dynamic mode, by determining the position and size of the active instrument in the surgical environment field of view interface, the display area of ​​the active instrument in the surgical environment field of view interface is determined, and the working field of view area and the idle field of view area are determined based on the display area of ​​the active instrument.

[0110] S13, Get the position of the interface icon;

[0111] S14, determine whether the interface icon is obstructing the view; if yes, execute S141; if no, execute S142 to keep the position of the interface icon unchanged;

[0112] S141, determine the idle field of view block with the largest idle field of view area, and adjust the interface icon to the idle field of view block; after S141, execute S15;

[0113] S122, determine whether an interface icon adjustment instruction to adjust the position of the interface icon has been received; if yes, execute S123; if no, execute S142 to keep the position of the interface icon unchanged.

[0114] S123, according to the adjustment direction indicated by the interface adjustment command, adjust the interface icon to the corresponding free field of view block; after S123, execute S15;

[0115] S15, update and save the positions of the interface icons;

[0116] S16: Do you want to stop controlling the endoscope? If yes, then execute S161; otherwise, return to S11.

[0117] S161, Remove the endoscope icon.

[0118] In the above embodiments, the interface display control method has at least the following characteristics:

[0119] First, it supports both fixed and dynamic modes for adjusting the UI display position, ensuring that the UI does not obstruct the operator's field of vision when moving the surgical area or the endoscope's field of view. In fixed mode, it offers multiple options corresponding to different free areas of view, thus providing more choices for the UI display orientation and allowing users to customize the UI display position to avoid obstructing their view.

[0120] Secondly, the dynamic mode can intelligently identify the user's operating area and then automatically and dynamically move the UI to a position that does not obstruct the view. This allows the operator to operate on the currently displayed UI area without moving the view (because the UI will automatically move to the free view area), which can reduce the doctor's operations in surgery where the operating area is frequently switched. Furthermore, the rule for UI movement in dynamic mode is that the proportion of free view is greater than a set threshold, which can avoid the phenomenon of frequent UI position jumps caused by frequent changes in the maximum free block.

[0121] Third, by optimizing the position of the interface icons in the surgical environment field of view in real time, the surgical environment field of view can display a richer number of icons that represent the positions and execution status of more types of instruments, while ensuring that the interface icons do not obstruct the doctor's view during the operation, so that the doctor can perform the operation better.

[0122] In some embodiments, step S105 above, adjusting the interface icon to the free field of view area, includes:

[0123] Obtain the interface icon and the size of the free view area.

[0124] Determine whether the size of the interface icon is smaller than the size of the free view area.

[0125] If so, adjust the interface icon to the free field of view area.

[0126] If not, adjust the size of the interface icon to be smaller than the size of the free field of view area, and then adjust the interface icon to the free field of view area.

[0127] Adjusting the size of the interface icon to be smaller than the size of the free field of view includes: adjusting the size of the interface icon to a certain proportion of the size of the free field of view, for example, adjusting the size of the interface icon to 0.5, 0.6, 0.7, 0.8, 0.9 times or any other multiple less than 1 of the size of the free field of view.

[0128] In some embodiments, the interface display control method may further include the following steps:

[0129] Determine the current display area of ​​the target instrument in the surgical environment field of view interface.

[0130] Based on the current display area of ​​the target instrument, the working field of view area and the idle field of view area in the surgical environment field of view interface are determined.

[0131] Get the current position of the interface icon. If the position falls within the working field of view area, get the size of the interface icon and the size of the free field of view area, and determine whether the size of the interface icon is smaller than the size of the free field of view area.

[0132] If so, adjust the interface icon to the free field of view area.

[0133] If not, adjust the size of the interface icons in their original positions to reduce obstruction of the work field of view.

[0134] The in-situ adjustment of the interface icon size includes: adjusting the size of the interface icon to a certain proportion of the original interface icon size, for example, adjusting the size of the interface icon to 0.5, 0.6, 0.7, 0.8, 0.9 times or any other multiple less than 1 of the free view area. The in-situ adjustment of the interface icon size also includes not changing the center position of the interface icon.

[0135] Please see Figure 7 In another aspect, this application provides an interface display control device, comprising: a determining module 211, configured to determine the current display area of ​​a target instrument in a surgical environment field of view interface; a field of view division module 212, configured to determine an empty field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument; and an adjusting module 213, configured to obtain the current position of an interface icon, and if it is determined based on the position that the interface icon is obstructed by a field of view, adjust the interface icon to the empty field of view area.

[0136] Optionally, the field of view segmentation module 212 is specifically used to determine the working field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument; and to determine the idle field of view area in the surgical environment field of view interface based on the working field of view area.

[0137] Optionally, the field of view segmentation module 212 is specifically used to acquire a surgical environment field of view image, perform target detection on the surgical environment field of view image, determine the position and size of the target instrument contained in the surgical environment field of view image; determine the current display area of ​​the target instrument based on the position and size of the target instrument, and determine the working field of view area and the idle field of view area in the surgical environment field of view interface based on the current display area.

[0138] Optionally, the field of view segmentation module 212 is further configured to acquire a surgical environment field of view image through a neural network model, perform target detection on the surgical environment field of view image, and determine the position and size of the target instrument contained in the surgical environment field of view image.

[0139] Optionally, the field of view segmentation module 212 is further configured to construct an initial neural network model; train the initial neural network model based on a training sample set of surgical environment field of view images containing target object annotations to obtain a trained neural network model; the target object annotations include the position and size annotations of the target instruments contained in the field of view images.

[0140] Optionally, the field of view segmentation module 212 is further configured to determine the relative distance between the target part on the target instrument and the endoscope mirror; determine the position and size of the target instrument contained in the field of view image based on the relative distance and the transformation relationship between the image coordinate system and the world coordinate system corresponding to the surgical environment field of view of the endoscope; determine the current display area of ​​the target instrument based on the position and size of the target instrument; and determine the working field of view area and the idle field of view area in the surgical environment field of view interface based on the current display area.

[0141] Optionally, the field of view division module 212 is further configured to acquire endoscope structural parameters; determine the type of the target instrument; determine the instrument structural parameters of the target instrument based on the type of the target instrument; acquire the driving parameters corresponding to the target instrument; and determine the relative distance between the end of the target instrument and the endoscope mirror surface based on the endoscope structural parameters, the instrument structural parameters, and the driving parameters.

[0142] Optionally, the adjustment module 213 is specifically used to obtain the current position of the interface icon; determine whether the interface icon overlaps with the working field of view area based on the position of the interface icon, and / or determine whether the idle field of view ratio of the area where the interface icon is located is less than a set threshold; if so, adjust the interface icon to the idle field of view area.

[0143] Optionally, the adjustment module 213 is further configured to: divide the idle field of view into multiple idle field of view blocks according to the relative position of the idle field of view area with respect to the working field of view area in the surgical environment field of view interface; calculate the idle field of view ratio of the multiple idle field of view blocks; and adjust the interface icon to a target idle field of view block in which the idle field of view ratio meets the preset requirements.

[0144] Optionally, the interface display control device further includes a judgment module for judging whether the icon display mode is in dynamic mode or fixed mode; if it is in dynamic mode, the step of determining the current display area of ​​the target instrument in the surgical environment field of view interface is executed; if it is in fixed mode, the interface icon adjustment instruction is waited for, and the position of the interface icon is adjusted accordingly based on the interface icon adjustment instruction.

[0145] Optionally, the free field of view area includes multiple free field of view blocks located in different directions within the surgical environment field of view interface. The adjustment module 213 is further configured to obtain a selection instruction for a free field of view block in the surgical environment field of view interface at a selectable direction, and adjust the interface icon to the corresponding free field of view block according to the selection instruction.

[0146] Optionally, the field of view segmentation module 212 is also used to update and save the current position of the interface icon.

[0147] Optionally, it also includes an acquisition module for acquiring surgical environment field images captured by the endoscope and displaying the surgical environment field images on the surgical environment field interface; wherein the interface icon includes an endoscope icon.

[0148] Optionally, the target device includes an activating device and / or a designated non-activating device.

[0149] Optionally, the adjustment module 213 is further configured to obtain the size of the interface icon and the size of the free field of view; determine whether the size of the interface icon is smaller than the size of the free field of view; if so, adjust the interface icon to the free field of view; or, if not, adjust the size of the interface icon to be smaller than the size of the free field of view, and then adjust the interface icon to the free field of view.

[0150] It should be noted that the interface display control device provided in the above embodiments is only illustrated by the division of the above-described program modules in the process of adjusting and optimizing the display position of interface icons. In practical applications, the above processing can be assigned to different program modules as needed, thus dividing the internal structure of the device into different program modules to complete all or part of the method steps described above. Furthermore, the interface display control device and the interface display control method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0151] Please see Figure 8In another aspect, this application provides a computer device including a processor 21, a memory 22 connected to the processor 21, and a computer program stored in the memory 22 and executable by the processor 21. When executed by the processor 21, the computer program implements the interface display control method described in any embodiment of this application. It should be noted that the processor 21 may include one or more physically separate processors, and multiple processors or intelligent devices including the processors are communicatively connected to jointly execute the interface display control method described in the embodiments of this application. Correspondingly, the memory 22 may also include one or more physically separate storage media of the same or different types.

[0152] In another aspect, this application provides a remote-operated medical system, including the computer device described in this application and a slave operating device connected to the master operating device. The slave operating device includes multiple instruments of different types and a drive component for driving the instruments to perform specified actions. The instruments may include an endoscope for acquiring images of the surgical environment and sending them to the computer device for display. The instruments include at least one of the following: an electrocautery device, a clamp, a stapler, a scissors, and an ultrasound probe.

[0153] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described collision warning method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0154] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for controlling interface display, applied to computer equipment, characterized in that, include: Determine the current display area of ​​the target instrument in the surgical environment field of view interface; Based on the current display area of ​​the target instrument, determine the free field of view area in the surgical environment field of view interface; Get the current position of the interface icon. If it is determined from the position that the interface icon is obstructed in the field of view, get the size of the interface icon and the size of the free field of view area. When the size of the interface icon is greater than or equal to the size of the free field of view, the size of the interface icon is adjusted to be smaller than the size of the free field of view, and the interface icon is adjusted to be within the free field of view. When the interface icon is smaller than the size of the free field of view area, the interface icon is adjusted to be within the free field of view area; The step of determining the free field of view area in the surgical environment field of view interface based on the current display area of ​​the target instrument includes: Determine the relative distance between the target part on the target instrument and the endoscope mirror surface; Based on the relative distance and the transformation relationship between the image coordinate system and the world coordinate system corresponding to the surgical environment field of view of the endoscope, the position and size of the target instrument contained in the field of view image are determined; The current display area of ​​the target instrument is determined based on its position and size. The operational field of view in the surgical environment interface is then determined based on the current display area. The operational field of view includes an area determined based on predictions of the areas that the surgeon will focus on during the surgical procedure. Based on the operational field of view area, the unoccupied field of view area in the surgical environment field of view interface is determined.

2. The interface display control method as described in claim 1, characterized in that, Determining the relative distance between the target area on the target instrument and the endoscope surface includes: Obtain endoscope structural parameters; Determine the type of the target device, and determine the device structure parameters of the target device based on the type of the target device; The drive parameters corresponding to the target instrument are obtained from the drive component that drives the target instrument to perform the corresponding action, and the attitude of the target instrument can be determined by the drive parameters; The relative distance between the endoscope tip and the endoscope mirror surface is determined based on the endoscope structural parameters, the instrument structural parameters, and the drive parameters.

3. The interface display control method as described in claim 1, characterized in that, Adjusting the interface icon to the free view area includes: Based on the relative position of the idle field of view area with respect to the working field of view area in the surgical environment field of view interface, the idle field of view area is divided into multiple idle field of view blocks; Calculate the percentage of free field of view in multiple free field of view blocks; Adjust the interface icon to a target free field of view block that meets the preset requirements for free field of view ratio.

4. The interface display control method as described in claim 3, characterized in that, The step of adjusting the interface icon to a target free view area where the free view percentage meets preset requirements includes: Adjust the interface icon to the target free view area with the largest free view area.

5. A remote-operated medical system, characterized in that, The device includes a computer device and a slave operating device connected to the computer device. The slave operating device includes multiple instruments of different types and a drive component that drives the instruments to perform specified actions. The computer device includes a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor. When the computer program is executed by the processor, it implements the interface display control method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the interface display control method as described in any one of claims 1 to 4.

Images