Endoscope automatic following method and device, electronic equipment and storage medium

Abstract

The invention provides an automatic endoscope following method and device, electronic equipment and a storage medium, and relates to the technical field of automatic control. The method comprises the following steps: identifying a plurality of instruments in an endoscope image; based on the endoscope image, obtaining position information of the instrument in the endoscope image; based on the position information of the multiple instruments, comprehensive representative points of the multiple instruments are obtained; and if the comprehensive representative point deviates from the center of the visual field of the endoscope, performing visual angle following adjustment on the endoscope. The method is used for dynamically balancing the influence of multiple instruments on the view, ensuring that key instruments are always in the center of the view, and solving the view management problem in a multi-instrument scene in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation control, and in particular, to an endoscope automatic following method and device, an electronic device, and a storage medium. BACKGROUND

[0002] The popularity of minimally invasive surgery in modern clinical practice makes the endoscope visual system an indispensable key component in the surgical process. Compared with traditional open surgery, minimally invasive surgery has become the preferred solution for many surgical procedures due to its small trauma, fast recovery, and fewer complications. However, this surgical approach, while reducing patient trauma, also places higher demands on the operating field of view for the surgeon. In particular, during delicate operations, the limitations of the endoscope field of view often affect the surgeon's accurate judgment of the spatial position of the instrument, thereby restricting the precision of the surgical operation.

[0003] In traditional surgery, the endoscope is usually manually controlled by an assistant, which not only has the problems of operation delay and unstable visual angle, but also makes it difficult to ensure that the surgical instrument is always in the best field of view position, increasing the difficulty of operation and the risk of judgment for the surgeon. Therefore, how to improve the automatic adjustment capability of the endoscope field of view has become a problem to be solved in modern minimally invasive surgery.

[0004] In order to solve this problem, in recent years, various endoscope automatic following systems have appeared. These endoscope automatic following systems mostly rely on installing markers or sensors on the instrument, positioning the instrument through optical or electromagnetic tracking, but this approach requires modification of the instrument and is easily affected by occlusion and external interference. In addition, some solutions use image recognition technology to detect the position of the instrument to control the adjustment of the endoscope visual angle. However, these solutions are difficult to apply to multi-instrument collaborative operation.

[0005] Therefore, there is an urgent need to provide an endoscope automatic following solution suitable for multi-instrument collaborative operation. SUMMARY

[0006] The embodiments of the present application provide an endoscope automatic following method, device, electronic device, and storage medium to keep the key instrument in the ideal position of the field of view in a complex operating environment.

[0007] In a first aspect, the embodiments of the present application provide an endoscope automatic following method, comprising:

[0008] identifying a plurality of instruments in an endoscope image;

[0009] obtaining position information of the instruments in the endoscope image based on the endoscope image;

[0010] obtaining a comprehensive representative point of the plurality of instruments based on the position information of the plurality of instruments;

[0011] If the comprehensive representative point deviates from the center of the field of view of the endoscope, the endoscope is adjusted in view angle following.

[0012] In a possible implementation, the position information comprises a center point coordinate of the instrument, and the comprehensive representative point of the plurality of instruments is obtained based on the position information of the plurality of instruments, comprising:

[0013] obtaining a weight of the instrument in determining the comprehensive representative point;

[0014] performing weighted summation on the center point coordinates of the instruments based on the weights corresponding to the instruments to obtain the coordinate of the comprehensive representative point of the plurality of instruments.

[0015] In a possible implementation, the weight of the instrument in determining the comprehensive representative point is obtained, comprising:

[0016] determining an in-frame area of the recognition bounding box of the instrument;

[0017] determining the weight of the instrument in determining the comprehensive representative point through a preset weighting mechanism based on the confidence score and the in-frame area of each instrument.

[0018] In a possible implementation, the weighted summation is performed on the center point coordinates of the instruments based on the weights corresponding to the instruments to obtain the coordinate of the comprehensive representative point of the plurality of instruments, comprising:

[0019] obtaining working information of the endoscope;

[0020] dynamically adjusting the weights corresponding to the instruments based on the working information to obtain adjusted weights;

[0021] performing weighted summation on the center point coordinates of the instruments based on the adjusted weights to obtain the coordinate of the comprehensive representative point of the plurality of instruments.

[0022] In a possible implementation, the position information of the instrument in the endoscope image is obtained based on the endoscope image, comprising:

[0023] after determining the plurality of instruments by recognizing each frame of image of the endoscope image, obtaining the bounding box corresponding to each instrument;

[0024] obtaining the center point coordinate of the bounding box as the position information of the instrument in the endoscope image.

[0025] In a possible implementation, if the comprehensive representative point deviates from the center of the field of view of the endoscope, the endoscope is adjusted in view angle following, comprising:

[0026] obtaining a deviation value between the comprehensive representative point and the center of the field of view of the endoscope;

[0027] When the deviation value is greater than the deviation threshold and the duration is greater than or equal to the preset time, it is determined that the comprehensive representative point deviates from the field of view center of the endoscope.

[0028] The endoscope is adjusted in the angle of view following based on the deviation value.

[0029] In a possible implementation, the endoscope is adjusted in the angle of view following based on the deviation value, including:

[0030] According to the deviation vector corresponding to the deviation value, a moving amount and a moving direction for the angle of view adjustment control are obtained;

[0031] The endoscope is adjusted in the angle of view following based on the moving amount and the moving direction.

[0032] In a second aspect, an embodiment of the present application provides an endoscope automatic following device, including:

[0033] A recognition unit is configured to recognize a plurality of instruments in an endoscope image;

[0034] A processing unit is configured to obtain position information of the instruments in the endoscope image based on the endoscope image;

[0035] A calculation unit is configured to obtain a comprehensive representative point of the plurality of instruments based on the position information of the plurality of instruments;

[0036] An adjustment execution unit is configured to adjust the endoscope in the angle of view following if the comprehensive representative point deviates from the field of view center of the endoscope.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor.

[0038] The memory stores computer execution instructions;

[0039] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and the computer program is executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0042] The endoscope automatic following method and device, the electronic device and the storage medium provided by the embodiments of the present application identify a plurality of instruments in an endoscope image, and obtain position information of the instruments in the endoscope image; based on the position information of the plurality of instruments, a comprehensive representative point of the plurality of instruments is obtained; if the comprehensive representative point deviates from the center of the field of view of the endoscope, the angle of view of the endoscope is adjusted for following, the influence of the plurality of instruments on the field of view is dynamically balanced, and it is ensured that the key instrument is always in the center of the field of view. Thus, the present application can process the position information of the plurality of instruments at the same time, and dynamically adjust the center of the field of view through the comprehensive representative point corresponding to the plurality of instruments, thereby breaking through the limitation of only being able to track a single instrument, so that the key instrument can still be kept in the ideal position of the field of view in a complex operation environment, the problem of field of view management in a multi-instrument scene is solved, the deviation of the field of view caused by instrument shielding or dynamic movement is avoided, and the practical value is improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0044] Figure 1 It is a schematic diagram of an implementation environment related to the present application;

[0045] Figure 2 It is a schematic diagram of the endoscope automatic following process provided by the present application;

[0046] Figure 3 In an exemplary embodiment provided by the present application, the endoscope automatic following method for endoscope image processing is realized through a YOLO model;

[0047] Figure 4 In the endoscope automatic following method of an exemplary embodiment of the present application, the flowchart of the angle of view following adjustment of the endoscope is shown;

[0048] Figure 5 It is a structural schematic diagram of the endoscope automatic following provided by the present application;

[0049] Figure 6 It is a structural schematic diagram of the electronic device provided by the present application.

[0050] Through the above drawings, the specific embodiments of the present application have been shown, and more detailed descriptions will be given hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to denote the same elements in different drawings. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0052] Figure 1 An exemplary environment for an embodiment consistent with the present application is shown in FIG. 1. The environment includes an endoscope 10 and a server 20, which are connected by wire or wirelessly.

[0053] The server 20 is configured to identify a plurality of instruments in an endoscopic image transmitted by the endoscope 10; obtain position information of the instruments in the endoscopic image based on the endoscopic image; obtain a comprehensive representative point of the plurality of instruments based on the position information of the plurality of instruments; and adjust a view angle of the endoscope 10 if the comprehensive representative point deviates from a center of a field of view of the endoscope 10.

[0054] It should be noted that, Figure 1 The server 20 in the exemplary environment can be a standalone server or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. basic cloud computing services, which are not limited herein.

[0055] In view of the above scenarios, there is a problem in the prior art that it is difficult to adapt to a surgical environment with multiple instruments operating in coordination. The endoscopic automatic following provided by the present application dynamically balances the influence of multiple instruments on the field of view, ensures that the key instruments are always in the center of the field of view, and solves the problem of field of view management in the prior art in a multiple instrument scenario.

[0056] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0057] Figure 2 An exemplary flowchart of the endoscopic automatic following provided by the present application is shown in FIG. 2, which includes the following steps: Figure 2

[0058] S201, identifying a plurality of instruments in an endoscopic image.

[0059] ​The endoscope image refers to a visualized picture of a surgical process collected and displayed in real time by an endoscope. The surgical instruments in the picture can be multiple.

[0060] In this embodiment, the endoscope image collected and displayed in real time is recognized and processed to identify the instruments in the image and determine the types of the instruments.

[0061] S202. Based on the endoscope image, the position information of the instruments in the endoscope image is obtained.

[0062] After the multiple instruments in the endoscope image are identified, the multiple instruments are further positioned to obtain the position information of the instruments in the endoscope image, which provides a data basis for subsequent adjustment control of the visual angle of the endoscope.

[0063] S203. Based on the position information of the multiple instruments, a comprehensive representative point of the multiple instruments is obtained.

[0064] In this embodiment, the position information of the multiple instruments is obtained to determine a comprehensive representative point corresponding to the multiple instruments. Through determination of the comprehensive representative point, the multiple target information (multiple instruments) scattered in the endoscope image can be fused into a single “focus” or “point of interest” having representativeness and decision value, which provides a key support for automatic following control of the endoscope.

[0065] S204. If the comprehensive representative point deviates from the center of the visual field of the endoscope, the visual angle of the endoscope is adjusted.

[0066] In the embodiment of the application, the comprehensive representative point corresponding to the multiple instruments is determined, and the comprehensive representative point is used as a basis for visual angle deviation judgment, so as to dynamically balance the influence of the multiple instruments on the visual field of the endoscope and ensure that the key instruments are always in the center of the visual field.

[0067] The endoscope automatic following method provided in the embodiment of the application is applied to the multiple-instrument application scenario in a complex operation environment. In order to dynamically balance the influence of the multiple instruments on the visual field and ensure that the key instruments are always in the center of the visual field, the position information of the multiple instruments is obtained to obtain a comprehensive representative point of the multiple instruments. If it is determined that the comprehensive representative point deviates from the center of the visual field of the endoscope, the visual angle of the endoscope is adjusted. It can be seen that the application can process the position information of the multiple instruments and dynamically adjust the center of the visual field through the comprehensive representative point corresponding to the multiple instruments, which breaks through the limitation of only tracking a single instrument, so that the key instruments can still be kept in the ideal position of the visual field in the complex operation environment, the problem of visual field management in the multiple-instrument scenario is solved, the deviation of the visual field caused by the shielding or dynamic movement of the instruments is avoided, and the practical value is improved.

[0068] In an example embodiment of the present application, the position information comprises the center point coordinates of the instruments, and correspondingly, based on the position information of the plurality of instruments, the step of obtaining the comprehensive representative point of the plurality of instruments can specifically include:

[0069] obtaining the weight of the instrument when determining the comprehensive representative point;

[0070] performing weighted summation on the center point coordinates of the instrument based on the corresponding weight of the instrument to obtain the coordinates of the comprehensive representative point of the plurality of instruments.

[0071] Among them, the plurality of instruments in the endoscope image has a corresponding importance degree for the current operating environment, which is reflected as the weight of the instrument when determining the comprehensive representative point.

[0072] In the embodiment, after obtaining the weight of each instrument when determining the comprehensive representative point, the center point coordinates of the instrument are processed by weighted summation to obtain the coordinates of the comprehensive representative point of the plurality of instruments. In another embodiment provided by the present application, the coordinate calculation of the comprehensive representative point can be expressed as:

[0073]

[0074]

[0075] Among them, is the weighted comprehensive representative point coordinates, is the center point coordinates of each instrument, is the weight of each instrument when determining the comprehensive representative point. Through the calculation of the comprehensive representative point, it is ensured that the instruments with high confidence and large area have greater influence in the representative point determination process.

[0076] In this way, through the above-mentioned embodiments, by obtaining the weight of each instrument when determining the comprehensive representative point, and performing weighted summation on the center point coordinates thereof, the scientificity and accuracy of the coordinate calculation of the comprehensive representative point can be effectively improved. In this way, the actual importance difference of different instruments in the operation is considered, and the deviation caused by simple average is avoided, thereby providing a more reliable and accurate spatial reference benchmark for subsequent operations.

[0077] In another example embodiment of the present application, the step of obtaining the weight of the instrument when determining the comprehensive representative point can specifically include:

[0078] determining the in-frame area of the recognition bounding box of the instrument;

[0079] determining the weight of the instrument when determining the comprehensive representative point based on the confidence score and the in-frame area of each instrument through a preset weighting mechanism.

[0080] The identification boundary box of the instrument is generated after the instrument in the endoscope image is identified. The identification of the instrument can be achieved by a pre-trained object detection model, such as a YOLO (You Only Look Once) model, and the object detection model also assigns a corresponding confidence score to each instrument after identifying the instrument.

[0081] In this embodiment, the area in the box of the identification boundary box is first calculated, and then the weight of the instrument in determining the comprehensive representative point is determined based on the confidence score and the area in the box of each instrument through a preset weighting mechanism (such as a softmax weighting mechanism). In another embodiment provided by the present application, the process of calculating the weight corresponding to the instrument can be represented as:

[0082]

[0083] wherein, wi represents the weight of the i th instrument; ci represents the confidence score of the i th instrument; ai represents the area in the box of the i th instrument; ni represents the total number of detected instruments. e is a natural constant, which is approximately equal to 2.71828; j is a summation index, which represents traversing all detected instruments; represents that the confidence scores of all instruments are taken as exponents and multiplied by the corresponding areas in the box, and then summed, that is, all are added.

[0084] In this way, the confidence score of the instrument is combined with the area in the box of the identification boundary box of the instrument, and the weight is calculated based on the preset weighting mechanism, so that the system can intelligently determine a stable comprehensive representative point in a multi-instrument scene. In this way, the reliability of the result is ensured by the confidence score, and the physical rationality of the focus selection is enhanced by introducing the area information, so that the comprehensive representative point is more inclined to gather to the instrument region which is both significant and reliable, thereby effectively improving the accuracy, stability and anti-interference ability of the subsequent automatic tracking of the surgical field of view.

[0085] In another exemplary embodiment of the present application, the step of weighting and summing the center point coordinates of the instruments based on the weights corresponding to the instruments to obtain the coordinates of the comprehensive representative point of the multiple instruments can specifically include:

[0086] Obtaining working information of the endoscope;

[0087] Dynamically adjusting the weights corresponding to the instruments based on the working information to obtain adjusted weights;

[0088] Weighting and summing the center point coordinates of the instruments based on the adjusted weights to obtain the coordinates of the comprehensive representative point of the multiple instruments.

[0089] The working information of the endoscope can include a surgical stage, a movement speed of the instrument, an operation frequency, and other factors that can affect the weight of the instrument.

[0090] In this embodiment, an attention mechanism (such as a Transformer architecture) is introduced to dynamically adjust the weight corresponding to the instrument according to the working information of the endoscope, and then the center point coordinates of the instrument are weighted and summed based on the adjusted weight to obtain the coordinates of the comprehensive representative point of the multiple instruments. For example, in the puncture stage, the system enhances the detection accuracy of the puncture needle through the attention mechanism; in the suturing stage, the feature extraction of the suture line or the suture needle is focused on.

[0091] In this way, the dynamic weight distribution mechanism is used to solve the limitations of the traditional fixed weight through the above-mentioned embodiments, so that the result is more in line with the actual surgical requirements. And the robustness of determining the comprehensive representative point in complex scenes is improved, thereby reducing the false detection and missed detection phenomenon. Further, the endoscope can maintain high-precision detection in different surgical stages and environments, ensure the accuracy of the comprehensive representative point calculation, further improve the real-time performance and reliability of the perspective adjustment, and provide more accurate visual assistance.

[0092] In an example embodiment of the present application, based on the endoscope image, the step of obtaining the position information of the instrument in the endoscope image can specifically include:

[0093] After determining the multiple instruments by recognizing each frame of image of the endoscope image, the corresponding bounding box of each instrument is obtained;

[0094] The center point coordinates of the bounding box are obtained as the position information of the instrument in the endoscope image.

[0095] In this embodiment, the target detection model can be used to obtain the position information of the instrument in the endoscope image. Specifically, the target detection model (such as a YOLO-detection model) is used to recognize and process each frame of image of the real-time endoscope image, detect the surgical instrument therein, and generate a corresponding bounding box for each detected instrument. Then, based on these bounding boxes, the center point coordinates of each instrument are calculated as the position information of the instrument in the endoscope image.

[0096] Figure 3 In an example embodiment provided by the present application, a schematic diagram of an endoscope automatic following method for endoscope image processing is realized by a YOLO model. As shown in Figure 3 The process of realizing the endoscope automatic following in this embodiment includes the following steps.

[0097] S301: Endoscope image input. The endoscope image collected by the endoscope is obtained as the input of the YOLO model.

[0098] S302: Detecting instruments by YOLO model. Multiple instruments in the endoscope image are identified by the YOLO model.

[0099] S303: Detecting instrument positions. The position information of the instruments in the endoscope image is obtained, and the identification bounding box corresponding to each instrument is determined.

[0100] S304: Determining whether an instrument is detected. After detecting an instrument, step S305 is entered, otherwise step S306 is entered.

[0101] S305: Calculating instrument center points. The center point coordinates of the instruments are calculated according to the identification bounding box of the instruments.

[0102] S306: Maintaining the current view angle.

[0103] S307: Weighted representative point calculation. The confidence score corresponding to each instrument and the area within the identification bounding box are obtained, the weight of the instrument is determined based on the confidence score and the area within the bounding box by a preset weighting mechanism, and the center point coordinates of the instruments are weighted and summed based on the weight corresponding to the instruments to obtain the coordinates of the comprehensive representative point of the multiple instruments.

[0104] S308: Calculating deviation information. The deviation information between the comprehensive representative point and the center of the field of view of the endoscope is obtained.

[0105] S309: Adjusting the view angle of the endoscope. The view angle of the endoscope is adjusted based on the deviation information.

[0106] Thus, by the above embodiments, the endoscope image is identified frame by frame, and the bounding box and center point coordinates of each instrument are obtained, the system realizes real-time and accurate quantitative positioning of the spatial position of the surgical instruments in the picture, and provides stable and reliable basic data for subsequent comprehensive representative point calculation and automatic view angle tracking.

[0107] In addition, in terms of system response performance, compared with existing solutions that rely on depth estimation models or complex hardware calculations, the present application uses two-dimensional visual detection to effectively reduce system delay while ensuring detection accuracy, can timely follow the movement of the instruments, and ensure that the field of view continuously aligns with the main operation area. In terms of system deployment, the present application does not need to rely on special depth sensors or complex hardware architecture. This feature enables the present application to be quickly deployed on the basis of existing equipment, significantly reducing the cost of transformation and technical threshold, and creating favorable conditions for popularization and application in diversified medical environments.

[0108] In an example embodiment of the present application, if the comprehensive representative point deviates from the center of the field of view of the endoscope, the step of adjusting the view angle of the endoscope to follow the view angle can specifically include:

[0109] obtaining a deviation value between the comprehensive representative point and the center of the field of view of the endoscope;

[0110] when the deviation value is greater than the deviation threshold and the duration is greater than or equal to the preset time, determining that the comprehensive representative point deviates from the center of the field of view of the endoscope;

[0111] adjusting the angle of view of the endoscope based on the deviation value.

[0112] In this embodiment, a deviation threshold (usually set to 15% of the field of view width) is preset to determine whether the instrument deviates from the center of the field of view. When the deviation value between the comprehensive representative point and the center of the field of view of the endoscope is greater than the deviation threshold and the duration is greater than or equal to the preset time (for example, 2 seconds), it is determined that the comprehensive representative point deviates from the center of the field of view of the endoscope, which triggers the adjustment mechanism to adjust the angle of view of the endoscope based on the deviation value.

[0113] In this way, the above-mentioned embodiments of the present application can effectively distinguish between real instrument displacement and instantaneous jitter or detection error, avoid unnecessary frequent adjustment, and thus maintain the continuity and stability of the surgical field, and improve the smoothness and safety of the surgical operation.

[0114] In another example embodiment, after determining that the comprehensive representative point deviates from the center of the field of view of the endoscope, an adjustment trigger reminder can also be issued, and feedback confirmation information generated by the operator after making an operation feedback (such as touching the corresponding pedal) according to the reminder can be received, and then the angle of view of the endoscope is adjusted based on the deviation value after receiving the feedback confirmation information. In this way, the endoscope and its mechanical arm can be limited in terms of safety, and the endoscope can be prevented from moving automatically and abruptly due to algorithm misjudgment, sensor noise or unexpected scenarios (such as temporary obstruction of the instrument), further improving the smoothness and safety of the surgical operation.

[0115] In another example embodiment of the present application, the step of adjusting the angle of view of the endoscope based on the deviation value can specifically include:

[0116] obtaining a movement amount and a movement direction for angle of view adjustment control according to the deviation vector corresponding to the deviation value;

[0117] adjusting the angle of view of the endoscope based on the movement amount and the movement direction.

[0118] In this embodiment, after determining that the angle of view of the endoscope needs to be adjusted, the deviation vector corresponding to the deviation value is determined according to the deviation value, the coordinates of the comprehensive representative point and the coordinates of the center of the field of view, and then the movement amount and the movement direction for angle of view adjustment control are determined, and then the angle of view of the endoscope is adjusted based on the movement amount and the movement direction using the mechanical arm of the endoscope. In this way, by calculating the deviation vector corresponding to the deviation value , can accurately control the movement amount and direction of the endoscope in the horizontal and vertical directions, and smoothly guide the target instrument back to the center of the field of view.

[0119] Figure 4 For the endoscope automatic following method of an exemplary embodiment of the present application, a flowchart of the process of adjusting the view angle of the endoscope is shown. As shown in Figure 4 the process of controlling the automatic following of the endoscope in this embodiment includes the following steps.

[0120] S401: Calculate the deviation vector. Calculate the deviation vector corresponding to the deviation value between the comprehensive representative point and the center of the field of view of the endoscope.

[0121] S402: Determine if the deviation value > deviation threshold and the duration > 2 seconds? If yes, go to step S403, if not, go to step S407.

[0122] S403: Generate adjustment instructions.

[0123] S404: Endoscope view angle adjustment. Adjust the view angle of the endoscope according to the movement amount and direction of the deviation vector.

[0124] S405: Update the display interface. Prompt the view angle adjustment on the interface displaying the endoscope image.

[0125] S406: Feedback to the doctor.

[0126] S407: Maintain the current view angle.

[0127] S408: Feedback that the view angle has no change.

[0128] According to the content display of the display interface, a layered visualization strategy is adopted to balance the completeness of the function and the user experience. During the development and debugging stage, detailed state information will be displayed, including the position of the representative point, the deviation vector and the direction indication, etc., to facilitate development monitoring and system performance optimization. In the actual use environment, only a simple interface prompt is provided: when the view angle needs to be adjusted, the "Auto-follow Available" prompt box will be displayed in the upper right corner of the interface, which is the only visual feedback. This ensures the simplicity of the operation interface and avoids the interference of too much information on the operation, while retaining the necessary function prompts.

[0129] In this way, through the above embodiments, the application based on the application of the deviation vector directly converts the spatial position deviation into executable control instructions, ensuring that the endoscope can quickly and stably guide the deviated comprehensive representative point back to the center of the field of view, both ensuring the accuracy of the adjustment and ensuring the smoothness of the movement.

[0130] Figure 5 A structure diagram of an endoscope automatic following device provided in the present application is shown in FIG. 1. As shown in the figure, the endoscope automatic following device 50 comprises: Figure 5

[0131] An identification unit 501 is configured to identify a plurality of instruments in an endoscope image.

[0132] A processing unit 502 is configured to obtain position information of the instruments in the endoscope image based on the endoscope image.

[0133] A calculation unit 503 is configured to obtain a comprehensive representative point of the plurality of instruments based on the position information of the plurality of instruments.

[0134] An adjustment execution unit 504 is configured to perform a view angle following adjustment on the endoscope if the comprehensive representative point deviates from a center of a field of view of the endoscope.

[0135] In a possible implementation, the position information comprises a center point coordinate of the instrument; the calculation unit 503 is further configured to obtain a weight of the instrument when determining the comprehensive representative point; and the center point coordinate of the instrument is weighted and summed based on the corresponding weight of the instrument to obtain a coordinate of the comprehensive representative point of the plurality of instruments.

[0136] In a possible implementation, the calculation unit 503 is further configured to determine an in-frame area of an identification bounding box of the instrument; and determine the weight of the instrument when determining the comprehensive representative point based on a confidence score of each instrument and the in-frame area through a preset weighting mechanism.

[0137] In a possible implementation, the calculation unit 503 is further configured to obtain working information of the endoscope; dynamically adjust the weight of the instrument based on the working information to obtain an adjusted weight; and weight and sum the center point coordinates of the instruments based on the adjusted weight to obtain a coordinate of the comprehensive representative point of the plurality of instruments.

[0138] In a possible implementation, the processing unit 502 is further configured to obtain a bounding box corresponding to each instrument after identifying and determining the plurality of instruments through each frame of the endoscope image; and obtain a center point coordinate of the bounding box as the position information of the instrument in the endoscope image.

[0139] In a possible implementation, the adjustment execution unit 504 is further configured to obtain a deviation value between the comprehensive representative point and a center of a field of view of the endoscope; and determine that the comprehensive representative point deviates from the center of the field of view of the endoscope when the deviation value is greater than a deviation threshold value and the duration is greater than or equal to a preset time; and perform a view angle following adjustment on the endoscope based on the deviation value.

[0140] ​In one possible implementation, the adjustment execution unit 504 is also used to obtain the amount of movement and the direction of movement for viewing angle adjustment control based on the deviation vector corresponding to the deviation value; and to perform viewing angle following adjustment of the endoscope based on the amount of movement and the direction of movement.

[0141] The endoscope automatic following device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0142] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0143] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0144] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0145] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0146] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0147] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0148] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described above.

[0149] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0150] The readable storage medium described above can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0151] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0152] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0154] In addition, each functional unit in various embodiments of the application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0155] If the function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiment methods of the application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0156] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0157] Finally, it should be noted that those skilled in the art, after considering the specification and practicing the application disclosed herein, will easily think of other embodiments of the application. The application is intended to cover any variations, uses, or adaptations of the application that follow the general principles of the application and include common knowledge or conventional techniques in the art that are not disclosed by the application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the application is only limited by the appended claims.

Claims

1. An automatic endoscope following method, characterized in that, include: Identify multiple instruments in endoscopic images; Based on the endoscopic image, the position information of the instrument in the endoscopic image is obtained; Based on the location information of the multiple instruments, a comprehensive representative point of the multiple instruments is obtained; If the comprehensive representative point deviates from the center of the endoscope's field of view, the endoscope's viewing angle is adjusted to follow the deviation.

2. The automatic endoscope following method according to claim 1, characterized in that, The location information includes the coordinates of the center point of the instrument; obtaining a comprehensive representative point of the multiple instruments based on their location information includes: Obtain the weight of the instrument in determining the comprehensive representative point; The coordinates of the center point of each instrument are weighted and summed based on the weights corresponding to the instruments to obtain the coordinates of the comprehensive representative point of the multiple instruments.

3. The method according to claim 2, characterized in that, The step of obtaining the weight of the instrument in determining the comprehensive representative point includes: Determine the area within the identification bounding box of the device; Based on the confidence score of each instrument and the area within the frame, the weight of each instrument in determining the comprehensive representative point is determined through a preset weighting mechanism.

4. The automatic endoscope following method according to claim 2, characterized in that, The step of weighted summation of the center point coordinates of the instruments based on their respective weights to obtain the coordinates of a comprehensive representative point of the multiple instruments includes: Obtain the working information of the endoscope; Based on the work information, the weights corresponding to the instruments are dynamically adjusted to obtain the adjusted weights. The coordinates of the center point of the instruments are weighted and summed based on the adjusted weights to obtain the coordinates of the comprehensive representative point of the multiple instruments.

5. The automatic endoscope following method according to any one of claims 1 to 4, characterized in that, The step of obtaining the position information of the instrument in the endoscopic image based on the endoscopic image includes: After identifying multiple instruments by analyzing each frame of the endoscopic image, the bounding box corresponding to each instrument is obtained. Obtain the coordinates of the center point of the bounding box as the position information of the instrument in the endoscopic image.

6. The automatic endoscope following method according to any one of claims 1 to 4, characterized in that, If the comprehensive representative point deviates from the center of the endoscopic field of view, the endoscopic angle is adjusted to follow the deviation, including: Obtain the deviation value between the comprehensive representative point and the field of view center of the endoscope; When the deviation value is detected to be greater than the deviation threshold and the duration is greater than or equal to a preset time, it is determined that the comprehensive representative point is deviated from the center of the field of view of the endoscope; The endoscope is adjusted to follow the deviation value.

7. The automatic endoscope following method according to any one of claims 6, characterized in that, The method of adjusting the viewing angle of the endoscope based on the deviation value includes: Based on the deviation vector corresponding to the deviation value, the amount and direction of movement used for viewpoint adjustment control are obtained; The endoscope is adjusted to follow the movement based on the amount and direction of movement.

8. An automatic endoscope following device, characterized in that, include: A recognition unit is used to identify multiple instruments in an endoscopic image; The processing unit is used to obtain the position information of the instrument in the endoscopic image based on the endoscopic image; A calculation unit is used to obtain a comprehensive representative point of the multiple instruments based on their location information; The adjustment execution unit is used to adjust the viewing angle of the endoscope if the comprehensive representative point deviates from the center of the endoscope's field of view.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the endoscope automatic following method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the endoscope automatic following method as described in any one of claims 1 to 7.

Images