Target tracking method, endoscope system, and computer-readable medium

Abstract

The application relates to a target tracking method, an endoscope system and a computer readable medium, wherein when it is detected that a target instrument is added in a current frame field of view image, a previous frame to a previous k frame field of view image is called, and whether a target instrument is close to a field of view image boundary is identified. If no target instrument is close to the field of view image boundary, it is indicated that the target instrument added in the current frame field of view image is a newly added instrument. If a target instrument is close to the field of view image boundary, if it is judged that the same instrument based on an instrument appearance feature, the same instrument ID is given. The current frame field of view image is automatically judged to be added to the target instrument due to the fact that the target instrument is moved out of the boundary and then reenters, so that under a main instrument tracking strategy, the field of view image can immediately locate the main instrument reentered into the field of view image and follow the main instrument to move, an operator does not need to stop a current operation to redefine the instrument, and the operation efficiency is ensured.

Description

Technical Field

[0001] This application relates to the field of medical device auxiliary control technology, and in particular to a target tracking method, an endoscope system, and a computer-readable medium. Background Technology

[0002] Endoscopic surgery is a procedure performed using an endoscope and surgical instruments. An endoscopic system is used to insert the endoscope's lens into the abdominal cavity. Images captured by the laparoscopic lens are transmitted via fiber optic cables to a signal processing system and displayed in real-time on a dedicated monitor. The surgeon then analyzes and assesses the patient's condition by viewing images of the organs from different angles on the monitor screen and performs the surgery using the surgical instruments.

[0003] In related technologies, tracking strategies for endoscopic visual field images include using a central element composed of multiple instruments or a single instrument as the primary instrument for tracking. However, when an instrument moves out of the visual field boundary in a frame due to misoperation and then re-enters in a subsequent frame, the system cannot automatically identify whether it was a newly added instrument or a misoperation, requiring a re-definition. This is particularly problematic with primary instrument tracking strategies; when the primary instrument moves out of and re-enters the visual field, the image cannot immediately locate and follow it. The operator must stop the current surgical procedure and redefine the instrument, impacting surgical efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a target tracking method, an endoscope system, and a computer-readable medium to address the problem that when the main instrument is removed from the field of view and then re-enters, the field of view cannot immediately locate the main instrument and follow its movement, requiring the operator to stop the current surgical operation and redefine it, thus affecting surgical efficiency.

[0005] This application provides a target tracking method applied to an endoscope system, the endoscope system including an endoscope arm and an endoscope, the endoscope arm being used to adjust the field of view image captured by the endoscope to follow the movement of a target instrument, the target tracking method including:

[0006] Acquire the current frame view image captured by the endoscope, and acquire the previous frame view image relative to the current frame view image;

[0007] Obtain the contour region of each target device in the current frame of the field of view image, and obtain the contour region of each target device in the previous frame of the field of view image.

[0008] Select a target device;

[0009] The similarity algorithm is used to determine whether the target device in the current frame of the visual field image appeared in the previous frame of the visual field image;

[0010] If the target device in the current frame of the field of view does not appear in the previous frame of the field of view, then the target device is defined as a device to be matched, and it is further determined whether there is at least one target device in the previous frame of the field of view to the previous k frames of the field of view where the minimum straight-line distance between the center point of the target device and the boundary of the field of view is less than a preset distance threshold; k is a positive integer greater than or equal to 2.

[0011] If, in the previous frame of the visual field image to the previous k frames of the visual field image, the minimum straight-line distance between the center point of at least one target device and the boundary of the visual field image is less than a preset distance threshold, then the target device is defined as a marked device, and the marked device is compared with the device to be matched based on the device's appearance characteristics, and a device ID is assigned to the device to be matched according to the comparison result.

[0012] Return to the process of selecting a target device, until all target devices in the current frame of the field of view have been selected;

[0013] Based on the center point coordinates of each target device from the first frame of the field of view to the current frame of the field of view, a motion trajectory dataset corresponding to the target device is created, and the motion trajectory dataset of each target device is included in the trajectory list;

[0014] The coordinates of the target center point are determined based on the current field of view control mode and the position of each target instrument in the current frame of the field of view image. The offset vector between the coordinates of the image center point and the coordinates of the target center point in the current frame of the field of view image is calculated.

[0015] Based on the offset vector and the current field of view control mode, the endoscope arm is controlled to move so that the field of view image captured by the endoscope moves.

[0016] Return to the current frame of the field of view image captured by the endoscope, and the previous frame of the field of view image relative to the current frame of the field of view image.

[0017] This application also provides an endoscope system, comprising:

[0018] A processor for executing the target tracking method described above;

[0019] The endoscope is electrically connected to the processor;

[0020] A display device is used to display images captured by the lens of the endoscope, and the display device is communicatively connected to the processor;

[0021] Endoscope holding arm, the endoscope is mounted on the endoscope holding arm;

[0022] The endoscope arm is electrically connected to the processor, and the processor controls the endoscope arm to adjust the image captured by the endoscope lens according to the image captured by the endoscope lens.

[0023] This application also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the target tracking method as described above.

[0024] This application relates to a target tracking method, an endoscope system, and a computer-readable medium. The method, when detecting an added target instrument in the current frame of the visual field image relative to the previous frame, retrieves the previous frame up to the k frames prior to the target instrument and identifies whether a target instrument is near the boundary of the visual field image. If no target instrument is near the boundary, it indicates that the added target instrument in the current frame is a newly added instrument. If a target instrument is near the boundary, it is compared with the added target instrument in the current frame based on the instrument's appearance characteristics. If the comparison determines they are the same instrument, they are assigned the same instrument ID. This method automatically identifies when a target instrument added to the current frame is due to accidental movement outside the boundary and subsequent re-entry. Under the main instrument tracking strategy, the visual field image can immediately locate and follow the main instrument that has re-entered the visual field image, eliminating the need for the operator to stop the current surgical procedure and redefine the instrument, thus ensuring surgical efficiency. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a target tracking method provided in an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of the pixel coordinate system and camera coordinate system in a target tracking method provided in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the minimum straight-line distance in a target tracking method provided in an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the identification result of the minimum bounding rectangle of the target device in a target tracking method provided in an embodiment of this application.

[0029] Figure 5 This is a schematic diagram of target device tracking and matching in two adjacent frames of field of view images provided in an embodiment of the present application.

[0030] Figure 6 This is a schematic diagram of the tracking region and the stopping region in the field of view image of a target tracking method provided in an embodiment of this application.

[0031] Figure 7 This is a schematic diagram of the joint center calculation of multiple target instruments in a target tracking method provided in an embodiment of this application.

[0032] Figure 8This is a schematic diagram of the structure of an endoscope system provided in an embodiment of this application.

[0033] Figure 9 This is a schematic diagram illustrating an application scenario of an endoscope system provided in an embodiment of this application.

[0034] Figure label:

[0035] 100 - Endoscopic system; 110 - Processor; 120 - Display device; 130 - Endoscope; 140 - Endoscope arm; 200 - Surgical instruments; 300 - Operator. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] This application provides a target tracking method.

[0038] It should be noted that the field of view control method provided in this application does not limit the subject of execution. Optionally, the field of view control method provided in this application is applied to an endoscope system, which includes an endoscope arm and an endoscope. The endoscope arm is used to manipulate the movement of the endoscope, thereby adjusting the image captured by the endoscope so that the target instrument is kept in the image.

[0039] like Figure 1 As shown, in one embodiment of this application, the target tracking method includes the following steps S100 to S930:

[0040] S100: Acquire the current frame view image captured by the endoscope, and acquire the previous frame view image relative to the current frame view image.

[0041] Specifically, the source of the field of view image can be a monocular camera, or the left or right camera of a binocular camera, and the field of view image captured by the endoscope is displayed on a display device.

[0042] S200, acquire the contour region of each target device in the current frame of the field of view image, and acquire the contour region of each target device in the previous frame of the field of view image.

[0043] Specifically, the contour region of the target instrument is identified based on machine vision methods, and the contour region of the target instrument can be obtained in the following way:

[0044] 1) The area occupied by all pixels of the target instrument in the field of view image.

[0045] 2) The area occupied by the circumscribed geometry of the target instrument in the field of view. This circumscribed geometry can be the circumscribed polygon or the circumscribed circle of the target instrument.

[0046] S300, Select a target device.

[0047] S400, determine whether the target device in the current frame of the visual field image appeared in the previous frame of the visual field image based on a similarity algorithm.

[0048] S510, if the target device in the current frame of the visual field image does not appear in the previous frame of the visual field image, then the target device is defined as a device to be matched, and it is further determined whether there exists at least one target device in the previous frame of the visual field image to the previous k frames of the visual field image whose minimum straight-line distance between the center point and the boundary of the visual field image is less than a preset distance threshold. k is a positive integer greater than or equal to 2.

[0049] Specifically, such as Figure 2 As shown, the center point coordinates of the target instrument are represented by a pixel coordinate system, that is, the upper left corner of the field of view image is taken as the origin O1, the right is the positive x-axis direction, and the down is the positive y-axis direction. The center point coordinates of the target instrument can be determined by one of the following methods.

[0050] 1) The geometric center coordinates of the x, y coordinates of each pixel of the target instrument are taken as the center point of the target instrument, as shown in Formulas 2 and 3 below. It is assumed that the target instrument has n pixels in the field of view image.

[0051]

[0052]

[0053] Where i represents the pixel number, and i is a positive integer greater than or equal to 2.

[0054] 2) such as Figure 4 As shown, the center point of the target instrument is taken as the center of the smallest bounding rectangle of the target instrument, as shown in Formulas 4 and 5 below:

[0055]

[0056]

[0057] Among them, (x lr ,y lr ) represents the pixel coordinates of the top-left corner of the minimum bounding rectangle, width represents the width of the minimum bounding rectangle along the x-axis, and height represents the height of the minimum bounding rectangle along the y-axis.

[0058] Specifically, such as Figure 3As shown, the field of view image is S0, the boundary of the field of view image is S1, the target instrument is E, and the center point of the target instrument is E0. The center point of the target instrument is close to the right boundary of the field of view image. Therefore, the minimum straight-line distance L between the center point EO of the target instrument and the boundary S1 of the field of view image is the perpendicular distance from the center point EO of the target instrument to the right boundary.

[0059] S610, if the minimum straight-line distance between the center point of at least one target device and the boundary of the field of view image is less than a preset distance threshold in the previous frame of the field of view image to the previous k frames of the field of view image, then the target device is defined as a marked device, and the marked device is compared with the device to be matched based on the device appearance characteristics, and a device ID is assigned to the device to be matched according to the comparison result.

[0060] S700, return to the step of selecting a target device, until all target devices in the current frame of the field of view have been selected.

[0061] S800 creates a motion trajectory dataset corresponding to each target device based on the center point coordinates from the first frame of the field of view to the current frame of the field of view, and incorporates the motion trajectory dataset of each target device into the trajectory list.

[0062] Specifically, the trajectory list includes the coordinates of the center point of each target device in the first frame of the visual field image to the current frame of the visual field image. A continuous motion trajectory is formed by the center points of each target device, and the motion trajectory is integrated into the device ID of each target device.

[0063] S910, determine the coordinates of the target center point based on the current field of view control mode and the position of each target instrument in the current frame field of view image, and calculate the offset vector between the coordinates of the image center point and the coordinates of the target center point in the current frame field of view image.

[0064] Specifically, the coordinates of the target center point in the pixel coordinate system are converted to coordinates in the camera coordinate system and normalized to obtain the offset vector.

[0065] Continue with Figure 2 The embodiment shown is described below. The origin of the camera coordinate system (i.e., the image center point) is O2. U and V are the two axes of the camera coordinate system, respectively. M1 is the coordinate (x, y) of the target center point in the obtained pixel coordinate system. It is converted to the coordinates in the camera coordinate system and normalized to obtain (u, v). (u, v) is calculated by the following formulas 6 and 7:

[0066]

[0067]

[0068] Where w represents the pixel width of the field of view image, and h represents the pixel height of the field of view image.

[0069] S920, based on the offset vector and the current field of view control mode, control the movement of the endoscope arm so that the field of view image captured by the endoscope moves.

[0070] Continue with Figure 2 The embodiment shown is described below. The attitude is calculated based on the coordinates (u,v) of the target center point. The target angle that each joint motor of each lens arm needs to rotate is calculated. Then, the lens is moved so that the image center point moves towards the target center point and finally coincides with the target center point.

[0071] S930, return to the current frame view image captured by the endoscope, and the previous frame view image relative to the current frame view image.

[0072] In this embodiment, when a target instrument is detected to have been added to the current frame of the visual field image relative to the previous frame, the visual field images from the previous frame to the previous k frames are retrieved to identify whether a target instrument is near the boundary of the visual field image. If no target instrument is near the boundary of the visual field image, it indicates that the target instrument added to the current frame of the visual field image is a newly added instrument. If a target instrument is near the boundary of the visual field image, it is compared with the target instrument added to the current frame of the visual field image based on the instrument's appearance characteristics. If the comparison result determines that they are the same instrument, they are assigned the same instrument ID. This enables automatic determination that the target instrument added to the current frame of the visual field image is due to a misoperation that caused it to move out of the boundary and then re-enter. Therefore, under the main instrument tracking strategy, the visual field image can immediately locate the main instrument that has re-entered the visual field image and follow its movement. The operator does not need to stop the current surgical operation to redefine the instrument, ensuring surgical efficiency.

[0073] In one embodiment of this application, S400 includes the following S410 to S440.

[0074] S410, calculate the area intersection ratio of a single target device in the current frame of the field of view with that of each target device in the previous frame of the field of view according to Formula 1.

[0075]

[0076] Where IOU(A, B) is the area intersection-union ratio, where A represents the area of ​​the contour region of a single target device in the current frame of the visual field image, B represents the area of ​​the contour region of a single target device in the previous frame of the visual field image, A∩B represents the intersection of the areas of A and B, and A∪B represents the union of the areas of A and B.

[0077] For example, the following is an embodiment, such as Figure 5As shown, the target devices in the previous frame of the visual field image are A1 and B1, and the target devices in the current frame of the visual field image are A2, B2, and C2. A1 and A2 intersect in the two frames of the visual field image, and B1 and B2 also intersect. Target device C2 in the previous frame of the visual field image has no intersecting target devices in the previous frame of the visual field image.

[0078] S420, determine whether the number of target devices in the previous frame of the field of view corresponding to an area crossover ratio greater than a set crossover ratio threshold is equal to 1.

[0079] Specifically, the intersection-union ratio (IURR) threshold can be set as needed. In one embodiment of this application, the IURR threshold ranges from 0.4 to 0.8.

[0080] S430, if the number of target devices in the previous frame of the visual field image corresponding to an area crossover ratio greater than the set crossover ratio threshold is equal to 1, then the two target devices in the current frame of the visual field image and the previous frame of the visual field image with an area crossover ratio greater than the set crossover ratio threshold are determined to be the same target device.

[0081] S440, if the number of target devices in the previous frame of the field of view corresponding to the area crossover ratio being greater than the set crossover ratio threshold is less than 1, then a single target device in the current frame of the field of view fails to find a matching target device in the previous frame of the field of view.

[0082] In this embodiment, the cross-union ratio is used in adjacent frame field-of-view images, which can quickly identify whether the target instrument in two adjacent frame field-of-view images is the same instrument, resulting in high processing efficiency.

[0083] In one embodiment of this application, after S420, S400 further includes the following S451 to S452.

[0084] S451, if the number of target devices in the previous frame of the visual field image corresponding to the area crossover ratio being greater than the set crossover ratio threshold is greater than 1, then the images of each target device in the previous frame of the visual field image corresponding to the area crossover ratio being greater than the set crossover ratio threshold and the target device in the current frame of the visual field image are respectively input into an appearance matching model so that the appearance matching model outputs a matching result.

[0085] Specifically, the training method for the appearance matching model is as follows: images of various surgical instruments taken from different angles are obtained to form a training set, the appearance matching model is trained, tested, and verified to obtain a converged appearance matching model.

[0086] S452, based on the matching result, select one of the target instruments in the previous frame of the field of view image that is the same as the target instrument in the current frame of the field of view image, from each target instrument in the previous frame of the field of view image whose area crossover ratio is greater than the set crossover ratio threshold.

[0087] In this embodiment, when there are multiple target instruments that meet the area intersection-union ratio in the previous frame of the field of view image, the appearance matching model is used for comparison, which can accurately determine the matching target instrument and avoid misjudgment.

[0088] In one embodiment of this application, S610 includes the following S611 to S612.

[0089] S611, input the image of the instrument to be matched and the image of the marked instrument into an appearance matching model, so that the appearance matching model outputs whether the instrument to be matched is the same as the marked instrument.

[0090] Specifically, the appearance matching model in this embodiment is the same as the appearance matching model described above.

[0091] S612, if the appearance matching model outputs the same device to be matched and the marked device, then assign the same device ID to the device to be matched and the marked device.

[0092] In this embodiment, the appearance matching model in S451 is reused. The marked device and the device to be matched are compared according to the appearance characteristics of the device. Then, it is determined that the device added in the current frame of the field of view image is the target device that has re-entered the field of view image, and the device ID is automatically assigned without redefining it.

[0093] In one embodiment of this application, after S510, the target tracking method further includes S620.

[0094] S620 If the minimum straight-line distance between the center point of the target device and the boundary of the field of view image in the previous frame of the field of view image and the previous k frames of the field of view image is less than a preset distance threshold, then the device to be matched is taken as a new target device and a new device ID is assigned.

[0095] Specifically, if there is no matching target device in the previous frame of the visual field image, and there is no target device near the boundary of the visual field in the previous frame to the previous k frames of the visual field image, then the device to be matched is a newly added target device.

[0096] In one embodiment of this application, after S400, the target tracking method further includes the following S521 to S522.

[0097] S521, if the target device in the current frame of the field of view appears in the previous frame of the field of view, then the target device in the current frame of the field of view and the target device that appeared in the previous frame of the field of view are assigned the same device ID.

[0098] S522, return to the step of selecting a target device until all target devices in the current frame of the field of view have been selected.

[0099] In one embodiment of this application, the target center point is the joint center of all target instruments or the center of the main instrument in the current frame field-of-view image.

[0100] Specifically, the field of view control modes include full tracking mode, master-slave mode, and stop mode. The command to switch between field of view control modes can be triggered by stepping on a foot pedal, via a button built into the instrument handle, or by clicking a button on the display device.

[0101] If the field of view control mode is set to full tracking mode, such as Figure 6 As shown, the surgical field image is divided into two regions: a stopping region Q1 and a tracking region Q2. The size of the stopping region Q1 can be determined by pre-set configuration parameters or can be adaptively adjusted according to the size of the target instrument in the image. The tracking region Q2 is the region excluding the stopping region Q1. When the target center point is located in the stopping region Q1, no adjustment of the surgical field image is required. When the target center point is located in the tracking region Q2, the surgical field will be adaptively adjusted. Adjustment will stop when the target center point is within the stopping region Q1.

[0102] When there are multiple target instruments in the field of view, such as Figure 7 As shown, the instruments are D1 (pixel coordinates x1, y1), D2 (pixel coordinates x2, y2), and D3 (pixel coordinates x3, y3), respectively. The coordinates (x, y) of the joint center M2 (i.e., the target center point) of the three instruments are calculated using the following formulas 8 and 9:

[0103]

[0104]

[0105] If the field of view control mode is set to master-slave mode, the center coordinates of the master instrument will be used as the tracking target point. The master instrument will be determined in one of the following ways:

[0106] 1) The operator designates one of the target instruments in the field of view as the main instrument, and the designated command triggering method can be a button or foot pedal on the instrument.

[0107] 3) The target device within a specified area of ​​the field of view is regarded as the main device. For example, when only one target device is located in the tracking area and all other target devices are located in the stopping area, the target device in the tracking area is regarded as the main device.

[0108] 4) The target device with the largest movement range within a specified time is regarded as the master device. Based on the trajectory of each target device recorded in the trajectory list, the displacement of each target device within n consecutive seconds is calculated from the moment the master-slave mode is switched. The device with the largest displacement is regarded as the master device.

[0109] If the field of view control mode is set to stop mode, the field of view will not be adjusted regardless of how the target instrument is moved, providing a stable field of view for surgical operations.

[0110] Based on the foregoing, if a target device moves out of the boundary of the field of view between the previous i-frame and the current frame, a matching target device cannot be found in the current frame. If the movement of the target device is due to an error, it may be re-entered into the field of view by the operator in a subsequent frame. If the current frame recalculates the target center point based on the target device within the field of view, the center of the field of view will move towards that target center point, potentially causing the field of view to move further away from the target device that has moved out of the field of view boundary, increasing the difficulty for the target device to find its way back into the field of view.

[0111] In one embodiment of this application, S910 includes the following S911 to S915.

[0112] S911 selects a target instrument from the previous i-frame field-of-view image.

[0113] S912, determine whether the target device in the previous i-frame view image appears in the current frame view image. Where i is a positive integer greater than or equal to 2.

[0114] S913, if it is determined that the target device in the previous i-frame visual field image does not appear in the current frame visual field image, then the target center point in the current frame visual field image is kept consistent with the target center point in the previous i-frame visual field image.

[0115] S914, if it is determined that the target device in the previous i-frame visual field image appears in the current frame visual field image, then the coordinates of the target center point are determined according to the current visual field control mode and the position of each target device in the current frame visual field image.

[0116] S915, return to the step of selecting a target device from the previous i-frame visual field image, until all target devices in the previous i-frame visual field image have been selected.

[0117] In this embodiment, if the target instrument in the previous i-frame visual field image does not appear in the current frame visual field image (i.e., there are few instruments in the current frame visual field image), the current frame visual field image will not search for the target center point again. This allows time for the target instrument that has been accidentally moved out of the visual field boundary to re-enter, reduces the time spent manipulating the target instrument to search for the visual field image during the operation, and ensures that the visual field image will not move frequently due to changes in the target center point during this period.

[0118] If the preset number of frames is exceeded and the instrument removed from the field of view does not re-enter, it indicates that the target instrument was removed normally during the surgical procedure, and the target center of the field of view can be recalculated based on the position of the target instrument.

[0119] This application also provides an endoscope system 100.

[0120] like Figure 8 As shown, in one embodiment of this application, the endoscope system 100 includes: a processor 110, an endoscope 120, a display device 130, and an endoscope arm 140.

[0121] The processor is used to execute the target tracking method mentioned above. The endoscope is electrically connected to the processor. A display device is used to display images captured by the endoscope's lens, and the display device is communicatively connected to the processor.

[0122] Specifically, the display device includes a touchscreen.

[0123] The endoscope is mounted on the endoscope-holding arm, which is a multi-jointed robotic arm used to control the endoscope to move around an RCM (remote center of motion). The endoscope-holding robotic arm is mounted on a movable trolley for easy relocation. The endoscope-holding arm is electrically connected to the processor, which controls the endoscope-holding arm to adjust the images captured by the endoscope's lens.

[0124] Figure 9 An application scenario of the endoscope system 100 of this application is shown. The scenario also shows an endoscope 120, surgical instruments 200, and an operator 300. The endoscope system of this application is used to provide a field of view to assist the operator in manipulating the surgical instruments 200.

[0125] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the target tracking method as described above.

[0126] The technical features of the above embodiments can be combined arbitrarily, and the execution order of the method steps is not restricted. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target tracking method, characterized in that, An endoscope system, comprising an endoscope arm and an endoscope, wherein the endoscope arm is used to adjust the field of view image captured by the endoscope to follow the movement of a target instrument, the target tracking method comprising: Acquire the current frame view image captured by the endoscope, and acquire the previous frame view image relative to the current frame view image; Obtain the contour region of each target device in the current frame of the field of view image, and obtain the contour region of each target device in the previous frame of the field of view image. Select a target device; The similarity algorithm is used to determine whether the target device in the current frame of the visual field image appeared in the previous frame of the visual field image; If the target device in the current frame of the field of view does not appear in the previous frame of the field of view, then the target device is defined as a device to be matched, and it is further determined whether there is at least one target device in the previous frame of the field of view to the previous k frames of the field of view where the minimum straight-line distance between the center point of the target device and the boundary of the field of view is less than a preset distance threshold; k is a positive integer greater than or equal to 2. If, in the previous frame of the visual field image to the previous k frames of the visual field image, the minimum straight-line distance between the center point of at least one target device and the boundary of the visual field image is less than a preset distance threshold, then the target device is defined as a marked device, and the marked device is compared with the device to be matched based on the device's appearance characteristics, and a device ID is assigned to the device to be matched according to the comparison result. Return to the process of selecting a target device, until all target devices in the current frame of the field of view have been selected; Based on the center point coordinates of each target device from the first frame of the field of view to the current frame of the field of view, a motion trajectory dataset corresponding to the target device is created, and the motion trajectory dataset of each target device is included in the trajectory list; The coordinates of the target center point are determined based on the current field of view control mode and the position of each target instrument in the current frame of the field of view image. The offset vector between the coordinates of the image center point and the coordinates of the target center point in the current frame of the field of view image is calculated. Based on the offset vector and the current field of view control mode, the endoscope arm is controlled to move so that the field of view image captured by the endoscope moves. Return to the current frame of the field of view image captured by the endoscope, and the previous frame of the field of view image relative to the current frame of the field of view image.

2. The target tracking method according to claim 1, characterized in that, The step of determining whether the target device in the current frame of the visual field image appeared in the previous frame of the visual field image based on a similarity algorithm includes: Calculate the area intersection-union ratio of a single target device in the current frame of the visual field image with that of each target device in the previous frame of the visual field image according to Formula 1. Where IOU(A, B) is the area intersection-union ratio, where A represents the area of ​​the contour region of a single target device in the current frame of the visual field image, B represents the area of ​​the contour region of a single target device in the previous frame of the visual field image, A∩B represents the intersection of the areas of A and B, and A∪B represents the union of the areas of A and B. Determine whether the number of target devices in the previous frame of the field of view corresponding to an area intersection-to-union ratio greater than a set intersection-to-union ratio threshold is equal to 1; If the number of target devices in the previous frame of the visual field image corresponding to an area crossover ratio greater than the set crossover ratio threshold is equal to 1, then the two target devices in the current frame of the visual field image and the previous frame of the visual field image with an area crossover ratio greater than the set crossover ratio threshold are determined to be the same target device. If the number of target devices in the previous frame of the field of view corresponding to an area crossover ratio greater than the set crossover ratio threshold is less than 1, then a single target device in the current frame of the field of view fails to find a matching target device in the previous frame of the field of view.

3. The target tracking method according to claim 2, characterized in that, After determining whether the number of target devices in the previous frame of the visual field image corresponding to an area intersection-union ratio greater than a set intersection-union ratio threshold is equal to 1, the step of determining whether the target device in the current frame of the visual field image appeared in the previous frame of the visual field image based on the similarity algorithm further includes: If the number of target devices in the previous frame of the visual field image corresponding to an area crossover ratio greater than the set crossover ratio threshold is greater than 1, then the images of each target device in the previous frame of the visual field image corresponding to an area crossover ratio greater than the set crossover ratio threshold and the target device in the current frame of the visual field image are respectively input into an appearance matching model so that the appearance matching model outputs a matching result. Based on the matching result, select one of the target instruments in the previous frame of the field of view that is the same as the target instrument in the current frame of the field of view, from each target instrument whose area crossover ratio is greater than the set crossover ratio threshold.

4. The target tracking method according to claim 1, characterized in that, Based on the device's appearance characteristics, the marked device is compared with the device to be matched, and a device ID is assigned to the device to be matched according to the comparison result, including: The images of the instrument to be matched and the marked instrument are input into an appearance matching model, so that the appearance matching model outputs whether the instrument to be matched is the same as the marked instrument. If the appearance matching model outputs the same device to be matched and the marked device, then the device to be matched is assigned the same device ID as the marked device.

5. The target tracking method according to claim 1, characterized in that, After further determining whether there exists at least one target device whose center point is less than a preset distance threshold from the boundary of the field of view image to the previous k-frame field of view image, the target tracking method further includes: If the minimum straight-line distance between the center point of each target device in the previous frame of the field of view and the boundary of the field of view is greater than or equal to a preset distance threshold, then the device to be matched is determined to be a newly added target device, and a device ID is assigned to the device to be matched. The process involves adding the center point coordinates of each target device to the trajectory list based on the device ID of each target device in the current frame's field of view image.

6. The target tracking method according to claim 1, characterized in that, The target center point is the joint center of all target instruments or the center of the main instrument in the current frame's field of view image.

7. The target tracking method according to claim 1, characterized in that, After determining whether the target device in the current frame of the visual field image appeared in the previous frame of the visual field image based on the similarity algorithm, the target tracking method further includes: If a single target device in the current frame of the field of view can be matched with a target device in the previous frame of the field of view, then the single target device in the current frame of the field of view and the matching target device in the previous frame of the field of view are assigned the same device ID. Return to the previous step of selecting a target device, until all target devices in the current frame of the field of view have been selected.

8. The target tracking method according to claim 1, characterized in that, The step of determining the coordinates of the target center point based on the current field of view control mode and the positions of each target device in the current frame of the field of view image, and calculating the offset vector between the coordinates of the image center point and the coordinates of the target center point in the current frame of the field of view image, includes: Select a target device from the previous i frames of the field of view; Determine whether the target device in the previous i frames of the field of view appears in the current frame of the field of view; where i is a positive integer greater than or equal to 2; If it is determined that the target device in the previous i-frame visual field image does not appear in the current frame visual field image, then the target center point in the current frame visual field image is kept consistent with the target center point in the previous i-frame visual field image. If it is determined that the target device in the previous i-frame visual field image appears in the current frame visual field image, then the coordinates of the target center point are determined according to the current visual field control mode and the position of each target device in the current frame visual field image. Return to the previous i-frame visual field image to select a target device, until all target devices in the previous i-frame visual field image have been selected.

9. An endoscope system, characterized in that, include: A processor for executing the target tracking method according to any one of claims 1 to 8; The endoscope is electrically connected to the processor; A display device is used to display images captured by the lens of the endoscope, and the display device is communicatively connected to the processor; Endoscope holding arm, the endoscope is mounted on the endoscope holding arm; The endoscope arm is electrically connected to the processor, and the processor controls the endoscope arm to adjust the image captured by the endoscope lens according to the image captured by the endoscope lens.

10. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the target tracking method as described in any one of claims 1 to 8.

Images