Method and device for determining clamped target and monocular endoscope system

Abstract

The invention relates to a method and device for determining a clamped target and a monocular endoscope system. The method comprises the following steps: acquiring each video frame to be identified; segmenting each to-be-identified video frame according to a preset first size to obtain each image block; determining a first horizontal offset between image blocks at the same position in each adjacent to-be-identified video frame by using an optical flow method; using a monocular depth estimation method to determine a first depth offset between the image blocks at the same position in the adjacent video frames to be identified; and determining the clamped target from the candidate targets in the to-be-identified video frames based on the first horizontal offset and the first depth offset between the image blocks located at the same position in the adjacent to-be-identified video frames. By adopting the method, the problem that the complexity and the cost of equipment are increased by the existing method for determining the clamped target can be solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for determining a clamped target and a monocular endoscope system. Background Art

[0002] With the advancement of science and technology, monocular endoscope technology has also developed to a certain extent. The use process of a monocular endoscope is to insert the monocular endoscope into the body to observe the lesion site and perform operations. In some scenarios, such as when conducting a monocular endoscope operation test, it is necessary to identify the operation of the instruments under the monocular endoscope. The traditional monocular endoscope operation test requires teachers to supervise the candidates one-on-one to determine whether the students' operations meet the requirements. Therefore, the progress of the monocular endoscope test is easily restricted by the subjective factors of the teacher. Based on this, image processing technology can be used to automatically evaluate the operation of the instruments under the monocular endoscope to reduce dependence on manual monitoring and improve the efficiency and objectivity of the monocular endoscope operation test.

[0003] In the process of automatically evaluating the operation of instruments under monocular endoscopes using image processing technology, it is necessary to determine the target clamped by the instruments under the monocular endoscope (such as clamps). However, the field of view under a monocular endoscope is relatively small, and the lens of the monocular endoscope, the clamp under the endoscope, and the clamped target are all moving. In this case, it is difficult to establish a stable spatial coordinate system to locate the clamped target. In addition, due to the parallax relationship, it is difficult for the two-dimensional image to accurately reflect whether the clamp is clamping the target.

[0004] Based on this, the prior art adds cameras to the original equipment, uses multiple cameras to obtain stereoscopic visual information, and determines the target clamped by the clamp through the stereoscopic visual information. The design of using multiple cameras changes the structure of the original instrument, and increases the complexity and cost of the equipment.

[0005] The existing method for determining the clamped target has the problem of increasing the complexity and cost of the equipment, and no effective solution has been proposed so far. Summary of the invention

[0006] Based on this, it is necessary to provide a method and device for determining a clamped target and a monocular endoscope system to address the above technical problems.

[0007] In a first aspect, the present application provides a method for determining a clamped target. For use in a monocular endoscope system, the method comprises the following steps:

[0008] Acquire each video frame to be identified that is captured by the monocular endoscope;

[0009] Splitting each of the to-be-recognized video frames according to a preset first size to obtain image blocks of each of the to-be-recognized video frames;

[0010] Determine, by using an optical flow method, a first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified;

[0011] Determine, by using a monocular depth estimation method, a first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified;

[0012] Based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, the clamped target is determined from the candidate targets in the video frames to be identified.

[0013] In one of the embodiments, determining the clamped target from the candidate targets in the video frames to be identified based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified comprises the following steps:

[0014] Based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified, and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified, determine the second horizontal offset and the second depth offset of the candidate targets in the video frames to be identified between the adjacent video frames to be identified; the second horizontal offset is a horizontal offset generated due to the movement of the candidate targets; the second depth offset is a depth offset generated due to the movement of the candidate targets;

[0015] The clamped target is determined from the candidate targets in the video frames to be identified based on the second horizontal offset and the second depth offset of each of the candidate targets in the video frames to be identified between each of the adjacent video frames to be identified.

[0016] In one of the embodiments, determining the second horizontal offset and the second depth offset of each of the candidate targets in the video frame to be identified between each of the adjacent video frames to be identified based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified comprises the following steps:

[0017] Based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, determine a third horizontal offset and a third depth offset of each of the candidate targets in the video frames to be identified between each of the adjacent video frames to be identified; the third horizontal offset is a horizontal offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope; the third depth offset is a depth offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope;

[0018] Determine an effective horizontal offset and an effective depth offset of each of the adjacent video frames to be identified based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified;

[0019] Based on the effective horizontal offset and the effective depth offset of each of the adjacent video frames to be identified, determine a fourth horizontal offset and a fourth depth offset between each of the adjacent video frames to be identified; the fourth horizontal offset is a horizontal offset generated due to the movement of the lens of the monocular endoscope; the fourth depth offset is a depth offset generated due to the movement of the lens of the monocular endoscope;

[0020] Based on the third horizontal offset and the third depth offset of each of the candidate targets between each of the adjacent video frames to be identified, and the fourth horizontal offset and the fourth depth offset between each of the adjacent video frames to be identified, determine the second horizontal offset and the second depth offset of each of the candidate targets in the video frames to be identified between each of the adjacent video frames to be identified.

[0021] In one of the embodiments, determining the effective horizontal offset and effective depth offset of each of the adjacent video frames to be identified based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified comprises the following steps:

[0022] Sorting each of the first horizontal offsets in a descending order or a descending order to obtain the sorted first horizontal offsets; based on the sorted first horizontal offsets, determining the first horizontal offsets sorted within a first preset threshold to a second preset threshold as the effective horizontal offsets of each of the adjacent video frames to be identified; sorting each of the first depth offsets in a descending order or a descending order to obtain the sorted first depth offsets; based on the sorted first depth offsets, determining the first depth offsets sorted within a third preset threshold to a fourth preset threshold as the effective depth offsets of each of the adjacent video frames to be identified;

[0023] Or, the first horizontal offsets between the image blocks of the adjacent video frames to be identified are eliminated, and the first horizontal offsets that are smaller than a preset horizontal offset threshold are obtained to obtain the preliminary screening effective horizontal offsets between the image blocks of the adjacent video frames to be identified; the average value of the preliminary screening effective horizontal offsets between the image blocks of the adjacent video frames to be identified is calculated to obtain the horizontal level offset; the first horizontal offsets that are smaller than the horizontal level offset in the preliminary screening effective horizontal offsets are determined as the effective horizontal offsets of the adjacent video frames to be identified; Eliminate the first depth offsets between the image blocks of each of the adjacent video frames to be identified, and those that are less than a preset depth offset threshold, to obtain a preliminary screening effective depth offset between the image blocks of each of the adjacent video frames to be identified; calculate the average value of the preliminary screening effective depth offsets between the image blocks of each of the adjacent video frames to be identified, to obtain an average depth offset; determine the first depth offsets in the preliminary screening effective depth offsets that are less than the average depth offset as the effective depth offset of each of the adjacent video frames to be identified.

[0024] In one of the embodiments, determining the clamped target from the candidate targets in the video frame to be identified based on the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified comprises the following steps:

[0025] Determine, based on the second horizontal offset and the second depth offset of each of the candidate targets in the video frame to be identified between each of the adjacent video frames to be identified, an average value of the offsets of each of the candidate targets in each direction between each of the adjacent video frames to be identified;

[0026] The candidate target having the largest average value of the offsets in each direction between the adjacent video frames to be identified is determined as the clamped target.

[0027] In one embodiment, the step of obtaining each video frame to be identified captured by the monocular endoscope comprises the following steps:

[0028] Acquire a video stream captured by the monocular endoscope;

[0029] The video stream captured by the monocular endoscope is frame-extracted at a preset frequency to obtain each of the video frames to be identified captured by the monocular endoscope.

[0030] In one of the embodiments, before determining the target to be clamped from the candidate targets in the video frame to be identified, the following steps are also included:

[0031] Performing target recognition on each of the video frames to be recognized, and obtaining the clamps and each target to be clamped in each of the video frames to be recognized;

[0032] Based on the intersection-and-union ratio between the clamp and each target to be clamped in each video frame to be identified, each candidate target in the video frame to be identified is obtained.

[0033] In one of the embodiments, before each of the to-be-identified video frames is segmented according to a preset size to obtain each image block of each of the to-be-identified video frames, the following steps are also included:

[0034] Each of the to-be-identified video frames is scaled or expanded to a preset second size; the preset second size is larger than the preset first size.

[0035] In a second aspect, the present application also provides a device for determining a clamped target. For use in a monocular endoscope system, the device comprises:

[0036] An acquisition module, used for acquiring each video frame to be identified captured by the monocular endoscope;

[0037] A segmentation module, used for segmenting each of the to-be-identified video frames according to a preset first size to obtain image blocks of each of the to-be-identified video frames;

[0038] A horizontal offset calculation module, used to determine a first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified by using an optical flow method;

[0039] A depth offset calculation module, used to determine a first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified by using a monocular depth estimation method;

[0040] And a target determination module is used to determine the clamped target from the candidate targets in the video frames to be identified based on the first horizontal offset between the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in each of the adjacent video frames to be identified.

[0041] In a third aspect, the present application further provides a monocular endoscope system, which includes a monocular lens and an operation recognition system, wherein the monocular lens is used to capture a video stream, and the operation recognition system is used to execute the steps of the method for determining a clamped target as described in the first aspect.

[0042] The above-mentioned method and device for determining the clamped target and a monocular endoscope system divide the video frame to be identified into image blocks, and use the optical flow method to determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and use the monocular depth estimation method to determine the first depth offset between each image block located at the same position in each adjacent video frame to be identified, thereby realizing the determination of the first horizontal offset and the first depth offset between the frames of each video block, determining the three-dimensional change of each video block, and then determining the clamped target from the candidate targets in the video frame to be identified based on the first horizontal offset and the first depth offset between the frames of each video block. The whole process only involves the video frames collected by the monocular endoscope, and does not require changing the structure of the device. The existing method for determining the clamped target solves the problem of increased complexity and cost of the device.

[0043] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 A hardware structure block diagram of a terminal for a method for determining a clamped target provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of a method for determining a clamped target provided in an embodiment of the present application;

[0047] Figure 3 A flowchart of a method for determining a clamped target provided in a preferred embodiment of the present application;

[0048] Figure 4 A structural block diagram of a device for determining a clamped target provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 4 is a hardware structure diagram of a terminal of the method for determining a clamped target in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0052] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the clamped target in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0054] In this embodiment, a method for determining a clamped target is provided, which is used in a monocular endoscope system. Figure 2 is a flow chart of the method for determining the clamped target of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0055] Step S210, obtaining each video frame to be identified that is captured by the monocular endoscope.

[0056] The above-mentioned monocular endoscope may be an endoscope device that uses a single lens for image acquisition. That is, an endoscope device with only one camera. The above-mentioned video frames to be identified may be video frames taken by the monocular endoscope and used to determine the clamped target. Specifically, the above-mentioned video frames to be identified may be a plurality of continuous video frames recording the process from the target being clamped to the target being lifted.

[0057] Step S220 , dividing each to-be-recognized video frame into segments according to a preset first size to obtain image blocks of each to-be-recognized video frame.

[0058] The above-mentioned preset first size can be specifically set according to the specific situation, and this embodiment does not make any specific limitation here. It should be noted that the segmentation of each video frame to be identified is actually to ensure that there are image blocks in the segmented image blocks that do not contain the clamped target and do not contain pixel points with inaccurate motion changes at the edge of the image frame. Therefore, the preset first size can be determined according to the size of each image frame and the size of the candidate target in each image frame. As long as the video frame to be identified is segmented according to the preset first size, there are image blocks in the segmented image blocks that do not contain the clamped target and do not contain pixel points with inaccurate motion changes at the edge of the image frame. For example, if the video frame size is 256×256, the video frame can be segmented in a size of 8×8. It should be noted that when the preset first size is set to be small, the amount of calculation will increase. When the preset first size is set to be large, there may be no image blocks that do not contain the clamped target and do not contain pixel points with inaccurate edge motion changes of the image frame, resulting in the inability to cut the image blocks and determine the clamped target from the candidate targets in the video frame to be identified based on the first horizontal offset and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified. Therefore, the preset first size should be set to a suitable size. For example, one quarter of the width or length of one or more candidate targets in the video frame to be identified can be used as the side length of the square area of ​​the preset first size. In addition, the image blocks obtained by segmentation according to the preset first size are equal in size, and can be square areas of equal size, or rectangular areas of equal size, etc. In order to simplify the calculation process, the segmented image blocks in this embodiment are square areas of equal size.

[0059] Step S230: using the optical flow method, determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified.

[0060] The above optical flow method may be one of the RAFT (Recursive Application of Flow Transformations, an optical flow algorithm based on deep learning) optical flow method, Lucas-Kanade optical flow method, Horn-Schunck optical flow method, Farneback optical flow method, etc. The above optical flow method is used to determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified. The optical flow method may be used to calculate the optical flow field between each image block located at the same position in each adjacent video frame to be identified, and then, based on the optical flow field between each image block located at the same position in each adjacent video frame to be identified, the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified is determined. The above horizontal offset includes a horizontal coordinate offset and a vertical coordinate offset.

[0061] The above method determines the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified based on the optical flow field between the image blocks located at the same position in the adjacent video frames to be identified. The horizontal coordinate offset and the vertical coordinate offset of each pixel point in the optical flow field between the image blocks located at the same position in the adjacent video frames to be identified can be extracted, and the horizontal coordinate offset of each pixel point in the optical flow field between the image blocks located at the same position in the adjacent video frames to be identified is averaged to obtain the horizontal coordinate offset between the image blocks located at the same position in the adjacent video frames to be identified, and the vertical coordinate offset of each pixel point in the optical flow field between the image blocks located at the same position in the adjacent video frames to be identified is averaged to obtain the vertical coordinate offset between the image blocks located at the same position in the adjacent video frames to be identified. Based on the horizontal coordinate offset and the vertical coordinate offset between the image blocks located at the same position in the adjacent video frames to be identified, the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified is determined.

[0062] Preferably, before using the optical flow method to determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, the video frame to be identified may be preprocessed to obtain a preprocessed video frame to be identified. The above preprocessing may include grayscale, mean subtraction, normalization, etc.

[0063] Step S240: using a monocular depth estimation method, determine a first depth offset between image blocks located at the same position in adjacent video frames to be identified.

[0064] Among them, the above-mentioned monocular depth estimation method can be a monocular depth estimation method based on deep learning (such as the unsupervised learning method of the Monodepth series), a DORN (Depth OF Rest Network, a deep learning model for monocular depth estimation) monocular depth estimation method, a depth estimation method based on the Transformer architecture, etc.

[0065] The above-mentioned use of the monocular depth estimation method to determine the first depth offset between each image block located at the same position in each adjacent video frame to be identified may be to use the monocular depth estimation method to determine the depth value of each image block located at the same position in each video frame to be identified, and based on the depth value of each image block located at the same position in each video frame to be identified, the depth value of each image block located at the same position in two adjacent video frames to be identified is subtracted to obtain the first depth offset between each image block located at the same position in each adjacent video frame to be identified. The above-mentioned use of the monocular depth estimation method to determine the depth value of each image block located at the same position in each video frame to be identified may be to use a preset depth estimation model to perform depth estimation on each image block located at the same position in each video frame to be identified, and obtain the depth value of each image block located at the same position in each video frame to be identified.

[0066] Step S250, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determine the clamped target from the candidate targets in the video frame to be identified.

[0067] The above-mentioned method of determining the clamped target from the candidate targets in the video frames to be identified based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified may be to determine the horizontal offset and the depth offset of each candidate target in the video frames to be identified between the adjacent video frames to be identified due to the movement of the candidate targets based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified, and then, determine the clamped target from the candidate targets in the video frames to be identified according to the horizontal offset and the depth offset of each candidate target in the video frames to be identified between the adjacent video frames to be identified due to the movement of the candidate targets.

[0068] In the above steps S210 to S250, the video frames to be identified are divided into image blocks, and the optical flow method is used to determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the monocular depth estimation method is used to determine the first depth offset between each image block located at the same position in each adjacent video frame to be identified, so as to determine the first horizontal offset and the first depth offset between frames of each video block, determine the three-dimensional change of each video block, and then determine the clamped target from the candidate targets in the video frame to be identified according to the first horizontal offset and the first depth offset between frames of each video block. The whole process only involves the video frames collected by the monocular endoscope, and does not require changing the structure of the device. The existing method for determining the clamped target solves the problem of increased complexity and cost of the device.

[0069] In one embodiment, the above step S250, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determines the target to be clamped from the candidate targets in the video frame to be identified, and includes the following steps:

[0070] Step S252, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determine the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified; the second horizontal offset is the horizontal offset generated due to the movement of the candidate target; the second depth offset is the depth offset generated due to the movement of the candidate target.

[0071] In this step, the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified are determined based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified and the first depth offset between each image block located at the same position in each adjacent video frame to be identified. It can be based on the first horizontal offset and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, to determine the horizontal offset and the depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified due to the joint movement of the candidate target and the lens of the monocular endoscope, and then determine the horizontal offset and the depth offset of the lens of the monocular endoscope between each adjacent video frame to be identified due to the movement of the lens of the monocular endoscope, and then obtain the second horizontal offset and the second depth offset of the lens of the monocular endoscope between each adjacent video frame to be identified due to the movement of the candidate target.

[0072] Step S254 , determining the clipped target from the candidate targets in the video frames to be identified based on the second horizontal offset and the second depth offset of each candidate target in the video frames to be identified between each adjacent video frames to be identified.

[0073] Among them, based on the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, the clamped target is determined from the candidate targets in the video frame to be identified. It can be based on the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, and the candidate target with the largest average value of the offset in each direction between each adjacent video frame to be identified is determined as the clamped target.

[0074] In the above steps S252 to S254, the clamped target is determined from the candidate targets in the video frames to be identified by determining the second horizontal offset and the second depth offset generated by the movement of each candidate target in the video frames to be identified between each adjacent video frames to be identified, and then based on the second horizontal offset and the second depth offset generated by the movement of the lens of the monocular endoscope.

[0075] Specifically, in one embodiment, based on step S252, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determining the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified includes the following steps:

[0076] Step S2522, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determine the third horizontal offset and the third depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified; the third horizontal offset is the horizontal offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope; the third depth offset is the depth offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope.

[0077] The above-mentioned determination of the third horizontal offset and the third depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, may be based on the position information of each candidate target, to determine the image block corresponding to each candidate target, and based on the first horizontal offset and the first depth offset of the image block corresponding to each candidate target between each adjacent video frame to be identified, to determine the third horizontal offset and the third depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified. The above-mentioned determination of the image block corresponding to each candidate target based on the position information of each candidate target may be to take the image block completely covered by the candidate target (that is, all the pixels of the image block are within the area of ​​the candidate target) as the image block corresponding to the current candidate target. The above-mentioned method of determining the third horizontal offset and the third depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified based on the first horizontal offset and the first depth offset of the image blocks corresponding to each candidate target between each adjacent video frame to be identified may be to use the average value of multiple first horizontal offsets of multiple image blocks corresponding to each candidate target between each adjacent video frame to be identified as the third horizontal offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, and to use the average value of multiple first depth offsets of multiple image blocks corresponding to each candidate target between each adjacent video frame to be identified as the third depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified.

[0078] Step S2524, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determine the effective horizontal offset and effective depth offset of each adjacent video frame to be identified.

[0079] The above-mentioned effective horizontal offset may be a horizontal offset generated by the movement of the lens of the monocular endoscope in the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified as the effective horizontal offset. The above-mentioned effective depth offset may be a depth offset generated by the movement of the lens of the monocular endoscope in the first depth offset between each image block located at the same position in each adjacent video frame to be identified as the effective depth offset. The above-mentioned determination of the effective horizontal offset and the effective depth offset of each adjacent video frame to be identified based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified and the first depth offset between each image block located at the same position in each adjacent video frame to be identified may be performed by filtering out the effective horizontal offset from the first horizontal offset between each image block and filtering out the effective depth offset from the first depth offset between each image block according to a preset filtering rule. The above-mentioned preset screening rules can be to set preset thresholds to eliminate horizontal offsets less than the first preset threshold and horizontal offsets greater than the second preset threshold in the first horizontal offsets, and to eliminate depth offsets less than the third preset threshold and depth offsets greater than the fourth preset threshold in the first depth offsets. Alternatively, the first horizontal offset corresponding to the image block including the black border in the first horizontal offset is first eliminated, and the remaining first horizontal offsets are averaged to obtain an average horizontal offset. Generally, the remaining first horizontal offset includes a horizontal offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope and a horizontal offset generated due to the movement of the lens of the monocular endoscope. A horizontal offset smaller than the average horizontal offset can be considered to be a horizontal offset generated due to the movement of the lens of the monocular endoscope. Based on this, an effective horizontal offset can be screened out. First, the first depth offset corresponding to the image block including the black border in the first depth offset is eliminated, and the remaining first depth offsets are averaged to obtain an average depth offset. Generally, the remaining first depth offset includes a depth offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope and a depth offset generated due to the movement of the lens of the monocular endoscope. A depth offset smaller than the average depth offset can be considered to be a depth offset generated due to the movement of the lens of the monocular endoscope. Based on this, an effective depth offset can be screened out.

[0080] Step S2526, based on the effective horizontal offset and effective depth offset of each adjacent video frame to be identified, determine the fourth horizontal offset and fourth depth offset between each adjacent video frame to be identified; the fourth horizontal offset is the horizontal offset generated due to the movement of the lens of the monocular endoscope; the fourth depth offset is the depth offset generated due to the movement of the lens of the monocular endoscope.

[0081] The fourth horizontal offset and the fourth depth offset between each adjacent video frame to be identified are determined based on the effective horizontal offset and the effective depth offset of each adjacent video frame to be identified. The average value of each effective horizontal offset of each adjacent video frame to be identified may be used as the fourth horizontal offset between each adjacent video frame to be identified; and the average value of each effective depth offset of each adjacent video frame to be identified may be used as the fourth depth offset between each adjacent video frame to be identified.

[0082] Step S2528, based on the third horizontal offset and the third depth offset of each candidate target between each adjacent video frame to be identified, and the fourth horizontal offset and the fourth depth offset between each adjacent video frame to be identified, determine the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified.

[0083] Wherein, based on the third horizontal offset and the third depth offset of each candidate target between each adjacent video frame to be identified, and the fourth horizontal offset and the fourth depth offset between each adjacent video frame to be identified, the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified are determined, and the horizontal coordinate offset of the third horizontal offset of each candidate target between each adjacent video frame to be identified is subtracted from the horizontal coordinate offset of the fourth horizontal offset between each adjacent video frame to be identified, as the horizontal coordinate offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified. The horizontal coordinate offset of the second horizontal offset of each candidate target between each adjacent video frame to be identified is subtracted from the vertical coordinate offset of the fourth horizontal offset between each adjacent video frame to be identified from the vertical coordinate offset of the third horizontal offset of each candidate target between each adjacent video frame to be identified, as the vertical coordinate offset of the second horizontal offset of each candidate target between each adjacent video frame to be identified; the third depth offset of each candidate target between each adjacent video frame to be identified is subtracted from the fourth depth offset between each adjacent video frame to be identified, as the second depth offset of each candidate target between each adjacent video frame to be identified.

[0084] In the above steps S2522 to S2528, by calculating the third horizontal offset and the third depth offset generated by the joint movement of the candidate target and the lens of the monocular endoscope, and calculating the fourth horizontal offset and the fourth depth offset between each adjacent video frame to be identified, the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, which are generated only because of the movement of the candidate target. The determination of the second horizontal offset and the second depth offset facilitates the subsequent determination of the clamped target from the candidate targets in the video frame to be identified based on the second horizontal offset and the second depth offset.

[0085] In addition, in one embodiment, the above step S2524, based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified and the first depth offset between each image block located at the same position in each adjacent video frame to be identified, determines the effective horizontal offset and the effective depth offset of each adjacent video frame to be identified, comprising the following steps:

[0086] Step S25241, sorting the first horizontal offsets in order from large to small or from small to large, to obtain sorted first horizontal offsets.

[0087] Step S25242: Based on the sorted first horizontal offsets, the first horizontal offsets sorted within a range from the first preset threshold to the second preset threshold are determined as valid horizontal offsets of each adjacent video frame to be identified.

[0088] The first preset threshold can be set based on experience, and its purpose is to filter out the first horizontal offset corresponding to the image block including the black edge area in the image. Because the offset of the image block including the black edge area in the adjacent frames is not obvious, the first preset threshold can be set to eliminate the first horizontal offset that is sorted before the first preset threshold. The second preset threshold can be set based on experience, and its purpose is to filter out the first horizontal offset generated by the joint movement of the candidate target and the lens of the monocular endoscope. Because the offset of the first horizontal offset generated by the joint movement of the candidate target and the lens of the monocular endoscope is large and ranked later, the second preset threshold can be set to eliminate the first horizontal offset that is sorted after the second preset threshold. In this step, the effective horizontal offset of each adjacent video frame to be identified is obtained by eliminating the first horizontal offset corresponding to the image block including the black edge area and the first horizontal offset generated by the joint movement of the candidate target and the lens of the monocular endoscope, so as to facilitate the determination of the fourth horizontal offset between each adjacent video frame to be identified according to the effective horizontal offset of each adjacent video frame to be identified. For example, the first preset threshold can be set to 10% and the second preset threshold can be set to 80%. The second preset threshold is greater than the first preset threshold.

[0089] Step S25243, sorting the first depth offsets in order from large to small or from small to large, to obtain sorted first depth offsets.

[0090] Step S25244: Based on the sorted first depth offsets, the first depth offsets sorted within the range from the third preset threshold to the fourth preset threshold are determined as effective depth offsets of each adjacent video frame to be identified.

[0091] The third preset threshold can be set based on experience, and its purpose is to filter out the first depth offset corresponding to the image block including the black edge area in the image. Because the offset of the image block including the black edge area in the adjacent frames is not obvious, the third preset threshold can be set to eliminate the first depth offset that is ranked before the third preset threshold. The fourth preset threshold can be set based on experience, and its purpose is to filter out the first depth offset generated by the joint movement of the candidate target and the lens of the monocular endoscope. Because the offset of the first depth offset generated by the joint movement of the candidate target and the lens of the monocular endoscope is large and ranked later, the fourth preset threshold can be set to eliminate the first depth offset that is ranked after the fourth preset threshold. In this step, the effective depth offset of each adjacent video frame to be identified is obtained by eliminating the first depth offset corresponding to the image block including the black edge area and the first depth offset generated by the joint movement of the candidate target and the lens of the monocular endoscope, so as to facilitate the determination of the fourth depth offset of the lens of the monocular endoscope between each adjacent video frame to be identified according to the effective depth offset of each adjacent video frame to be identified. For example, the third preset threshold may be set to 30% and the fourth preset threshold may be set to 73%. It should be noted that the fourth preset threshold is greater than the third preset threshold.

[0092] Or, step S25245, eliminate the first horizontal offsets between the image blocks of each adjacent video frame to be identified, which are smaller than a preset horizontal offset threshold, to obtain the preliminary screening effective horizontal offsets between the image blocks of each adjacent video frame to be identified.

[0093] The above-mentioned preset horizontal offset threshold can be determined based on experience. Because the offset of the image block including the black border area in the adjacent frames is not obvious, the first horizontal offset corresponding to the image block including the black border area can be eliminated by setting a preset horizontal offset threshold. The above-mentioned initial screening effective horizontal offset is the remaining first horizontal offset after eliminating the first horizontal offset corresponding to the image block including the black border area.

[0094] Step S25246, calculate the average value of the preliminary screening effective horizontal offsets between each image block of each adjacent video frame to be identified, and obtain the horizontal average offset; determine the first horizontal offset among the preliminary screening effective horizontal offsets that is smaller than the horizontal average offset as the effective horizontal offset of each adjacent video frame to be identified.

[0095] Step S25247, eliminate the first depth offsets between the image blocks of each adjacent video frame to be identified, which are smaller than a preset depth offset threshold, to obtain the preliminary screening effective depth offsets between the image blocks of each adjacent video frame to be identified.

[0096] The above-mentioned preset depth offset threshold can be determined based on experience. Because the offset of the image block including the black border area in the adjacent frame is not obvious, the first depth offset corresponding to the image block including the black border area can be eliminated by setting a preset depth offset threshold. The above-mentioned initial screening effective depth offset is the remaining first depth offset after eliminating the first depth offset corresponding to the image block including the black border area.

[0097] Step S25248, calculate the average value of the initial screening effective depth offsets between each image block of each adjacent video frame to be identified, and obtain the average depth offset.

[0098] Step S25249, determining the first depth offset that is smaller than the average depth offset among the initially screened effective depth offsets as the effective depth offset of each adjacent video frame to be identified.

[0099] The above steps S25241 to S25249, by determining the effective horizontal offset and effective depth offset of each adjacent video frame to be identified, facilitate the subsequent determination of the fourth horizontal offset and fourth depth offset between each adjacent video frame to be identified based on the effective horizontal offset and effective depth offset of each adjacent video frame to be identified.

[0100] In one embodiment, based on step S254, based on the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, determining the target to be clamped from the candidate targets in the video frame to be identified includes:

[0101] Step S2542, based on the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, determine the average value of the offset of each candidate target in each direction between each adjacent video frame to be identified.

[0102] The above directions may include a horizontal coordinate direction, a vertical coordinate direction and a depth direction, wherein the offset in the horizontal coordinate direction is the horizontal coordinate offset, the offset in the vertical coordinate direction is the vertical coordinate offset, and the offset in the depth direction is the depth offset.

[0103] Step S2544, the candidate target with the largest average value of the offset in each direction between each adjacent video frame to be identified is determined as the target to be clamped.

[0104] In the above steps S2542 to S2544, the average value of the offsets of each candidate target in each adjacent video frame to be identified is determined by using the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified, and then the candidate target with the largest average value of the offsets in each direction between each adjacent video frame to be identified is determined as the target to be clamped. Because the second horizontal offset is the horizontal offset generated due to the movement of the candidate target, and the second depth offset is the depth offset generated due to the movement of the candidate target, the candidate target with the largest average value of the offsets in each direction between each adjacent video frame to be identified is the candidate target with the largest actual movement, that is, the target to be clamped.

[0105] In addition, in one embodiment, the above step S210, obtaining each video frame to be identified captured by the monocular endoscope, includes:

[0106] Step S212, obtaining the video stream captured by the monocular endoscope.

[0107] The video stream captured by the monocular endoscope may be a video stream captured by the endoscope and recording the process from the target being clamped to the target being lifted during the automatic evaluation of the operation of the instrument under the monocular endoscope.

[0108] Step S214: extract frames from the video stream captured by the monocular endoscope at a preset frequency to obtain various video frames to be identified that are captured by the monocular endoscope.

[0109] The above preset frequency can be set according to specific circumstances or specific requirements, and this embodiment does not specifically limit it. For example, the preset frequency can be 1 frame per second, that is, one frame of image is extracted every 1 second.

[0110] In the above steps S212 to S214, the video stream captured by the monocular endoscope is acquired, and frames are extracted at a preset frequency to obtain each video frame to be identified captured by the monocular endoscope. By determining each video frame to be identified captured by the monocular endoscope, it is convenient to determine the clamped target according to each video frame to be identified captured by the monocular endoscope.

[0111] Furthermore, in one embodiment, before determining the target to be clamped from the candidate targets in the video frame to be identified, the method further includes:

[0112] Step S242, performing target recognition on each video frame to be recognized, and obtaining the clamps and each target to be clamped in each video frame to be recognized.

[0113] In this step, the target recognition for each video frame to be recognized can be performed by using a preset target recognition method to recognize the target for each video frame to be recognized. The preset target recognition method can be one or more of the methods such as R-CNN (Region-based Convolutional Neural Networks, a region-based convolutional neural network model), YOLO (You Only Look Once, a popular target detection algorithm), Faster R-CNN (Faster Region-based Convolutional Neural Network, a faster region-based convolutional neural network), DETR (DEtection TRansformer, a Transformer model-based and used for target detection architecture). The target to be clamped can be a target based on a specific application scenario, for example, it can be a tissue sample, a foreign body (such as a swallowed foreign body), a surgical residue, a diseased tissue, an implant (such as a drainage tube, a stent), etc.

[0114] Step S244, obtaining each candidate target in the video frame to be identified based on the intersection-and-union ratio between the clamp and each target to be clamped in each video frame to be identified.

[0115] The above-mentioned method of obtaining each candidate target in the video frame to be identified based on the intersection-and-union ratio between the clamps and each target to be clamped in each video frame to be identified may be to calculate each intersection-and-union ratio between the clamps and each target to be clamped in each video frame to be identified, and then, based on each intersection-and-union ratio between the clamps and each target to be clamped in each video frame to be identified, select a preset number of targets to be clamped with the highest intersection-and-union ratio between the clamps as each candidate target in the video frame to be identified. The above-mentioned method of selecting a preset number of targets to be clamped with the highest intersection-and-union ratio between the clamps and each target to be clamped as each candidate target in the video frame to be identified based on each intersection-and-union ratio between the clamps and each target to be clamped in each video frame to be identified may be to calculate the average value of the intersection-and-union ratio between each target to be clamped and the clamps in each video frame to be identified, and select the preset number of targets to be clamped with the highest average value of the intersection-and-union ratio between the clamps as each candidate target in the video frame to be identified. The above-mentioned preset number may be set according to the specific situation, for example, the preset number may be set to 3.

[0116] In the above steps S242 to S244, by performing target recognition on each video frame to be recognized, the clamps and each target to be clamped in each video frame to be recognized are obtained, and then based on the intersection-and-union ratio between the clamps and each target to be clamped in each video frame to be recognized, each candidate target in the video frame to be recognized is obtained, so as to facilitate the determination of the clamped target among the candidate targets. By determining the candidate targets, the targets that are obviously not clamped are screened out, thereby reducing the amount of calculation in the process of determining the clamped targets.

[0117] In one embodiment, before step S220, the method further includes:

[0118] Step S216, scaling or expanding each to-be-identified video frame to a preset second size; the preset second size is larger than the preset first size.

[0119] The above-mentioned preset second size can be specifically set according to specific requirements. For example, an interpolation method (such as bilinear interpolation) can be used to scale the video frame to be identified to a size of 256×256.

[0120] The present embodiment is described and illustrated below through preferred embodiments.

[0121] Figure 3 FIG. 1 is a flow chart of a method for determining a clamped target provided by a preferred embodiment of the present application. Figure 3 As shown, the method for determining the clamped target includes the following steps:

[0122] Step S310, acquiring each video frame to be identified captured by the monocular endoscope;

[0123] Step S320, dividing each video frame to be identified according to a preset first size to obtain each image block of each video frame to be identified;

[0124] Step S330, using an optical flow method, determining a first horizontal offset between image blocks located at the same position in adjacent video frames to be identified;

[0125] Step S340, using a monocular depth estimation method, determining a first depth offset between image blocks located at the same position in adjacent video frames to be identified;

[0126] Step S350, determining a second horizontal offset and a second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified based on a first horizontal offset between each image block located at the same position in each adjacent video frame to be identified and a first depth offset between each image block located at the same position in each adjacent video frame to be identified;

[0127] Step S360, determining an average value of the offsets of each candidate target in each adjacent video frame to be identified in each direction based on the second horizontal offset and the second depth offset of each candidate target in the video frame to be identified between each adjacent video frame to be identified;

[0128] Step S370 , determining the candidate target with the largest average value of the offsets in each direction between each adjacent video frame to be identified as the target to be clamped.

[0129] In the above steps S310 to S370, the video frames to be identified are divided into image blocks, and the optical flow method is used to determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and the monocular depth estimation method is used to determine the first depth offset between each image block located at the same position in each adjacent video frame to be identified, so as to determine the first horizontal offset and the first depth offset between frames of each video block, determine the three-dimensional change of each video block, and then determine the clamped target from the candidate targets in the video frame to be identified according to the first horizontal offset and the first depth offset between frames of each video block. The whole process only involves the video frames collected by the monocular endoscope, and does not require changing the structure of the device. The existing method for determining the clamped target solves the problem of increased complexity and cost of the device.

[0130] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, a device for determining a clamped target is also provided in this embodiment, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0132] In one embodiment, Figure 4 is a structural block diagram of a device for determining a clamped target provided in an embodiment of the present application, such as Figure 4 As shown, the device for determining the clamped target is used in a monocular endoscope system, comprising:

[0133] An acquisition module 41 is used to acquire each video frame to be identified that is captured by the monocular endoscope;

[0134] A segmentation module 42, configured to segment each to-be-identified video frame according to a preset first size to obtain image blocks of each to-be-identified video frame;

[0135] A horizontal offset determination module 43 is used to determine a first horizontal offset between image blocks located at the same position in adjacent video frames to be identified by using an optical flow method;

[0136] A depth offset determination module 44 is used to determine a first depth offset between each image block located at the same position in each adjacent video frame to be identified by using a monocular depth estimation method;

[0137] And a target determination module 45 is used to determine the clamped target from the candidate targets in the video frames to be identified based on the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified and the first depth offset between each image block located at the same position in each adjacent video frame to be identified.

[0138] The above-mentioned device for determining the clamped target divides the video frame to be identified into image blocks, and uses the optical flow method to determine the first horizontal offset between each image block located at the same position in each adjacent video frame to be identified, and uses the monocular depth estimation method to determine the first depth offset between each image block located at the same position in each adjacent video frame to be identified, thereby realizing the determination of the first horizontal offset and the first depth offset between frames of each video block, determining the three-dimensional change of each video block, and then determining the clamped target from the candidate targets in the video frame to be identified according to the first horizontal offset and the first depth offset between frames of each video block. The whole process only involves the video frames collected by the monocular endoscope, and does not need to change the structure of the equipment. The problem of increasing the complexity and cost of the equipment in the existing method for determining the clamped target is solved.

[0139] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0140] In one embodiment, a monocular endoscope system is provided, including a monocular lens and an operation recognition system, wherein the monocular lens is used to capture a video stream, and the operation recognition system implements any one of the methods for determining a clamped target in the above embodiments.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for determining a clamped target, for use in a monocular endoscope system, characterized in that: The method comprises: Acquire each video frame to be identified that is captured by the monocular endoscope; Splitting each of the to-be-recognized video frames according to a preset first size to obtain image blocks of each of the to-be-recognized video frames; Determine, by using an optical flow method, a first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified; Determine, by using a monocular depth estimation method, a first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified; Based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, the clamped target is determined from the candidate targets in the video frames to be identified.

2. The method for determining a clamped target according to claim 1, characterized in that: The method of determining the clamped target from the candidate targets in the video frames to be identified based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified comprises: Based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified, and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified, determine the second horizontal offset and the second depth offset of the candidate targets in the video frames to be identified between the adjacent video frames to be identified; the second horizontal offset is a horizontal offset generated due to the movement of the candidate targets; the second depth offset is a depth offset generated due to the movement of the candidate targets; The clamped target is determined from the candidate targets in the video frames to be identified based on the second horizontal offset and the second depth offset of each of the candidate targets in the video frames to be identified between each of the adjacent video frames to be identified.

3. The method for determining a clamped target according to claim 2, characterized in that: The determining, based on the first horizontal offset between the image blocks located at the same position in the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in the adjacent video frames to be identified, a second horizontal offset and a second depth offset of each candidate target in the video frames to be identified between each adjacent video frames to be identified, comprises: Based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified, determine a third horizontal offset and a third depth offset of each of the candidate targets in the video frames to be identified between each of the adjacent video frames to be identified; the third horizontal offset is a horizontal offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope; the third depth offset is a depth offset generated due to the joint movement of the candidate target and the lens of the monocular endoscope; Determine an effective horizontal offset and an effective depth offset of each of the adjacent video frames to be identified based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified; Based on the effective horizontal offset and the effective depth offset of each of the adjacent video frames to be identified, determine a fourth horizontal offset and a fourth depth offset between each of the adjacent video frames to be identified; the fourth horizontal offset is a horizontal offset generated due to the movement of the lens of the monocular endoscope; the fourth depth offset is a depth offset generated due to the movement of the lens of the monocular endoscope; Based on the third horizontal offset and the third depth offset of each of the candidate targets between each of the adjacent video frames to be identified, and the fourth horizontal offset and the fourth depth offset between each of the adjacent video frames to be identified, determine the second horizontal offset and the second depth offset of each of the candidate targets in the video frames to be identified between each of the adjacent video frames to be identified.

4. The method for determining a clamped target according to claim 3, characterized in that: The determining of the effective horizontal offset and the effective depth offset of each of the adjacent video frames to be identified based on the first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified includes: Sorting each of the first horizontal offsets in a descending order or a descending order to obtain the sorted first horizontal offsets; based on the sorted first horizontal offsets, determining the first horizontal offsets sorted within a first preset threshold to a second preset threshold as the effective horizontal offsets of each of the adjacent video frames to be identified; sorting each of the first depth offsets in a descending order or a descending order to obtain the sorted first depth offsets; based on the sorted first depth offsets, determining the first depth offsets sorted within a third preset threshold to a fourth preset threshold as the effective depth offsets of each of the adjacent video frames to be identified; Or, the first horizontal offsets between the image blocks of the adjacent video frames to be identified are eliminated, and the first horizontal offsets that are smaller than a preset horizontal offset threshold are obtained to obtain the preliminary screening effective horizontal offsets between the image blocks of the adjacent video frames to be identified; the average value of the preliminary screening effective horizontal offsets between the image blocks of the adjacent video frames to be identified is calculated to obtain the horizontal level offset; the first horizontal offsets that are smaller than the horizontal level offset in the preliminary screening effective horizontal offsets are determined as the effective horizontal offsets of the adjacent video frames to be identified; Eliminate the first depth offsets between the image blocks of each of the adjacent video frames to be identified, and those that are less than a preset depth offset threshold, to obtain a preliminary screening effective depth offset between the image blocks of each of the adjacent video frames to be identified; calculate the average value of the preliminary screening effective depth offsets between the image blocks of each of the adjacent video frames to be identified, to obtain an average depth offset; determine the first depth offsets in the preliminary screening effective depth offsets that are less than the average depth offset as the effective depth offset of each of the adjacent video frames to be identified.

5. The method for determining a clamped target according to claim 2, characterized in that: The step of determining the clamped target from the candidate targets in the video frames to be identified based on the second horizontal offset and the second depth offset of each candidate target in the video frames to be identified between each adjacent video frames to be identified, comprises: Determine, based on the second horizontal offset and the second depth offset of each of the candidate targets in the video frame to be identified between each of the adjacent video frames to be identified, an average value of the offsets of each of the candidate targets in each direction between each of the adjacent video frames to be identified; The candidate target having the largest average value of the offsets in each direction between the adjacent video frames to be identified is determined as the clamped target.

6. The method for determining a clamped object according to any one of claims 1 to 5, characterized in that: The step of obtaining each video frame to be identified captured by the monocular endoscope comprises: Acquire a video stream captured by the monocular endoscope; The video stream captured by the monocular endoscope is frame-extracted at a preset frequency to obtain each of the video frames to be identified captured by the monocular endoscope.

7. The method for determining a clamped object according to any one of claims 1 to 5, characterized in that: Before determining the target to be clamped from the candidate targets in the video frame to be identified, the method further includes: Performing target recognition on each of the video frames to be recognized, and obtaining the clamps and each target to be clamped in each of the video frames to be recognized; Based on the intersection-and-union ratio between the clamp and each target to be clamped in each of the video frames to be identified, each of the candidate targets in the video frames to be identified is obtained.

8. The method for determining a clamped object according to any one of claims 1 to 5, characterized in that: Before dividing each of the to-be-recognized video frames into a preset size to obtain each image block of each of the to-be-recognized video frames, the method further includes: Each of the to-be-identified video frames is scaled or expanded to a preset second size; the preset second size is larger than the preset first size.

9. A device for determining a clamped target, used in a monocular endoscope system, characterized in that: The device comprises: An acquisition module, used for acquiring each video frame to be identified captured by the monocular endoscope; A segmentation module, used for segmenting each of the to-be-identified video frames according to a preset first size to obtain image blocks of each of the to-be-identified video frames; A horizontal offset determination module, used to determine a first horizontal offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified by using an optical flow method; A depth offset determination module, used to determine a first depth offset between each of the image blocks located at the same position in each of the adjacent video frames to be identified by using a monocular depth estimation method; And a target determination module is used to determine the clamped target from the candidate targets in the video frames to be identified based on the first horizontal offset between the image blocks located at the same position in each of the adjacent video frames to be identified and the first depth offset between the image blocks located at the same position in each of the adjacent video frames to be identified.

10. A monocular endoscope system, characterized in that: The monocular endoscope system includes a monocular lens and an operation recognition system, wherein the monocular lens is used to capture a video stream, and the operation recognition system is used to execute the steps of the method for determining a clamped target described in any one of claims 1 to 8.

Images