Laparoscope tracking control method and system based on image processing

Abstract

The invention provides a laparoscope tracking control method and system based on image processing, and relates to the technical field of image processing, and the method comprises the steps: obtaining a key image frame sequence of a laparoscope video stream, and carrying out the aberration correction of each key image frame; applying a feature extraction algorithm to determine feature points of the key image frames after aberration correction, and obtaining a first feature map based on the plurality of feature points of each key image frame; inputting the first feature map into a convolutional neural network for enhancement operation to form a second feature map; obtaining a movement route of the laparoscope based on the plurality of second feature maps; acquiring a corresponding coordinate position of the second feature map of each key image frame on the moving route of the laparoscope, comparing the second feature map of each key image frame with a plurality of target template feature maps at the corresponding coordinate position, and outputting a target coordinate position of the most conforming target template feature map, and controlling the laparoscope to be positioned at the target coordinate position.

Description

Technical Field

[0001] The present invention provides a laparoscopic tracking control method and system based on image processing, which relates to the technical field of image processing. Background Art

[0002] Laparoscopic surgery is representative of minimally invasive surgery and has been widely used clinically due to its significant advantages such as small trauma, fast patient recovery, and few postoperative complications. However, in laparoscopic surgery, a holding hand is required to provide corresponding lighting and the best visual feedback. However, long-term surgery makes the holding hand prone to fatigue, hand tremors, distraction, etc., directly leading to a decrease in surgical efficiency. Moreover, the intraperitoneal environment is complex and changeable during the operation, and it is difficult to accurately position the unmarked surgical instruments in the laparoscopic field of view.

[0003] In laparoscopic surgery, the precise control of the holding arm is crucial for the stability of the surgical field of view and surgical efficiency. By real-time monitoring the pixel distance and comparing it with a preset threshold, the control system of the surgical robot can dynamically adjust the deflection angle of the holding arm to adapt to the changes during the operation. This automated adjustment mechanism not only reduces the burden on the surgical team but also improves the safety and accuracy of the surgery. In addition, some advanced surgical robot systems, such as the da Vinci surgical system, have been able to achieve a higher level of automated control, including the movement of the holding arm based on visual tracking technology. These systems use artificial intelligence algorithms, such as Gaussian mixture models, to predict the position and movement route of the surgical instruments, thereby achieving more precise and rapid control of the laparoscopic field of view. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a laparoscopic tracking control method based on image processing, including the following steps:

[0005] S1. Obtain the key image frame sequence of the laparoscopic video stream, and perform aberration correction on each key image frame;

[0006] S2. Apply a feature extraction algorithm to determine the feature points of the key image frame after aberration correction, and obtain the first feature map based on multiple feature points of each key image frame;

[0007] S3. Input the first feature map into a convolutional neural network for enhancement operation to form a second feature map;

[0008] S4. Obtain multiple second feature maps corresponding to consecutive key image frames according to step S3, and obtain the movement route of the laparoscope based on the multiple second feature maps;

[0009] S5. Obtain the coordinate position corresponding to the second feature map of each key image frame on the moving route of the laparoscope, compare the second feature map of each key image frame with multiple target template feature maps at the corresponding coordinate position, output the target coordinate position of the most consistent target template feature map, and control the laparoscope to be positioned at the target coordinate position.

[0010] In a preferred embodiment, in step S5, a plurality of enhanced feature points in the second feature map of each key image frame are used as a sample set to be tested G = [g1, g2 ..., g i …, g n ],g i For the i-th enhanced feature point, construct a similarity function S:

[0011] in, is the filter coefficient, the target feature point set M of the target template feature map is [m1, m2…, m i …, m n ], the i-th target feature point of the target template feature map is m i , the total number of target feature points and the total number of enhanced feature points are equal, both are n,

[0012] When the output value of the similarity function is greater than the similarity threshold, the target template feature map is output, and the target coordinate position of the target template feature map on the moving route of the laparoscope is obtained.

[0013] In a preferred embodiment, in step S1, the coordinate transformation expression for aberration correction is as follows:

[0014]

[0015]

[0016] Where R is the lens radius of the laparoscope's attached camera, u and v are the position coordinates of the pixel points on the image, x and y are the coordinates of points u and v after aberration correction, k1, k2, and k3 are the axial aberration coefficients of the laparoscope's attached camera, and p1 and p2 are the circumferential aberration coefficients of the laparoscope's attached camera.

[0017] In a preferred embodiment, in step S2, the gradient I of the gray value of each pixel in the horizontal and vertical directions is calculated. x and I y , use the filter function to generate the matrix M:

[0018]

[0019] Calculate the pixel filtering function E based on the matrix M:

[0020] ;

[0021] where x and y represent the coordinates of the pixel point in the horizontal and vertical directions respectively, represents the filtering function, D(M) represents the determinant of matrix M, and T(M) represents the trace of matrix M, represents the empirical value;

[0022] Select the pixel points whose output range of the screening function E is higher than the screening threshold as feature points.

[0023] In a preferred embodiment, in the step S3, the first feature map is subjected to a convolution operation through the convolution layer of the convolutional neural network to achieve a linear transformation between the feature channels; all negative values in the gray value h of the feature points of the first feature map are set to 0 through the maximum function max(0, h), and the positive values remain unchanged. The formula is:

[0024] F = max(0, h);

[0025] F is the gray value after gray scale transformation through the maximum function.

[0026] In a preferred embodiment, in the step S4, multiple second feature maps corresponding to consecutive key image frames are obtained, the central feature points of the second feature map of each key image frame are acquired, the moving distances between the central feature points of the second feature maps of consecutive key image frames are calculated, and the moving distances of the central feature points of the second feature maps of all key image frames are connected to obtain the movement trajectory of the laparoscope.

[0027] The present invention also proposes a laparoscope tracking control system based on image processing for implementing the above-mentioned laparoscope tracking control method based on image processing. The laparoscope tracking control system is mounted on a laparoscope system, and the laparoscope system includes a laparoscope and an externally connected attached camera; the laparoscope tracking control system includes a data acquisition unit, a data processor, and a control unit;

[0028] The data acquisition unit acquires the video stream data of the abdominal cavity environment captured by the attached camera and feeds back the video stream data to the data processor in real time;

[0029] The data processor performs image analysis and data processing on the video stream data;

[0030] The control unit controls the laparoscope to be positioned at the target coordinate position according to the results of image analysis and data processing.

[0031] In a preferred embodiment, the data processor includes: an image frame extraction and processing unit, a feature extraction unit, a convolutional neural network processing unit, a movement trajectory formation unit, a coordinate position marking unit, and a target determination unit;

[0032] The image frame extraction and processing unit is used to obtain the key image frame sequence of the laparoscopic video stream and perform aberration correction on each key image frame;

[0033] The feature extraction unit is used to apply a feature extraction algorithm to determine the feature points of the key image frame after aberration correction and obtain a first feature map based on multiple feature points;

[0034] The convolutional neural network processing unit is used to input the first feature map into a convolutional neural network for enhancement operations to form a second feature map;

[0035] The motion trajectory formation unit is used to form the movement route of the laparoscope based on multiple second feature maps corresponding to consecutive key image frames output by the convolutional neural network processing unit;

[0036] The coordinate position marking unit is used to obtain the coordinate positions corresponding to the second feature map of each key image frame on the movement route of the laparoscope;

[0037] The target determination unit is used to compare the second feature map of each key image frame with multiple target template feature maps at the corresponding coordinate positions, output the target coordinate position where the most conforming target template feature map is located, and control the laparoscope to be positioned at this target coordinate position.

[0038] In a preferred embodiment, the motion trajectory formation unit includes: a central feature point extraction module and a movement distance connection unit;

[0039] The central feature point extraction module is used to obtain the central feature points of the second feature map of each key image frame;

[0040] The movement distance connection unit is used to calculate the movement distances between the central feature points of the second feature maps of consecutive key image frames, connect the movement distances of the central feature points of the second feature maps of all key image frames to form a path, and obtain the movement route of the laparoscope.

[0041] Compared with the prior art, the present invention has the following beneficial technical effects:

[0042] The present invention proposes a laparoscopic tracking control method based on image processing, which obtains a key image frame sequence of a laparoscopic video stream, performs aberration correction on each key image frame. By performing aberration correction on the key image frames, the geometric distortion of the image can be significantly improved, making the laparoscopic field of view clearer; applying a feature extraction algorithm to obtain a first feature map of each key image frame; inputting the first feature map into a convolutional neural network for enhancement operation to form a second feature map; obtaining multiple second feature maps of consecutive key image frames, and obtaining the movement route of the laparoscope based on the multiple second feature maps, so as to better determine the dynamic position and movement of the laparoscope in the abdominal cavity area; obtaining the coordinate position of the central feature point of the second feature map of each key image frame on the movement route of the laparoscope, comparing the second feature map of each key image frame with the target template feature map, outputting the target coordinate position, and controlling the laparoscope to be positioned at the target coordinate position. Through feature map matching, the target position can be accurately identified, enabling the laparoscope to accurately locate at the specific coordinates of the area to be processed, improving the laparoscopic control accuracy, reducing human errors at the same time, and reducing the errors caused by improper human operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic flowchart of the laparoscopic tracking control method based on image processing of the present invention;

[0045] Figure 2 It is a schematic diagram of aberration correction of key image frames of the present invention;

[0046] Figure 3 It is a schematic diagram of the comparison between the actual movement route and the expected trajectory of the laparoscope in the embodiment of the present invention;

[0047] Figure 4 It is a schematic structural diagram of the laparoscopic tracking control method system based on image processing of the present invention;

[0048] Figure 5 It is a schematic diagram of the output target template feature map. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0050] In the accompanying drawings of the specific embodiments of the present invention, to better and more clearly describe the working principles of the components in the system and show the connection relationships of the various parts of the device, only the relative positional relationships between the components are clearly distinguished, which does not constitute a limitation on the signal transmission direction, connection sequence, and the sizes, sizes, and shapes of the various parts of the structure within the components or the structure.

[0051] Embodiment 1

[0052] The present invention mounts a laparoscopic tracking control system on a laparoscopic system. In the laparoscopic system, a camera is externally attached to the laparoscope to capture the abdominal cavity environment, and the video data is fed back to the laparoscopic tracking control system in real time. Through image analysis and data processing results, the laparoscope can accurately obtain the coordinate position of the target area to be processed and control the laparoscope to be positioned at the target coordinate position.

[0053] As Figure 1 shown, it is a schematic flow chart of the laparoscopic tracking control method based on image processing of the invention. The laparoscopic tracking control method based on image processing includes the following steps:

[0054] S1. Obtain a key image frame sequence of the laparoscopic video stream and perform aberration correction on each key image frame.

[0055] Video stream data of the abdominal cavity environment is obtained through the laparoscope. To convert the video stream data into static images, the video stream is converted into an RGB image frame sequence through a video processing tool, and the key image frame sequence of the video stream is extracted for subsequent feature extraction.

[0056] The aberration coefficients of the attached camera are crucial for ensuring the accuracy and quality of the images. The axial aberration coefficients k1, k2, k3 and the circumferential aberration coefficients p1, p2 jointly describe the aberration characteristics of the attached camera and are used for aberration correction of the images. Therefore, in this embodiment, after the key image frame sequence of the video stream is extracted, aberration correction is performed on each key image frame through coordinate transformation.

[0057] Specifically, the coordinate transformation expression for aberration correction is as follows:

[0058]

[0059]

[0060] In the formula, R is the lens radius of the attached camera of the laparoscope, u and v are the position coordinates of the pixel points on the image, and x and y are the coordinates of the points u and v after aberration correction respectively.

[0061] Among them, k1, k2, and k3 are the axial aberration coefficients of the attached camera of the laparoscope:

[0062] k1 is the main axial aberration coefficient, which is related to the convex aberration and concave aberration of the attached camera. A positive value indicates convex aberration, while a negative value indicates concave aberration.

[0063] k2 affects the secondary axial aberration and is used to further correct the image aberration and improve the accuracy of correction.

[0064] k3 affects the tertiary axial aberration and is used to handle the aberration situation at the edge of the field of view.

[0065] Among them, p1 and p2 are the circumferential aberration coefficients of the attached camera of the laparoscope, and the circumferential aberration is mainly manifested as the tilt or distortion of the image.

[0066] Such as Figure 2 As shown, by performing aberration correction on the key image frames, the geometric distortion of the image can be significantly improved, making the laparoscope field of view clearer.

[0067] S2. Apply the feature extraction algorithm to determine the feature points of the key image frames after aberration correction, and obtain the first feature map based on the multiple feature points of each key image frame.

[0068] Specifically, the feature extraction algorithm is applied to the pixel points of the key image frames after aberration correction to determine the feature points. For each pixel point, calculate the gradients I x and I y of the gray value of this pixel point in the horizontal and vertical directions. The calculation of the gradient helps to understand the change of the gray value in the image, so as to identify the important feature points in the image.

[0069] Based on the gradients I x and I y of the gray value in the horizontal and vertical directions, use the filtering function to generate the matrix M:

[0070]

[0071] Calculate the pixel point screening function E based on the matrix M:

[0072]

[0073] Among them, x and y respectively represent the coordinates of the pixel point in the horizontal and vertical directions, I x and Iy respectively represent the gradients of the gray value of the pixel point in the horizontal and vertical directions represents the filtering function, D(M) represents the determinant of matrix M, and T(M) represents the trace of matrix M represents the empirical value

[0074] For the output range of the screening function E for all pixel points, select the pixel points with the output range higher than the screening threshold as feature points

[0075] Among them, this screening threshold can be fixed or dynamically calculated based on the content of the key image frame

[0076] Finally, for each key image frame, multiple feature points are output, and these multiple feature points constitute the first feature map corresponding to the key image frame

[0077] Specifically, once multiple feature points of each key image frame are detected, a descriptor needs to be generated for each feature point. The descriptor is a quantization representation of the neighborhood around the feature point, which is invariant to image rotation, scaling, and partial brightness changes, and helps to match corresponding pixel points between the original image and the feature map

[0078] Each feature point is represented by its position (x, y coordinates) and attributes (such as scale, direction) in the image. Integrate all the detected feature points and their descriptors to form the first feature map of the key image frame

[0079] S3. Input the first feature map into a convolutional neural network for enhancement operations to form a second feature map

[0080] First, perform a convolution operation on the first feature map through the convolution layer of the convolutional neural network to achieve a linear transformation between feature channels

[0081] The convolution layer performs a linear transformation between feature channels on the input first feature map through the convolution operation. During this convolution operation process, the 1x1 convolution kernel slides on the first feature map and calculates the weighted sum of the sliding area

[0082] Among them, the size and number of the convolution kernels determine the number of feature channels of the output second feature map. By designing the number of convolution kernels, the number of feature channels of the first feature map can be reduced, which helps to reduce the computational amount and prevent overfitting

[0083] For example: the size of the input first feature map is H×W×D in (height, width, number of feature points), and the number of feature channels of the output can be reduced by using a convolution kernel of K×K×D in and D out convolution kernels, where K is the size of the convolution kernel, Dout is the number of feature channels of the output.

[0084] Secondly, all negative values in the gray value h of the feature points of the first feature map are set to 0 through the maximum function max(0, h), and positive values remain unchanged. The formula is:

[0085] F = max(0, h).

[0086] F is the gray value after gray-scale transformation through the maximum function. Finally, multiple transformed gray values form the second feature map.

[0087] S4. Obtain multiple second feature maps corresponding to consecutive key image frames according to step S3, and obtain the movement route of the laparoscope based on the multiple second feature maps. According to the process of step S3, obtain multiple consecutive second feature maps corresponding to consecutive key image frames, and obtain the central feature points of the second feature map of each key image frame.

[0088] The central feature points of the second feature map can be determined by various methods. The centroid of the key image frame, the brightest point in the predefined region of interest, or feature matching algorithms can be used for localization.

[0089] In an exemplary embodiment, the centroid of the key image frame is used to determine the central feature point of the second feature map.

[0090] The centroid is the weighted average position of all feature points in the key image frame, representing the geometric center of the key image frame. In the key image frame, the centroid is calculated through the following steps:

[0091] For each feature point of the key image frame, multiply the gray value of the feature point by the position of the feature point, sum all feature points to obtain the weighted abscissa and ordinate, and divide the weighted coordinates by the total gray value of the key image frame to obtain the position of the centroid.

[0092] Secondly, calculate the movement distance between the central feature points of the second feature maps of consecutive key image frames. This can be achieved by calculating the Euclidean distance or Manhattan distance between the central feature points. Connect the movement distances of the central feature points of the second feature maps of all key image frames to form a path, and the movement route of the laparoscope can be obtained.

[0093] In a preferred embodiment, the calculated movement route is visualized through a graphical interface so that the movement of the laparoscope can be intuitively seen. At the same time, by comparing the actual movement route with the expected trajectory, the effect of laparoscope control can be better evaluated. As Figure 3 shown, the actual movement route is very consistent with the expected trajectory.

[0094] S5. Obtain the coordinate positions corresponding to the second feature maps of each key image frame on the movement route of the laparoscope. Compare the second feature map of each key image frame with multiple target template feature maps at the corresponding coordinate positions, output the target coordinate position where the most conforming target template feature map is located, and control the laparoscope to be positioned at this target coordinate position.

[0095] Multiple enhanced feature points in the second feature map of each key image frame are used as the test sample set G = [g1, g2…, g i …, g n , and a similarity function S is constructed:

[0096] Among them, is the filtering coefficient, and the set of target feature points of the target template feature map M = [m1, m2…, m i …, m n , n is the total number of feature points, and the i-th target feature point of the target template feature map is m i .

[0097] When the output value of the similarity function is greater than the similarity threshold, output this target template feature map, and obtain the target coordinate position of this target template feature map on the movement route of the laparoscope. It should be emphasized that the multiple target template feature maps at the corresponding coordinate positions are relatively similar to each other. Therefore, if the similarity threshold is set relatively low, more than one target template feature map will be output. As Figure 5 shown, it is a schematic diagram of the output target template feature map.

[0098] Preferably, the similarity threshold can be set between 70% - 100%, such as 95%, 90%, 85%, 80%, 75%. As shown in Table 1, it is the corresponding relationship between the setting of the similarity threshold and the number of target template feature maps during the actual operation process.

[0099] Table 1 Similarity Threshold Setting Table

[0100]

[0101] Obtain the coordinate positions corresponding to the second feature map of the output key image frame on the movement route of the laparoscope, so as to obtain the target coordinate position, that is, the lesion position. If the output values of the similarity function are all not greater than the similarity threshold, it means that there is no lesion position.

[0102] Example 2

[0103] On the basis of Embodiment 1, step S5 further records the target coordinate position, and then selects video segments of a preset length forward and backward respectively at the target coordinate position, and finally outputs the time information of the video segment in the entire video stream and the feature information of the key image frames, which is convenient for directly finding the key video segment in the entire video stream.

[0104] The specific method of selecting video segments of a preset length forward and backward respectively at the target coordinate position is as follows: the associated target coordinate position of the current frame is calculated through the target coordinate position of the previous frame, and the associated target coordinate position Z of the current frame t can be expressed as:

[0105]

[0106] where Z t-1 represents the target coordinate position of the (t - 1)-th frame, Z t-2 represents the target coordinate position of the (t - 2)-th frame, Z t-3 represents the target coordinate position of the (t - 3)-th frame, Z t-1 and Z t-2 and Z t-3 are all obtained by output of the similarity function S in step S5 of Embodiment 1, represents the first adjustable parameter, represents the second adjustable parameter, both are decimals greater than 0 and less than 1.

[0107] If the associated target coordinate position of the current image frame is calculated from the previous image frame, the overall operation efficiency is greatly improved. At the same time, the target position is kept consistent with the previous image frame, and a better visual effect can be obtained.

[0108] Embodiment 3

[0109] The present invention also proposes a laparoscopic tracking control system based on image processing. As Figure 4 shown in the schematic diagram of the laparoscopic tracking control system based on image processing, the laparoscopic tracking control system is carried on the laparoscopic system. The laparoscopic system includes a laparoscope and an externally connected attached camera, and the attached camera is used to capture the video of the abdominal cavity environment.

[0110] The laparoscopic tracking control system includes a data acquisition unit, a data processor, and a control unit.

[0111] Among them, the data acquisition unit acquires the video stream data of the abdominal cavity environment captured by the attached camera and feeds back the video stream data to the data processor of the laparoscopic tracking control system in real time.

[0112] The data processor performs image analysis and data processing on the video stream data, and the control unit controls the laparoscope to be positioned at the target coordinate position, that is, the position of the existing lesion, according to the results of the image analysis and data processing.

[0113] In a preferred embodiment, the data processor specifically includes an image frame extraction and processing unit, a feature extraction unit, a convolutional neural network processing unit, a movement route formation unit, a coordinate position marking unit, and a target determination unit.

[0114] The image frame extraction and processing unit is used to obtain a sequence of key image frames of the laparoscope video stream and perform aberration correction on each key image frame for subsequent feature extraction.

[0115] The feature extraction unit is used to apply a feature extraction algorithm to determine the feature points of the key image frames after aberration correction, and obtain a first feature map based on the multiple feature points of each key image frame.

[0116] The convolutional neural network processing unit is used to input the first feature map into a convolutional neural network for enhancement operations to form a second feature map.

[0117] The movement route formation unit is used to output multiple second feature maps corresponding to consecutive key image frames through the convolutional neural network processing unit, and obtain the movement route of the laparoscope based on the multiple second feature maps.

[0118] In a preferred embodiment, the movement route formation unit includes: a central feature point extraction module and a movement distance connection unit.

[0119] The central feature point extraction module is used to obtain the central feature points of the second feature maps of each key image frame using the centroid of the key image frame, the brightest point in the predefined region of interest, or through a feature matching algorithm.

[0120] The movement distance connection unit is used to calculate the movement distances between the central feature points of the second feature maps of consecutive key image frames, connect the movement distances of the central feature points of the second feature maps of all key image frames to form a path, and obtain the movement route of the laparoscope.

[0121] In a preferred embodiment, the movement route forming unit further includes a display module. The display module visualizes the calculated movement route through a graphical interface so that the movement of the laparoscope can be intuitively seen. At the same time, by comparing the actual movement route with the expected trajectory, the effect of laparoscope control can be better evaluated. The display module is an interaction bridge between the doctor and the system, which allows the doctor to monitor the movement process of the laparoscope in real time and adjust the control parameters and issue control instructions as needed. The display module also includes a variety of input and output devices such as a display screen, a touch screen, a handle, and a foot pedal to provide an intuitive and convenient operation experience. The coordinate position marking unit is used to obtain the coordinate positions corresponding to the second feature map of each key image frame on the movement route of the laparoscope.

[0122] The target determination unit is used to compare the second feature map of each key image frame with multiple target template feature maps at the corresponding coordinate positions, output the target coordinate position where the most matching target template feature map is located, and control the laparoscope to be positioned at this target coordinate position.

[0123] The laparoscope tracking control system based on image processing of the present invention integrates multiple technical means such as machine vision, image processing, and automatic control to achieve precise tracking and control of the laparoscope; especially through feature map matching, the target position can be accurately identified, enabling the laparoscope to accurately locate to the specific coordinates of the area to be processed, improving the accuracy of laparoscope control, reducing human errors at the same time, and reducing the errors caused by improper human operation.

[0124] Table 2 shows the summary of the image analysis and data processing process of the data processor for video stream data, and the calculation time performance of 5 judgment criteria (C1 - C5). The test environment is a computer equipped with a Linux system, an Intel i9 - 10900K CPU, and an Nvidia RTX 2080Ti GPU.

[0125] Average: Average calculation time (unit: microsecond, μs).

[0126] σ (standard deviation): Reflects the degree of fluctuation of the calculation time. The larger the value, the more dispersed the data.

[0127] Min / Max: Minimum and maximum calculation times, reflecting the boundary range of the calculation time-consuming of this standard.

[0128] Table 2 Data Processor Processing Parameters

[0129]

[0130] Through data comparison, it can be seen that C1 and C2 have high computing efficiency and good stability, and are suitable for scenarios with high real-time requirements; the computing time and fluctuations of C3 and C4 increase significantly; C5 has the highest computing complexity, and the time consumption is significantly higher than other standards, which may involve more complex operation logics.

[0131] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0132] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0133] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device implements the steps in the above method embodiments.

[0134] Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0135] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0136] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A laparoscopic tracking control method based on image processing, characterized in that, It includes the following steps: S1. Obtain the key image frame sequence of the laparoscopic video stream, and perform aberration correction on each key image frame; S2. Apply a feature extraction algorithm to determine the feature points of the key image frames after aberration correction, and obtain a first feature map based on multiple feature points of each key image frame; S3. Input the first feature map into a convolutional neural network for enhancement operation to form a second feature map; S4. Obtain multiple second feature maps corresponding to consecutive key image frames according to step S3, and obtain the movement route of the laparoscope based on the multiple second feature maps; S5. Obtain the coordinate positions corresponding to the second feature map of each key image frame on the movement route of the laparoscope, compare the second feature map of each key image frame with multiple target template feature maps at the corresponding coordinate positions, output the target coordinate position where the most conforming target template feature map is located, and control the laparoscope to be positioned at the target coordinate position.

2. The laparoscopic tracking control method based on image processing according to claim 1, wherein In the step S5, multiple enhanced feature points in the second feature map of each key image frame are used as a set of samples to be measured \(G = [g_1, g_2,\cdots, g i \cdots, g n \), where \(g i \) is the \(i\)-th enhanced feature point, and a similarity function \(S\) is constructed: Among them, is the filtering coefficient, and the set of target feature points M = [m1, m2…, m i …, m n of the target template feature map, and the i-th target feature point of the target template feature map is m i . The total number of target feature points is equal to the total number of enhanced feature points, both of which are n. When the output value of the similarity function is greater than the similarity threshold, output the target template feature map, and obtain the target coordinate position of the target template feature map on the movement route of the laparoscope.

3. The laparoscopic tracking control method based on image processing according to claim 1, wherein In the step S1, the coordinate transformation expression of aberration correction is as follows: In the formula, R is the lens radius of the attached camera of the laparoscope, u and v are the position coordinates of the pixel points on the image, x and y are the coordinates after aberration correction of the u and v points respectively, k1, k2, and k3 are the axial aberration coefficients of the attached camera of the laparoscope, and p1 and p2 are the circumferential aberration coefficients of the attached camera of the laparoscope.

4. The laparoscopic tracking control method based on image processing according to claim 3, wherein In the step S2, calculate the gradients I x and I y of the gray value of each pixel point in the horizontal and vertical directions, and generate a matrix M by using a filtering function: Calculate the pixel point screening function E based on the matrix M: ； where x and y respectively represent the coordinates of the pixel point in the horizontal and vertical directions, represents the filtering function, D(M) represents the determinant of matrix M, and T(M) represents the trace of matrix M, represents the empirical value; Select the pixel points whose output range of the screening function E is higher than the screening threshold as feature points.

5. The laparoscopic tracking control method based on image processing according to claim 1, characterized in that, In the step S3, perform a convolution operation on the first feature map through the convolutional layer of the convolutional neural network to achieve a linear transformation between feature channels; set all negative values in the gray value h of the feature points of the first feature map to 0 through the maximum value function max(0, h), and keep the positive values unchanged. The formula is: F = max(0, h); F is the gray value after gray level transformation through the maximum value function.

6. The laparoscopic tracking control method based on image processing according to claim 1, wherein In the step S4, obtain multiple second feature maps corresponding to consecutive key image frames, obtain the central feature points of the second feature map of each key image frame, calculate the movement distance between the central feature points of the second feature map of consecutive key image frames, and connect the movement distances of the central feature points of the second feature map of all key image frames to obtain the movement trajectory of the laparoscope.

7. A laparoscopic tracking control system based on image processing, characterized in that For implementing the laparoscopic tracking control method based on image processing according to any one of claims 1-6, the laparoscopic tracking control system is mounted on a laparoscopic system, and the laparoscopic system includes a laparoscope and an externally connected attached camera; the laparoscopic tracking control system includes a data acquisition unit, a data processor, and a control unit; The data acquisition unit acquires the video stream data of the abdominal cavity environment captured by the attached camera, and feeds back the video stream data to the data processor in real time; The data processor performs image analysis and data processing on the video stream data; The control unit controls the laparoscope to be positioned at the target coordinate position according to the results of image analysis and data processing.

8. The laparoscopic tracking control system based on image processing according to claim 7, characterized in that The data processor includes: an image frame extraction and processing unit, a feature extraction unit, a convolutional neural network processing unit, a motion trajectory formation unit, a coordinate position marking unit, and a target determination unit; The image frame extraction and processing unit is used to obtain a key image frame sequence of the laparoscopic video stream and perform aberration correction on each key image frame; The feature extraction unit is used to apply a feature extraction algorithm to determine the feature points of the key image frame after aberration correction, and obtain a first feature map based on multiple feature points; The convolutional neural network processing unit is used to input the first feature map into a convolutional neural network for enhancement operations to form a second feature map; The motion trajectory formation unit is used to form the movement route of the laparoscope based on multiple second feature maps corresponding to consecutive key image frames output by the convolutional neural network processing unit; The coordinate position marking unit is used to obtain the coordinate positions corresponding to the second feature map of each key image frame on the movement route of the laparoscope; The target determination unit is used to compare the second feature map of each key image frame with multiple target template feature maps at the corresponding coordinate positions, output the target coordinate position where the most matching target template feature map is located, and control the laparoscope to be positioned at the target coordinate position.

9. The laparoscopic tracking control system based on image processing according to claim 8, characterized in that, The motion trajectory formation unit includes: a central feature point extraction module and a movement distance connection unit; The central feature point extraction module is used to obtain the central feature points of the second feature map of each key image frame; The movement distance connection unit is used to calculate the movement distances between the central feature points of the second feature maps of consecutive key image frames, connect the movement distances of the central feature points of the second feature maps of all key image frames to form a path, and obtain the movement route of the laparoscope.

Images