A method, apparatus, electronic device, and storage medium for tracking key feature points.

Abstract

The present application provides a key feature point tracking method, device, electronic device and storage medium, which are applied to the field of image processing technology. The method effectively solves the problem of inaccurate depth information caused by the sparsity of point clouds in endoscopic surgery scenes by judging in real time whether the starting feature point falls into a sparse point cloud and automatically replacing it with a feature point in a dense point cloud when necessary. This dynamic adjustment mechanism ensures the continuity and accuracy of feature point tracking and provides reliable technical support for accurate surgical path planning.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a key feature point tracking method, device, electronic device and storage medium. Background Art

[0002] A 3D point cloud is a data structure used to represent objects or scenes in a 3D space. In mathematical terms, it is a collection of multiple 3D coordinate points (X, Y, Z). An important feature of a point cloud is that it has depth information, which refers to the distance information from each point on the surface of the object to the camera. Through depth information, doctors can more intuitively perceive the 3D spatial structure within the surgical area, thereby more accurately locating lesions and surgical sites, reducing surgical errors caused by inaccurate positioning; and more accurately determine the entry and exit paths of surgical instruments, including the location and size of incisions, to avoid damage to surrounding tissues, which is crucial for minimally invasive surgery.

[0003] When planning the path of surgical instruments, the starting point needs to be manually selected in advance and the features of the starting point need to be tracked. However, the above method has the problem of inaccurate distance measurement due to the sparse point cloud. The sparsity of the point cloud means that during the sampling process of the sensor, due to various factors, the point cloud data obtained is unevenly distributed in the three-dimensional space, and the density of some point cloud data points is too low relative to the entire scene. This will cause the depth information of the local area to change dramatically, resulting in inaccurate depth information. In order to obtain accurate distance, it is necessary to ensure that the target point and the tracking process are not located in the sparse point cloud.

[0004] For sparse point clouds, most methods choose to expand the sampling range, combine multiple sensors or use various filtering algorithms to improve the accuracy of depth information. However, surgical scenes are obviously different from natural scenes: 1) The endoscopic camera has a limited field of view in the abdominal cavity and a limited sampling range. 2) The abdominal cavity space is limited and cannot accommodate multiple sensors at the same time. In addition, equipment entering the human body usually has strict regulations that most sensors cannot meet. 3) In surgical scenes, the endoscopic images sent back need to restore the real scene as much as possible. Filtering operations such as median filtering may be beneficial to depth information, but they will ignore the detailed information in the scene, which is crucial for surgery.

[0005] Therefore, considering the above problems, the prior art lacks a method that can automatically determine whether the current selected point is located in a sparse point cloud and ensure that the target point does not fall into the sparse area during tracking. Summary of the invention

[0006] In view of the above-mentioned shortcomings of the prior art, the present application provides a key feature point tracking method, device, electronic device and storage medium, which are applied to the field of image processing technology and have the advantages of automatically determining the starting feature point position, dynamically avoiding sparse point cloud areas, and ensuring tracking accuracy.

[0007] In a first aspect, a key feature point tracking method is provided, the method comprising the steps of:

[0008] S1: Acquire a two-dimensional image to be processed, and perform three-dimensional reconstruction on the two-dimensional image to be processed to obtain a three-dimensional reconstruction result;

[0009] S2: arbitrarily selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to a sparse point cloud of the three-dimensional reconstruction result;

[0010] S3: When the starting feature point is not in the sparse point cloud, tracking the change position of the starting feature point, and determining in real time whether the starting feature point falls into the sparse point cloud;

[0011] S4: During the tracking process, if the starting feature point falls into the sparse point cloud, a pixel feature point in the dense point cloud closest to the real-time position of the starting feature point is obtained, and the starting feature point is replaced by the pixel feature point for subsequent calculations.

[0012] A key feature point tracking method proposed in this application obtains a three-dimensional reconstruction result by performing three-dimensional reconstruction on the acquired two-dimensional image to be processed. The three-dimensional reconstruction result can provide more accurate spatial information, which helps to improve the accuracy and robustness of feature point tracking. By judging whether the starting feature point is in the sparse point cloud, it is ensured that the tracked starting feature point has depth information. Real-time tracking can ensure that the position information of the starting feature point is the latest, which helps to improve the real-time and accuracy of tracking. At the same time, by judging whether the starting feature point enters the sparse point cloud, the tracking strategy can be dynamically adjusted, and the sparse point cloud can be replaced by the dense point cloud for subsequent calculations. This method effectively solves the problem of inaccurate depth information caused by the sparsity of the point cloud in endoscopic surgery scenes by judging in real time whether the starting feature point falls into the sparse point cloud, and automatically replacing it with the feature point in the dense point cloud when necessary. This dynamic adjustment mechanism ensures the continuity and accuracy of feature point tracking and provides reliable technical support for accurate surgical path planning.

[0013] Further, step S4 includes:

[0014] S41: During the tracking process, if the starting feature point falls into the sparse point cloud, then calculating the XYZ channel variance of the local area centered on the real-time position of the starting feature point;

[0015] S42: setting the pixel of the channel with the largest variance in the XYZ channel to 0, and calculating the n point cloud feature points closest to the starting feature point in the remaining two channels;

[0016] S43: Sort the n point cloud feature points in ascending order according to the point cloud distance from the camera, and calculate in turn whether the n point cloud feature points are in the dense point cloud, until the m-th point cloud feature point falling in the dense point cloud is found, the m-th point cloud feature point is the pixel feature point, and the pixel feature point is used to replace the starting feature point for subsequent calculations, wherein n is the total number of point cloud feature points in the remaining two channels that are closest to the starting feature point, and m is the number of points in the sorted point cloud feature points that fall in the dense point cloud, and m is less than or equal to n.

[0017] The present application proposes a key feature point tracking method, which calculates the XYZ channel variance of the local area of ​​the starting feature point and performs special processing on the channel with the largest variance, thereby achieving accurate positioning and replacement of the starting feature point.

[0018] Further, step S1 includes:

[0019] S11: calibrating the camera, obtaining camera parameters, and reading endoscopic surgery video frames, and using the camera parameters to perform left-right eye correction on the video frames to obtain corrected video frames, where the corrected video frames are two-dimensional images to be processed;

[0020] S12: performing stereo matching on the two-dimensional image to be processed to obtain a corresponding disparity map;

[0021] S13: Performing three-dimensional reconstruction on the two-dimensional image to be processed by using the disparity map and the camera parameters to obtain a three-dimensional reconstruction result.

[0022] This application proposes a key feature point tracking method. Camera calibration is to obtain internal and external parameters to ensure the accuracy of image correction. Video frame correction can eliminate lens distortion and improve image quality. The generation of disparity map is the key to stereo matching. It reflects the position difference of pixel points in the image under different viewing angles and provides depth information for three-dimensional reconstruction. The three-dimensional reconstruction process converts the two-dimensional image into point cloud data with spatial coordinates, laying the foundation for subsequent feature point tracking.

[0023] Further, step S2 includes:

[0024] S21: Calculate the distance array of the entire current point cloud in the three-dimensional reconstruction result;

[0025] S22: according to the distance array, taking a local area with a radius of r from the starting feature point, and calculating the average spatial distance of the local area;

[0026] S23: Compare the average spatial distance with the preset critical distance. If the average spatial distance is greater than the preset critical distance, it is determined that the starting feature point corresponds to the sparse point cloud in the three-dimensional reconstruction result; if the average spatial distance is less than or equal to the preset critical distance, it is determined that the starting point is not in the sparse point cloud of the three-dimensional reconstruction result.

[0027] This application proposes a key feature point tracking method, which realizes accurate judgment of whether the starting feature point is located in a sparse point cloud by calculating and comparing the spatial distance of the local area. Specifically, when the average spatial distance of the local area is large, it means that the point cloud distribution in the area is sparse; conversely, when the average spatial distance is small, it means that the point cloud distribution in the area is relatively dense.

[0028] Furthermore, after step S2, the following steps are further included:

[0029] S24: When the starting feature point is in the sparse point cloud, the currently selected starting feature point is discarded, and a pixel point is randomly selected again as the starting feature point until the starting feature point is no longer in the sparse point cloud of the three-dimensional reconstruction result.

[0030] Furthermore, in step S1, the distance array of the entire current point cloud in the three-dimensional reconstruction result is calculated and sorted from small to large as follows: , is the distance array of the entire current point cloud, is the current point cloud in the 3D reconstruction result, is the point cloud closest to the starting feature point, A function that sorts the distance array from small to large.

[0031] Furthermore, in step S23, the preset critical distance is ,in, is the distance array of the entire current point cloud, is the preset critical distance, is a given threshold factor, , Is a function to get the number of elements in the distance array.

[0032] In a second aspect, a key feature point tracking device is provided, the device comprising:

[0033] 3D reconstruction module: used to obtain the 2D image to be processed, and perform 3D reconstruction on the 2D image to be processed to obtain a 3D reconstruction result;

[0034] Point selection module: used for arbitrarily selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to the sparse point cloud of the three-dimensional reconstruction result;

[0035] A judgment module: when the starting feature point is not in the sparse point cloud, tracking the change position of the starting feature point, and judging in real time whether the starting feature point falls into the sparse point cloud;

[0036] Tracking module: used for obtaining the pixel feature point in the dense point cloud closest to the real-time position of the starting feature point if the starting feature point falls into the sparse point cloud during the tracking process, and replacing the starting feature point with the pixel feature point for subsequent calculation.

[0037] In a third aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in any of the above methods are executed.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above methods are executed.

[0039] Beneficial effects: The present application provides a key feature point tracking method, device, electronic device and storage medium, which obtains a three-dimensional reconstruction result by performing three-dimensional reconstruction on the acquired two-dimensional image to be processed. The three-dimensional reconstruction result can provide more accurate spatial information, which helps to improve the accuracy and robustness of feature point tracking. By judging whether the starting feature point is in the sparse point cloud, it is ensured that the tracked starting feature point has depth information. Real-time tracking can ensure that the position information of the starting feature point is the latest, which helps to improve the real-time and accuracy of tracking. At the same time, by judging whether the starting feature point enters the sparse point cloud, the tracking strategy can be dynamically adjusted, and the sparse point cloud can be replaced by a dense point cloud for subsequent calculations. This method effectively solves the problem of inaccurate depth information caused by the sparsity of the point cloud in endoscopic surgery scenes by judging in real time whether the starting feature point falls into the sparse point cloud, and automatically replacing it with the feature point in the dense point cloud when necessary. This dynamic adjustment mechanism ensures the continuity and accuracy of feature point tracking and provides reliable technical support for accurate surgical path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a key feature point tracking method proposed in this application.

[0041] Figure 2 This is a structural diagram of a key feature point tracking device proposed in this application.

[0042] Figure 3A schematic diagram of the structure of the electronic device provided in this application.

[0043] Description of reference numerals: 201, 3D reconstruction module; 202, point acquisition module; 203, judgment module; 204, tracking module; 301, processor; 302, memory; 303, communication bus; 3, electronic device. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and marked in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0045] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first, second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0046] In the application of medical three-dimensional point cloud technology, the accuracy of depth information is a key technical difficulty. Existing methods increase the point cloud density by expanding the sampling range or combining multiple sensors, but there are obvious limitations in endoscopic surgery scenarios. The endoscopic camera has a limited field of view, a small abdominal cavity space, and the equipment is strictly regulated, so traditional methods are difficult to apply directly. In order to solve this problem, the present application proposes a key feature point tracking method, device, electronic device, and storage medium. The following disclosure provides many different implementation methods or examples to achieve the purpose of the present invention.

[0047] Please refer to Figure 1 In a first aspect, a key feature point tracking method is provided, the method comprising the steps of:

[0048] S1: Acquire a two-dimensional image to be processed, and perform three-dimensional reconstruction on the two-dimensional image to obtain a three-dimensional reconstruction result;

[0049] S2: randomly selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to the sparse point cloud of the three-dimensional reconstruction result;

[0050] S3: When the starting feature point is not in the sparse point cloud, track the change position of the starting feature point and determine in real time whether the starting feature point falls into the sparse point cloud;

[0051] S4: During the tracking process, if the starting feature point falls into the sparse point cloud, the pixel feature point in the dense point cloud closest to the real-time position of the starting feature point is obtained, and the pixel feature point is used to replace the starting feature point for subsequent calculations.

[0052] The two-dimensional image to be processed refers to the image obtained from the endoscopic surgery video frame, which can be achieved by camera calibration and left-right eye correction. Three-dimensional reconstruction refers to the three-dimensional spatial reconstruction of the two-dimensional image using the disparity map and camera parameters, which can be achieved by stereo matching algorithm.

[0053] Among them, sparse point cloud refers to the point cloud area with uneven distribution and low density in the 3D reconstruction result, which can be judged by calculating the average spatial distance of the local area. The corresponding one is dense point cloud. Dense point cloud refers to the point cloud area with uniform distribution and high density in the 3D reconstruction result, which can be judged by calculating the average spatial distance of the local area.

[0054] To obtain the two-dimensional image to be processed and perform three-dimensional reconstruction, the camera parameters can be first obtained through camera calibration, and the left and right eye correction of the endoscopic surgery video frame can be performed to obtain the corrected video frame as the two-dimensional image to be processed. Then, stereo matching is performed on the image to obtain a disparity map. Finally, the disparity map and camera parameters are used to perform three-dimensional reconstruction to obtain a three-dimensional reconstruction result.

[0055] Among them, a starting feature point is arbitrarily selected from the two-dimensional image to be processed, and it is determined whether the point corresponds to the sparse point cloud of the three-dimensional reconstruction result. The judgment method is to calculate the average spatial distance of the local area centered on the starting feature point and compare it with the preset critical distance. If the average spatial distance is greater than the preset critical distance, it is determined that the starting point is in the sparse point cloud; otherwise, it is not in the sparse point cloud. During the operation, the endoscope will move slightly with the change of the surgical position, and the surgical instruments will also move in and out under the endoscope. This complex situation makes the selected starting feature point not fixed at a certain position, and the tracking position of the starting feature point is constantly changing. Therefore, it is necessary to ensure in real time during the tracking process that the selected starting feature point does not fall into the sparse area. When the starting feature point is not in the sparse point cloud, start tracking the changing position of the point and determine in real time whether it falls into the sparse point cloud. This step is achieved by continuously monitoring the average spatial distance of the local area around the starting feature point. During the tracking process, if the starting feature point is found to fall into the sparse point cloud, the pixel feature point in the dense point cloud closest to the real-time position of the point is obtained, and the new feature point is used to replace the original starting feature point for subsequent calculations. This replacement process ensures the continuity of tracking and the accuracy of depth information, and ensures that the key point features selected by the doctor are always located in the dense point cloud. It is necessary to expand the sampling range and add sensors without adding additional costs to achieve real-time tracking of target points and automatically avoid target points falling into the sparse point cloud. It has the advantages of simplicity, low cost and high real-time.

[0056] Further, step S4 includes:

[0057] S41: During the tracking process, if the starting feature point falls into the sparse point cloud, the XYZ channel variance of the local area centered on the real-time position of the starting feature point is calculated;

[0058] S42: Set the pixel of the channel with the largest variance in the XYZ channel to 0, and calculate the n point cloud feature points closest to the starting feature point in the remaining two channels;

[0059] S43: Sort the n point cloud feature points in ascending order according to the point cloud distance from the camera, and calculate in turn whether the n point cloud feature points are in the dense point cloud, until the mth point cloud feature point that falls in the dense point cloud is found, the mth point cloud feature point is a pixel feature point, and the pixel feature point is used to replace the starting feature point for subsequent calculations, where n is the total number of point cloud feature points in the remaining two channels that are closest to the starting feature point, and m is the number of points in the sorted point cloud feature points that fall in the dense point cloud, and m is less than or equal to n.

[0060] Among them, by calculating the XYZ channel variance of the local area of ​​the starting feature point and performing special processing on the channel with the largest variance, the starting feature point can be accurately located and replaced. Among them, the local area is an area with the starting feature point as the center and a preset range as the radius. Specifically, the scheme first calculates the XYZ channel variance of the local area of ​​the real-time position of the starting feature point; secondly, the channel pixel with the largest variance is set to 0; then the n point cloud feature points closest to the starting feature point in the remaining two channels are calculated; finally, they are sorted in order of distance from the camera and these point cloud feature points are determined one by one to see whether they fall in the dense point cloud.

[0061] In some specific implementations, the key to this solution is to achieve accurate positioning of feature points through variance analysis and point cloud screening. For example, when the Z-axis variance in the XYZ channel is the largest, the Z channel pixel will be set to 0, and then the nearest point cloud feature point will be searched in the X and Y channels. This method can effectively prevent the starting feature point from falling into the sparse point cloud area, thereby ensuring the accuracy of the tracking process. Furthermore, through distance sorting and dense point cloud screening, the most appropriate pixel feature point can be selected for replacement. Through dynamic adjustment and precise positioning, the problem of inaccurate feature point tracking caused by sparse point clouds in traditional methods can be effectively solved. Specifically, when the starting feature point is about to enter the sparse point cloud area, a more accurate replacement point can be found in time to ensure the continuity and accuracy of the tracking process.

[0062] Further, step S1 includes:

[0063] S11: Calibrate the camera, obtain camera parameters, read the endoscopic surgery video frame, and use the camera parameters to perform left and right eye correction on the video frame to obtain a corrected video frame, which is a two-dimensional image to be processed;

[0064] S12: performing stereo matching on the two-dimensional image to be processed to obtain a corresponding disparity map;

[0065] S13: Perform three-dimensional reconstruction on the two-dimensional image to be processed by using the disparity map and the camera parameters to obtain a three-dimensional reconstruction result.

[0066] Among them, camera calibration is to obtain internal and external parameters to ensure the accuracy of image correction. Video frame correction can eliminate lens distortion and improve image quality. The generation of disparity map is the key to stereo matching. It reflects the position difference of pixel points in the image under different viewing angles and provides depth information for 3D reconstruction. The 3D reconstruction process converts the 2D image into point cloud data with spatial coordinates, laying the foundation for subsequent feature point tracking.

[0067] Further, step S2 includes:

[0068] S21: Calculate the distance array of the entire current point cloud in the 3D reconstruction result;

[0069] S22: Taking the starting feature point as the local area with a radius of r, calculate the average spatial distance of the local area;

[0070] S23: Compare the average spatial distance with the preset critical distance. If the average spatial distance is greater than the preset critical distance, it is determined that the starting feature point corresponds to the sparse point cloud in the three-dimensional reconstruction result; if the average spatial distance is less than or equal to the preset critical distance, it is determined that the starting point is not in the sparse point cloud of the three-dimensional reconstruction result.

[0071] The distance array of the current point cloud as a whole includes the distance between each point cloud in the current point cloud as a whole and the starting feature point.

[0072] Among them, by calculating and comparing the spatial distance of the local area, it is possible to accurately judge whether the starting feature point is located in the sparse point cloud. Specifically, when the average spatial distance of the local area is large, it means that the point cloud distribution in this area is sparse; conversely, when the average spatial distance is small, it means that the point cloud distribution in this area is dense.

[0073] Among them, in specific applications, the distance array calculation and sorting steps can be implemented in a variety of ways. For example, the Euclidean distance formula can be used to calculate the distance between each point in the point cloud, and then the distance array can be sorted using algorithms such as quick sort or merge sort. When calculating the average spatial distance of a local area, a sliding window method can be used. Specifically, with the starting feature point as the center, a number of points are sampled in a circular area with a radius of r, and the average distance between these points is calculated. The choice of radius r can be adjusted according to the specific application scenario and point cloud density to ensure a balance between the representativeness of the sampling and the computational efficiency. The preset critical distance can be used to determine a suitable threshold through statistical analysis or experimental verification. For example, sample data with known point cloud density can be selected to analyze the average spatial distance distribution of different density areas to obtain a reasonable critical value. As a preferred embodiment, the preset critical distance can be set to α times the median of the overall point cloud distance array, where α is a given threshold factor. This method can adaptively adjust the judgment criteria according to the density distribution of the overall point cloud, thereby improving the robustness and applicability of the method.

[0074] Furthermore, after step S2, the following steps are further included:

[0075] S24: When the starting feature point is in the sparse point cloud, the currently selected starting feature point is discarded, and a pixel point is randomly selected again as the starting feature point until the starting feature point is no longer in the sparse point cloud of the three-dimensional reconstruction result.

[0076] In practical applications, the radius r of the local area can be adjusted according to the specific scene requirements, and generally 3-10 pixel units can be selected. When it is determined that the starting feature point is located in the sparse point cloud, the system will automatically abandon the point and randomly select a new pixel point in the image as the starting feature point. This process will be repeated until a suitable point that is not in the sparse point cloud is found.

[0077] In some specific embodiments, suppose that when tracking a surgical instrument, the tip of the instrument is selected as the starting feature point. The system first calculates the average spatial distance of a local area with a radius of 5 pixels centered on the point. If the calculation result shows that the point is located in a sparse point cloud, the system will immediately abandon this point and reselect feature points at other locations on the surface of the instrument until a suitable tracking starting point is found. This solution ensures the reliability of the tracking starting point by promptly discovering and abandoning feature points located in a sparse point cloud, avoids possible errors caused by tracking feature points in a sparse point cloud area, and thus improves the overall tracking accuracy.

[0078] Furthermore, in step S1, the distance array of the entire current point cloud in the 3D reconstruction result is calculated and the formula for sorting from small to large is: , is the distance array of the entire current point cloud, is the current point cloud in the 3D reconstruction result, is the point cloud closest to the starting feature point, A function that sorts the distance array from small to large.

[0079] The scheme uses a specific formula to calculate the distance array of the current point cloud in the 3D reconstruction result and sorts it from small to large. This method can effectively process 3D point cloud data and provide important basic information for subsequent feature point tracking.

[0080] Furthermore, in step S23, the critical distance is preset to ,in, is the distance array of the entire current point cloud, is the preset critical distance, is a given threshold factor, , Is a function to get the number of elements in the distance array.

[0081] Among them, the preset critical distance can be adaptively adjusted according to the distribution characteristics of the point cloud. In the specific implementation, the given threshold factor can be adjusted according to the actual application scenario to obtain the best judgment effect. Compared with the prior art, this application proposes a sparse point cloud judgment method based on average spatial distance and preset critical distance, which realizes the automatic screening of the starting feature points and can improve the accuracy of feature point tracking while maintaining the details of the original image.

[0082] Please refer to Figure 2 , a key feature point tracking device, the device comprising:

[0083] The three-dimensional reconstruction module 201 is used to obtain a two-dimensional image to be processed, and perform three-dimensional reconstruction on the two-dimensional image to be processed to obtain a three-dimensional reconstruction result;

[0084] Point selection module 202: used to select a starting feature point from the two-dimensional image to be processed, and determine whether the starting feature point corresponds to the sparse point cloud of the three-dimensional reconstruction result;

[0085] Judgment module 203: when the starting feature point is not in the sparse point cloud, track the change position of the starting feature point and judge in real time whether the starting feature point falls into the sparse point cloud;

[0086] Tracking module 204: used to obtain the pixel feature point in the dense point cloud closest to the real-time position of the starting feature point if the starting feature point falls into the sparse point cloud during the tracking process, and replace the starting feature point with the pixel feature point for subsequent calculations.

[0087] Among them, the three-dimensional reconstruction module 201 is responsible for acquiring the two-dimensional image to be processed and performing three-dimensional reconstruction to obtain the three-dimensional reconstruction result. The point extraction module 202 obtains the starting feature point arbitrarily selected from the two-dimensional image to be processed, and determines whether the starting feature point exists in the sparse point cloud of the three-dimensional reconstruction result. When the starting feature point is not in the sparse point cloud, the judgment module 203 continuously tracks the position change of the starting feature point and judges in real time whether the starting feature point enters the sparse point cloud. During the tracking process, if the starting feature point enters the sparse point cloud, the tracking module 204 obtains the dense point cloud pixel feature point closest to the starting feature point, and replaces the original starting feature point for subsequent calculations.

[0088] In a specific implementation, the 3D reconstruction module 201 can use stereo matching and disparity map calculation, the point acquisition module 202 can determine the point cloud distribution by calculating the average spatial distance of the local area, the judgment module 203 can monitor the position change of the feature point in real time, and the tracking module 204 can be implemented by calculating the XYZ channel variance and selecting the nearest point cloud feature point. These technical means can effectively prevent the starting feature point from falling into the sparse point cloud area and improve the accuracy of feature point tracking.

[0089] In practical applications, the device performs camera calibration and image correction on the endoscope video, and then performs 3D reconstruction. During the reconstruction process, the starting feature point is dynamically selected and it is determined whether it is located in the sparse point cloud. If it is not in the sparse point cloud, tracking begins. During the tracking process, once the starting feature point enters the sparse point cloud, it immediately switches to the nearest dense point cloud feature point to ensure the continuity and accuracy of the tracking process.

[0090] Please refer to Figure 3 , Figure 3 The present invention provides a structural schematic diagram of an electronic device provided in an embodiment of the present application. The present application provides an electronic device 3, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not shown). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiment to achieve the following functions: obtaining a two-dimensional image to be processed, and performing three-dimensional reconstruction on the two-dimensional image to be processed to obtain a three-dimensional reconstruction result; arbitrarily selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to a sparse point cloud of the three-dimensional reconstruction result; when the starting feature point is not in the sparse point cloud, tracking the change position of the starting feature point, and determining in real time whether the starting feature point falls into the sparse point cloud; during the tracking process, if the starting feature point falls into the sparse point cloud, obtaining a pixel feature point in the dense point cloud closest to the real-time position of the starting feature point, and replacing the starting feature point with the pixel feature point for subsequent calculation.

[0091] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method in any optional implementation of the above-mentioned embodiment is executed to achieve the following functions: obtaining a two-dimensional image to be processed, and performing three-dimensional reconstruction on the two-dimensional image to be processed to obtain a three-dimensional reconstruction result; arbitrarily selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to a sparse point cloud of the three-dimensional reconstruction result; when the starting feature point is not in the sparse point cloud, tracking the changing position of the starting feature point, and determining in real time whether the starting feature point falls into the sparse point cloud; during the tracking process, if the starting feature point falls into the sparse point cloud, obtaining the pixel feature point in the dense point cloud closest to the real-time position of the starting feature point, and replacing the starting feature point with the pixel feature point for subsequent calculations.

[0092] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0093] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0095] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0096] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A key feature point tracking method, characterized in that: The method comprises the steps of: S1: Acquire a two-dimensional image to be processed, and perform three-dimensional reconstruction on the two-dimensional image to be processed to obtain a three-dimensional reconstruction result; S2: arbitrarily selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to a sparse point cloud of the three-dimensional reconstruction result; S3: When the starting feature point is not in the sparse point cloud, tracking the change position of the starting feature point, and determining in real time whether the starting feature point falls into the sparse point cloud; S4: During the tracking process, if the starting feature point falls into the sparse point cloud, a pixel feature point in the dense point cloud closest to the real-time position of the starting feature point is obtained, and the starting feature point is replaced by the pixel feature point for subsequent calculation; Step S2 includes: S21: Calculate the distance array of the entire current point cloud in the three-dimensional reconstruction result; S22: according to the distance array, taking a local area with a radius of r from the starting feature point, and calculating the average spatial distance of the local area; S23: comparing the average spatial distance with a preset critical distance, and if the average spatial distance is greater than the preset critical distance, determining that the starting feature point corresponds to the sparse point cloud in the three-dimensional reconstruction result; if the average spatial distance is less than or equal to the preset critical distance, determining that the starting feature point is not in the sparse point cloud of the three-dimensional reconstruction result; Step S4 includes: S41: During the tracking process, if the starting feature point falls into the sparse point cloud, then calculating the XYZ channel variance of the local area centered on the real-time position of the starting feature point; S42: setting the pixel of the channel with the largest variance in the XYZ channel to 0, and calculating the n point cloud feature points closest to the starting feature point in the remaining two channels; S43: Sort the n point cloud feature points in ascending order according to the point cloud distance from the camera, and calculate in turn whether the n point cloud feature points are in the dense point cloud, until the m-th point cloud feature point falling in the dense point cloud is found, the m-th point cloud feature point is the pixel feature point, and the pixel feature point is used to replace the starting feature point for subsequent calculations, wherein n is the total number of point cloud feature points in the remaining two channels that are closest to the starting feature point, and m is the number of points in the sorted point cloud feature points that fall in the dense point cloud, and m is less than or equal to n.

2. A key feature point tracking method according to claim 1, characterized in that: Step S1 includes: S11: calibrating the camera, obtaining camera parameters, and reading endoscopic surgery video frames, and using the camera parameters to perform left-right eye correction on the video frames to obtain corrected video frames, where the corrected video frames are two-dimensional images to be processed; S12: performing stereo matching on the two-dimensional image to be processed to obtain a corresponding disparity map; S13: Performing three-dimensional reconstruction on the two-dimensional image to be processed by using the disparity map and the camera parameters to obtain a three-dimensional reconstruction result.

3. A key feature point tracking method according to claim 1, characterized in that: After step S2, the following steps are also included: S24: When the starting feature point is in the sparse point cloud, the currently selected starting feature point is discarded, and a pixel point is randomly selected again as the starting feature point until the starting feature point is no longer in the sparse point cloud of the three-dimensional reconstruction result.

4. A key feature point tracking method according to claim 3, characterized in that: The formula for calculating the distance array of the entire current point cloud in the three-dimensional reconstruction result in step S21 is: , is the distance array of the entire current point cloud, is the current point cloud in the 3D reconstruction result, is the point cloud closest to the starting feature point, A function that sorts the distance array from small to large.

5. A key feature point tracking method according to claim 1, characterized in that: In step S23, the preset critical distance is ,in, is the distance array of the entire current point cloud, is the preset critical distance, is a given threshold factor, , Is a function that gets the number of elements in the distance array.

6. A key feature point tracking device, characterized in that: Applied in the steps of any one of the methods of claims 1 to 5 above, the device comprises: 3D reconstruction module: used to obtain the 2D image to be processed, and perform 3D reconstruction on the 2D image to be processed to obtain a 3D reconstruction result; Point selection module: used for arbitrarily selecting a starting feature point from the two-dimensional image to be processed, and determining whether the starting feature point corresponds to the sparse point cloud of the three-dimensional reconstruction result; A judgment module: when the starting feature point is not in the sparse point cloud, tracking the change position of the starting feature point, and judging in real time whether the starting feature point falls into the sparse point cloud; Tracking module: used for obtaining the pixel feature point in the dense point cloud closest to the real-time position of the starting feature point if the starting feature point falls into the sparse point cloud during the tracking process, and replacing the starting feature point with the pixel feature point for subsequent calculation.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 5 are executed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are executed.

Images